Key Responsibilities
· Design, develop, and maintain scalable data pipelines using
PySpark
on AWS.
· Build and orchestrate ETL/ELT workflows using
AWS Glue
and
AWS Step Functions
.
· Develop serverless applications and automation using
AWS Lambda
.
· Write clean, efficient, and maintainable Python/PySpark code following engineering best practices.
· Provision and manage cloud infrastructure using
Terraform
(Infrastructure as Code).
· Implement and maintain
CI/CD pipelines
to automate code deployment, testing, and infrastructure changes.
· Monitor, troubleshoot, and optimize data pipelines for performance, reliability, and cost efficiency.
· Collaborate with business stakeholders to deliver data solutions.
· Follow DevOps, security, and coding standards throughout the engagement.
Required Skills
· Strong hands-on experience with
PySpark
and Python for data engineering.
· Experience developing ETL pipelines using
AWS Glue
.
· Proficiency with
AWS Step Functions
for workflow orchestration.
· Experience building serverless solutions using
AWS Lambda
.
· Hands-on experience with
Terraform
for Infrastructure as Code (IaC).
· Experience implementing
CI/CD
pipelines using tools such as GitLab, GitHub Actions, Jenkins, or similar.
· Good understanding of AWS services, data lakes, IAM, S3, CloudWatch, and monitoring.