Data Engineer – AWS Data Platform
Job Summary
We are looking for a highly skilled and motivated data engineer with strong expertise in AWS data services to join our data platform team. The ideal candidate will have hands-on experience designing scalable data pipelines, workflow orchestration frameworks, and large-scale data migration solutions.
This role will be responsible for building robust cloud-native data engineering solutions on AWS, migrating datasets from legacy systems and data warehouses, and ensuring secure and efficient data processing pipelines across distributed environments.
Key Responsibilities
AWS Data Pipeline Development
- Design and implement scalable ETL/ELT data pipelines using AWS Glue, AWS Lambda, and AWS S3.
- Build and maintain high-performance data ingestion frameworks for processing large-scale datasets.
- Implement data pipelines for data warehousing and analytics platforms such as AWS Redshift.
- Optimize storage and querying strategies using AWS S3 data lakes.
Data Workflow Orchestration
- Develop and maintain data workflow orchestration frameworks using tools such as Apache Airflow or AWS Step Functions.
- Automate complex workflows including data ingestion, transformation, validation, and loading processes.
- Build reusable and configurable workflows to support multiple data processing use cases.
Data Migration & Integration
- Lead data migrations from legacy data warehouse technologies to modern AWS data platforms.
- Perform data migration from RDBMS systems (e.g., MySQL, SQL Server, Oracle) to AWS S3 or AWS Redshift.
- Design scalable migration frameworks for large datasets with minimal downtime.
- Integrate data sources from enterprise applications and external systems.
Data Security & Governance
- Implement secure data pipelines using AWS security best practices.
- Manage access control and data governance using AWS IAM and Lake Formation.
- Ensure data encryption, access management, and compliance across all data platforms.
Performance Optimization & Monitoring
- Monitor data pipelines and troubleshoot performance issues.
- Optimize ETL workflows for scalability, reliability, and cost efficiency.
- Implement logging, monitoring, and alerting mechanisms for data pipelines.
Required Skills & Qualifications (Must Have)
- 5+ years of experience in Data Engineering or Data Platform development
- Strong hands-on experience with:
- AWS
- AWS Glue
- AWS S3
- AWS Lambda
- Experience with Data Workflow Orchestration tools such as Apache Airflow or AWS Step Functions
- Experience performing data migrations from other data warehouse technologies
- Experience performing data migrations from RDBMS systems to AWS S3 or AWS Redshift
- Strong expertise in Python and SQL for building scalable data pipelines
- Solid understanding of ETL/ELT concepts, data partitioning, and distributed data processing
- Experience working with version control systems such as GitLab or Bitbucket
- Strong debugging, analytical thinking, and problem-solving skills
- Basic understanding of Object-Oriented Programming concepts
Industry Knowledge & Experience
- Experience building cloud-native data engineering solutions on AWS
- Experience with data warehouse architectures and large-scale analytics platforms
- Hands-on experience with data extraction, transformation, and migration frameworks
- Experience working in high-volume data environments such as FinTech, analytics platforms, or enterprise data systems
Good to Have Skills
- IBM Cognos
- AWS Athena
- AWS Lake Formation
- AWS Redshift
- AWS Glue Data Catalog
- AWS SageMaker
- AWS IAM
Soft Skills
- Strong communication skills to present technical solutions and recommendations to stakeholders
- Ability to work cross-functionally in a fast-paced and evolving environment
- Detail-oriented with a proactive approach to identifying and solving data platform challenges
- Ability to collaborate effectively with data scientists, analysts, and platform engineering teams