M28 - Data Engineer

Role Overview

We are seeking an experienced Data Engineer to design, build, and deploy a scalable and reliable data lakehouse platform on AWS. The role will focus on developing end-to-end data pipelines, establishing robust data quality frameworks, and delivering production-ready solutions that form the foundation of the organisation's data infrastructure.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using AWS services such as Glue, Step Functions, Lambda, and S3.
• Architect and implement data lakehouse solutions using Apache Iceberg, S3 Tables, or similar open table formats.
• Implement lakehouse capabilities including schema evolution, partition evolution, and ACID transactions.
• Optimise data pipelines and storage solutions for performance, scalability, reliability, and cost efficiency.
• Define and implement automated data quality validation frameworks to ensure data accuracy, completeness, and consistency.
• Establish data quality metrics, monitoring, and alerting mechanisms across the data platform.
• Implement and maintain data governance standards and ensure compliance with relevant policies and requirements.
• Develop production-quality code and deploy solutions on AWS cloud infrastructure.
• Implement CI/CD practices to support automated, repeatable, and reliable deployments.
• Use infrastructure-as-code tools such as Terraform or CloudFormation to provision and manage cloud infrastructure.
• Work closely with cross-functional teams to design and deliver data engineering solutions.
• Communicate technical designs and concepts clearly to both technical and non-technical stakeholders.
• Produce comprehensive technical documentation covering architecture, pipelines, deployment, operations, and support procedures.
• Support knowledge transfer and handover of developed solutions to Day 2 operations and support teams.

Qualifications & Experience

• Degree in Computer Science, Data Engineering, Information Systems, or a related discipline.
• 3–5 years of relevant experience in data engineering, ETL/ELT development, or data platform engineering.
• Strong hands-on experience with AWS cloud services, particularly Glue, Step Functions, Lambda, and S3.
• Strong experience designing and developing scalable data pipelines and data lakehouse architectures.
• Hands-on experience with Apache Iceberg, S3 Tables, or similar open table formats.
• Good understanding of Apache Iceberg capabilities including schema evolution, partition evolution, ACID transactions, and table optimisation.
• Experience implementing automated data quality frameworks, validation rules, monitoring, and data governance controls.
• Experience developing and deploying production-grade applications or data solutions in cloud environments.
• Experience with CI/CD practices and infrastructure-as-code tools such as Terraform or CloudFormation.
• Experience working in a Government Commercial Cloud (GCC) environment is strongly preferred.
• Strong analytical, problem-solving, communication, and stakeholder management skills.
• AWS certifications such as AWS Certified Data Engineer, AWS Certified Solutions Architect, or equivalent certifications would be an advantage.