About the Role
We are looking for an experienced Senior Data Engineer to design, build, and optimize scalable data platforms and pipelines that enable data-driven decision-making. You will work closely with our client's stakeholders to develop reliable, secure, and high-performance data solutions while driving best practices in Data Engineering, CI/CD, and Site Reliability Engineering (SRE).
This role is ideal for engineers who are passionate about building modern data platforms, ensuring data quality, and delivering resilient cloud-based data solutions.
Key Responsibilities
Data Pipeline Development & Architecture
- Translate business data requirements into technical solutions.
- Design, develop, and maintain scalable batch and real-time data pipelines.
- Build data ingestion pipelines to collect, cleanse, transform, and integrate data from multiple source systems.
- Design reusable, scalable, and optimized data models to support analytics and reporting.
- Develop secure mechanisms for users and applications to access enterprise data.
- Research, evaluate, and implement new technologies to improve the organization's data infrastructure.
Data Quality, Testing & Validation
- Design and implement data quality frameworks to ensure data accuracy, completeness, and consistency.
- Develop and execute functional, regression, and exploratory testing for data pipelines.
- Build automated testing capabilities to improve deployment quality and reduce production defects.
- Perform data reconciliation and end-to-end validation across multiple systems.
- Support User Acceptance Testing (UAT) and collaborate with business stakeholders to validate solutions.
Operations, Governance & Continuous Improvement
- Monitor and maintain databases and ETL platforms, including performance tuning, capacity planning, troubleshooting, and issue resolution.
- Collaborate with infrastructure and security teams to ensure scalable and secure data platforms.
- Work with data stewards to establish and enforce data governance standards and best practices.
- Maintain metadata repositories and data catalogues.
- Implement data security controls, including access management, encryption, and data masking.
- Drive continuous improvement in CI/CD pipelines, automation, and Site Reliability Engineering (SRE) practices.
Requirements
Qualifications
- Bachelor's Degree in Computer Science, Information Technology, Software Engineering, or a related discipline.
- Minimum 5 years of experience in Data Engineering or related roles.
Technical Skills
- Strong understanding of Data architecture Data modelling Data structures and algorithms Database design and data storage concepts
- Hands-on experience designing and maintaining batch and streaming data pipelines.
- Strong programming skills in Python SQL Shell Scripting
- Experience with workflow orchestration tools such as Apache Airflow Azure Data Factory
- Hands-on experience with Databricks.
- Experience working with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform (GCP).
Data Quality & Testing
- Experience implementing data quality frameworks and validation processes.
- Experience developing automated testing for data pipelines.
- Familiarity with defect tracking and test management tools such as JIRA.
DevOps & CI/CD
- Experience building and maintaining CI/CD pipelines.
- Familiarity with Docker Git Terraform
- Understanding of infrastructure automation and deployment best practices.
- Experience implementing data governance, security, and compliance standards.
Preferred Qualifications
Candidates with the following experience will have an advantage:
- Government sector projects.
- Data governance, data infrastructure, or data security initiatives.
- Climate or Weather domain projects.
Required Technical Skills
- Python
- SQL
- Shell Scripting
- Apache Airflow
- Azure Data Factory
- Databricks
- AWS / Azure / Google Cloud Platform (GCP)
- ETL Development
- Data Pipelines
- Data Modelling
- Data Quality Testing
- Data Governance
- CI/CD
- Docker
- Git
- Terraform
- Site Reliability Engineering (SRE)
- JIRA