The Senior Data Engineer is responsible for designing, developing, implementing, and supporting scalable enterprise data solutions that enable reliable, secure, and high-quality data delivery across the organisation. The role focuses on building and maintaining data pipelines, data transformation processes, data models, and data platforms using technologies such as Databricks, SQL, and Informatica Intelligent Data Management Cloud (IDMC).
The incumbent collaborates with business stakeholders, project teams, infrastructure teams, and vendors to deliver robust data engineering solutions while ensuring data quality, governance, operational stability, and continuous improvement of the enterprise data platform.
DUTIES & RESPONSIBILITIES
- Design, develop, and maintain scalable data pipelines, ETL/ELT processes, data ingestion frameworks, and transformation solutions using Databricks, SQL, and IDMC.
- Develop and maintain data models, fact and dimension tables, Delta tables, reusable data assets, and enterprise data structures to support analytical and reporting requirements.
- Ensure data quality through validation, reconciliation, data integrity checks, source-to-target verification, and automated quality controls. Investigate and resolve data discrepancies.
- Provide production support, troubleshoot pipeline failures, perform root cause analysis, implement permanent fixes, and maintain operational documentation and runbooks.
- Maintain compliance with enterprise standards, security requirements, coding practices, testing frameworks, deployment processes, and governance controls. Participate in peer reviews and technical oversight activities.
- Collaborate with business analysts, data analysts, project managers, platform teams, vendors, and stakeholders to translate requirements into technical solutions. Drive continuous improvement, automation, performance optimisation, and platform modernisation initiatives.
QUALIFICATIONS
Education:
- Bachelor's Degree in Computer Science, Information Technology, Data Engineering, Information Systems, or a related discipline.
- Relevant professional certifications in Databricks, Cloud Data Engineering, Informatica, or Data Platforms will be advantageous.
Experience:
- Minimum 8 years of relevant experience in Data Engineering, ETL/ELT development, Data Integration, or Data Platform implementation.
- Experience working with multiple source systems and complex data ecosystems.
- Experience supporting production data environments and business-critical data pipelines.
- Candidates with fewer years of experience may be considered for a Data Engineer role.
Skills & Knowledge:
Mandatory
- Strong hands-on experience with SQL.
- Strong experience developing batch and incremental data pipelines using Databricks and IDMC.
- Good understanding of ETL/ELT, data integration, data warehousing, data lakes, and lakehouse architectures.
- Experience in data modelling, including relational and dimensional modelling techniques.
- Experience in data quality management, reconciliation, troubleshooting, and root cause analysis.
- Experience supporting production environments and operational support activities.
- Experience implementing automation, DataOps practices, and CI/CD processes within data engineering environments.
Preferred
- Experience designing, administering, or architecting enterprise data platforms using Databricks and/or IDMC.
- Experience implementing streaming data pipelines and real-time data processing solutions.
- Experience with Unity Catalog, access management, and data governance frameworks.
- Knowledge of performance tuning and cost optimisation for cloud-based data platforms.
COMPETENCIES / PERSONAL ATTRIBUTES
Communication: Able to communicate technical concepts effectively to both technical and non-technical stakeholders.
Collaboration: Works effectively with cross-functional teams, business users, vendors, and project stakeholders.
Problem-Solving: Demonstrates strong analytical skills and the ability to diagnose and resolve complex data issues.
Adaptability: Responds positively to changing priorities, technologies, and business needs.
Technical Leadership: Provides technical guidance, contributes to code reviews, and promotes engineering best practices.
Continuous Learning: Maintains awareness of emerging technologies and seeks opportunities to improve data platforms and processes.
Integrity & Accountability: Takes ownership of deliverables and adheres to organisational governance, security, and compliance requirements.
Team Player: Demonstrates humility, professionalism, and willingness to support team objectives.