Role Overview
We are seeking an experienced Data Engineer to join a Data Engineering & Infrastructure team supporting a large-scale enterprise data platform.
The platform consolidates workforce and organisational data from multiple source systems to enable analytics, reporting, data sharing, and data-driven decision-making across the organisation.
This is a hands-on, delivery-focused role covering data ingestion, transformation, data quality, data modelling, pipeline operations, and downstream data services.
You will work closely with technical teams, source system owners, business users, and technology partners to maintain and enhance the data platform while ensuring data remains accurate, reliable, traceable, and accessible for approved analytics and reporting requirements.
The current environment primarily uses AWS data services, SQL and Python, with Tableau supporting downstream analytics and visualisation.
Key Responsibilities
Data Pipeline & Interface Engineering
- Develop and maintain ETL/data pipelines and file-based interfaces for ingesting data from multiple source systems.
- Perform data profiling, source-to-target mapping, interface specifications, and dataset onboarding.
- Build reliable and maintainable ingestion processes for structured and file-based data sources.
Data Transformation & Modelling
- Develop and maintain SQL and Python processing logic to cleanse, standardise, transform, and reconcile data.
- Work with both current and historical datasets, including complex or legacy data structures.
- Transform source data into clean, reusable, and reliable datasets for downstream consumption.
Data Model & Dataset Maintenance
- Maintain and enhance data models, tables, views, datasets, and dependencies based on evolving business requirements.
- Maintain clear and traceable data mappings, definitions, transformation logic, and processing rules.
- Assess the downstream impact of changes to data structures and transformation processes.
Data Quality, Validation & Lineage
- Develop and automate data validation, reconciliation, and cleansing checks.
- Investigate data anomalies and discrepancies across source, interface, and transformation layers.
- Trace data lineage and identify root causes of data quality issues.
- Implement and validate appropriate fixes and preventive measures.
Pipeline Operations & Reliability
- Monitor data pipelines, interfaces, and scheduled processing activities.
- Troubleshoot pipeline failures and data processing issues.
- Perform reruns and recovery activities where required.
- Continuously improve pipeline reliability, performance, maintainability, and operational efficiency.
System Enhancements & Testing
- Support system enhancements and change requests by reviewing requirements and assessing technical impact.
- Develop or modify data processing logic based on approved requirements.
- Execute functional and data validation testing.
- Document defects, investigate issues, and support resolution through implementation.
- Support UAT and production deployment activities where required.
Cloud Data & Downstream Consumption
- Work with AWS-based data services, including technologies such as Amazon S3 and Athena.
- Develop and maintain tables, views, queries, and governed datasets for approved reporting and analytics requirements.
- Support Tableau data sources and workbooks where required.
- Enable reliable and controlled consumption of enterprise data by downstream users and applications.
Technical Collaboration & Documentation
- Translate business and stakeholder requirements into data mappings, transformation rules, and practical technical solutions.
- Collaborate with business users, technical teams, source system owners, and external technology partners.
- Maintain technical documentation, data mappings, processing rules, runbooks, and operational procedures.
- Communicate technical findings, dependencies, risks, and issues clearly to relevant stakeholders.
Data Governance & Assurance
- Follow established data governance, security, access control, and change management practices.
- Support audit, access review, and assurance activities by providing required technical information and evidence.
- Assist with technical remediation activities arising from governance or assurance reviews.
Requirements
Technical Skills
- Strong proficiency in SQL and working proficiency in Python.
- Hands-on experience developing ETL/data pipelines, data transformations, validation, and automation.
- Experience working with cloud-based data platforms.
- Experience with AWS data services such as Amazon S3, Amazon Athena, IAM, CSV/Parquet, data partitioning, tables, views, and queries.
Data partitioning
- Familiarity with Spark, Databricks, or similar modern data engineering technologies would be advantageous but is not mandatory.
- Data Engineering & Data Warehousing
- Strong understanding of ETL, data pipeline design, and pipeline operations.
- Experience working with relational and file-based datasets.
- Good understanding of data warehousing and data modelling concepts.
- Ability to work with complex, inconsistent, or legacy source structures and transform them into clean and standardised datasets.
Data Quality & Troubleshooting
- Experience implementing automated data validation and reconciliation checks.
- Strong analytical and troubleshooting capabilities.
- Ability to investigate data discrepancies across multiple systems and processing layers.
- Understanding of data lineage, root-cause analysis, and data quality management.
Stakeholder Management
- Strong communication and collaboration skills.
- Comfortable working with business users, technical teams, system owners, and external vendors.
- Able to translate business requirements into technical data solutions.
- Able to explain technical findings and data issues clearly to both technical and non-technical stakeholders.
Analytics & Visualisation
- Experience working with enterprise, workforce, operational, or similarly complex datasets would be advantageous.
- Experience with Tableau or similar BI and visualisation tools is beneficial.
- Experience developing or maintaining reporting datasets, data sources, and dashboards would be an advantage.
Personal Attributes
- Analytical: Able to investigate complex data problems and identify root causes.
- Independent: Comfortable managing assigned engineering work with minimal supervision.
- Detail-oriented: Maintains high standards for data accuracy, documentation, and technical quality.
- Pragmatic: Able to assess technical dependencies and propose workable solutions.
- Collaborative: Works effectively across business, engineering, and vendor teams.
- Ownership-driven: Takes responsibility for pipeline reliability and the quality of delivered datasets.
- Delivery-focused: Able to manage priorities, meet timelines, and escalate risks or blockers promptly.