Lead Data Engineer - AWS / AI / SQL

Salary: £70,000 - 90,000 per year

Requirements:

• We are looking for an experienced Data Engineer who has built and operated complete data platforms rather than solely delivering individual components.
• Experience across batch data processing, ELT and orchestration.
• Experience with modern streaming technologies such as Flink or comparable platforms.
• Experience with AWS data services, including S3, Lambda, Glue, Athena, Kinesis and RDS.
• Experience with infrastructure as code using Terraform.
• Experience with CI/CD and automated testing.
• Experience with lakehouse architecture and open table formats such as Iceberg.
• Experience with medallion architecture and dimensional data modelling.
• Strong SQL and Python skills.
• Experience with SQL Server, SSIS/SSDT and Power BI.
• Experience with semantic layers, common metrics and data glossaries.
• Experience with data governance, lineage, retention, classification and auditability.
• Experience managing and developing a small technical team.
• Comfortable using advanced AI coding assistants, such as Claude Code or equivalent tools, to improve engineering productivity while maintaining high standards of quality, governance and security.
• Desirable experience in a regulated environment.
• Desirable experience with PySpark for large-scale data transformations.
• Desirable experience migrating legacy data estates to lakehouse or metadata-driven architectures.
• Desirable experience with Customer Data Platforms or event-driven systems.
• Desirable experience with data-quality tools such as dbt tests or Great Expectations.
• Desirable experience with MCP servers or governed natural-language data interfaces.
• Desirable experience with financial crime, fraud or regulatory-reporting data.
• Desirable experience with Power BI administration and semantic modelling.

Responsibilities:

• Own the reliability, security and ongoing development of the data platform.
• Modernise the existing SQL Server, SSIS/SSDT and Power BI environment.
• Develop scalable batch and real-time data-processing solutions.
• Lead the design of a Customer Data Platform.
• Build and evolve a lakehouse architecture using AWS technologies.
• Create governed semantic models, consistent metrics and clear data definitions.
• Enable secure, self-service data analysis using modern AI tooling.
• Automate manual, repetitive or fragile data processes.
• Support BI, Data Science, Machine Learning and Applied AI teams.
• Take ownership of data governance, security, monitoring and platform costs.
• Line-manage, develop and help grow the Data Engineering team.
• Work closely with technical, operational and senior business stakeholders.

Technologies:

• AI
• AWS
• Lambda
• Business Intelligence
• CI/CD
• Claude Code
• Flink
• Support
• Machine Learning
• MCP
• Power BI
• Python
• PySpark
• SQL
• Security
• SSIS
• Terraform
• dbt
• Cloud

More:

We are an established organisation offering a hands-on Lead Data Engineer opportunity in Manchester on a hybrid working pattern, with a salary circa £85K. This is an influential role where we want someone to take ownership of our existing data estate and lead the development of a modern, scalable data platform. You will help build a platform supporting Business Intelligence, Data Science and Applied AI, introduce modern batch and real-time data-processing capabilities, automate inefficient processes and make trusted data more accessible across the organisation. Alongside remaining technically hands-on, you will manage and develop a small team of Data Engineers, with the scope to shape how data and AI are used across the organisation.

last updated 40 week of 2026