Lead Data Engineer

Overview

Associate Director – Data & AI Recruitment

Location: South England x2 days a month in office. Package: Competitive salary + strong benefits.

We’re representing a major financial services organisation that is investing heavily in next-generation data and digital platforms. They are building a modern, cloud-native data ecosystem to power AI, analytics, and customer-facing innovation and now seek a Lead Data Engineer to guide this transformation. The company has an incredible Data Leadership team with good tenure, and alongside impactful Data Science projects, they have multiple AI / LLM based projects in production due to their mature Data offering. This is a rare opportunity to join the team at an impactful, senior level due to the typically very long tenure.

The role

This position blends technical leadership with hands-on engineering. You’ll oversee the design and delivery of scalable data solutions, lead and mentor a small engineering group, and set the standards for how data is built, managed, and consumed across the organisation.

It’s a chance to make architectural decisions, work directly with product and AI specialists, and drive forward best practices in a rapidly modernising environment.

Key responsibilities

• Directing major data engineering initiatives and overseeing a team of engineers
• Designing and enhancing data warehouses, pipelines, and integration frameworks at scale
• Partnering with analytics, AI, product, and business teams to create end-to-end data solutions
• Defining coding guidelines, frameworks, and operating models across the platform
• Leading the migration of legacy environments into a flexible, cloud-first stack
• Contributing to the build and delivery of advanced data products and GenAI-enabled applications
• Raising the bar on data quality, governance, and compliance processes

Skills & experience required

• Proven background leading data engineering teams or projects in a technology-driven business
• Expert knowledge of modern cloud data platforms (Databricks, Snowflake, ideally AWS)
• Advanced Python programming skills and fluency with the wider Python data toolkit
• Strong capability with SQL, Spark, Airflow, Terraform, and workflow orchestration tools
• Solid understanding of CICD practices using Git, Jenkins, or equivalent tooling
• Hands-on experience building both batch and streaming data integration pipelines
• Depth in data modelling, ingestion, and optimisation for large-scale environments
• Any exposure to production use of Generative AI is highly advantageous

What’s on offer

• A hybrid working approach that values flexibility
• The opportunity to combine leadership responsibilities with technical delivery
• Influence over the design of next-generation data and AI capabilities within a household-name financial services brand