Data Platform Lead Engineer (SEL & SWL)

Job overview

Can you engineer data differently – and demonstrate the difference it makes?

We are looking for an exceptional Senior Data Platform Engineer to help shape, engineer and mature the data platforms and engineering capability supporting analytics across South East and South West London.

This is a senior technical practitioner and engineering leadership role for someone who combines deep Data Engineering expertise with strategic and analytical thinking. We are looking for someone who can design and build, provide technical challenge and assurance, raise engineering standards and demonstrate how better engineering translates into better analytical delivery.

You will move comfortably between architecture and implementation—understanding complex analytical requirements, determining appropriate engineering solutions and working collaboratively with Data Engineers, analysts, Data Scientists and other technical specialists to turn those solutions into reliable production services.

Advert

The Opportunity

You will have the opportunity to help shape and mature the Data Engineering capability supporting analytics across South East and South West London, working across Microsoft Fabric, Snowflake and Azure in a complex health and care data environment.

This is an opportunity to influence not only what we build, but how we engineer—establishing better approaches to architecture, ingestion, orchestration, transformation, data modelling, testing, deployment, monitoring, automation and reuse.

Working closely with Data Engineers, Data Scientists and analysts, you will see how engineering decisions affect analytical delivery and be able to target technical improvements where they can make the greatest difference.

You will combine hands-on engineering with technical leadership and assurance, helping us develop scalable, resilient and production-quality data products while demonstrably improving the speed, quality, reliability and efficiency of analytical delivery.

Working for our organisation

In 2025, the boards of NHS South East London ICB and NHS South West London ICB entered into a clustering arrangement, meaning that we share a chair, chief executive officer and an executive team, but we remain as separate, statutory organisations.

This also means that the ICBs will share some teams, who will be working across both organisations. Some teams will also be working across both geographical areas and integrated care systems.

ICBs are also changing, with a clear focus on strategic commissioning and delivering neighbourhood working.

This is an exciting time to work within our ICBs. We currently have the opportunity both to shape the future of our clustered organisations, and to be part of a radical transformation of care for the local people and communities across south London.

And while our two ICBs are different in some ways, improving lives for the people we serve is at the core of what we are both here to do.

Please note as you can only be employed by one ICB, this job description refers to ‘the ‘ICB’, but your role may cover both ICB organisations, and / or geographical areas.

Equality and Diversity

The ICB is committed to providing services and employment to a community with an increasing variety of backgrounds. To do this effectively it is essential that we promote equality and embrace diversity and treat everyone with dignity and respect. This includes a focus on building a just culture

Detailed job description and main responsibilities

Who are we looking for?

We are looking for an accomplished Data Platform Engineer with substantial technical depth and experience designing, building and operating complex data platforms, pipelines and analytical data models at scale.

You will understand the complete data lifecycle—from ingestion and orchestration through transformation, modelling, validation, deployment, monitoring and optimisation—and be able to determine the appropriate engineering approach for complex analytical requirements.

You will combine engineering expertise with strategic and analytical judgement. You will be comfortable interrogating architecture and code, challenging technical approaches, identifying technical debt and establishing engineering standards that improve reliability, maintainability, scalability and reuse.

Your experience is likely to include several of the following:

Microsoft Fabric • Snowflake • Azure • OneLake • Lakehouse architecture • Data Factory • Spark • Python • SQL • ETL/ELT • dimensional modelling • Data Vault • orchestration • Git • Azure DevOps/GitHub • CI/CD • Infrastructure as Code • data quality • MLOps • performance optimisation

You will also be pragmatic and highly collaborative. You will understand the problem before designing the solution, communicate complex engineering concepts clearly and take ownership of difficult technical problems from initial design through to stable production operation.

Above all, we are interested not simply in what you have built, but what became measurably better because you built it.

What makes the role different?

For us, successful Data Engineering is not simply about moving data from one place to another. The defining proposition of this role is to engineer for analytical value.

We want to move away from unnecessary manual processing and isolated technical solutions towards automated, reusable, governed, integrated and production-quality data products.

You will work directly with the analysts and Data Scientists who consume the data, enabling you to understand how engineering decisions affect analytical delivery and target technical effort where it can make the greatest difference.

You will have the authority and technical credibility to challenge existing approaches, improve patterns for ingestion, orchestration, transformation and modelling, and ensure engineering solutions are appropriately designed, tested, monitored and optimised.

Success will therefore be visible and measurable. It could mean reducing a data preparation process from hours to minutes; eliminating repetitive analyst intervention; improving data quality or pipeline reliability; increasing reuse of curated datasets and common transformation logic; reducing compute consumption; shortening the time required to develop analytical products; or releasing analytical capacity for higher-value work.

Ultimately, we want someone who can connect engineering excellence with measurable improvements in analytical delivery.

What will you do?

1. Lead the design, build and optimisation of scalable data pipelines and architecture — create resilient, secure and high-performing data flows from source systems through ingestion and transformation to curated analytical data products.
2. Provide technical leadership and engineering assurance for data orchestration and transformation — set and challenge engineering standards and ensure ETL/ELT solutions are automated, maintainable, observable, appropriately tested and engineered for performance, resilience and reuse.
3. Lead and technically assure analytical data modelling — work directly with analysts and Data Scientists to translate complex analytical requirements into robust, reusable data structures and semantic models, constructively challenging modelling approaches where better solutions are available.
4. Drive automation across Data Engineering and the wider analytical function — identify repetitive, manual and inefficient processes and develop engineering solutions that release capacity, increase reuse and improve analytical productivity.
5. Establish strong software and Data Engineering disciplines — embed automated testing, Git-based version control, CI/CD, deployment pipelines, monitoring, documentation, change control and Infrastructure as Code.
6. Engineer for reliability, resilience and data quality — establish validation, reconciliation, exception handling, recovery, backfill, monitoring and audit processes so analytical teams can confidently rely on the data they receive.
7. Optimise Microsoft Fabric, Snowflake and Azure workloads — apply strong technical judgement to performance, scalability, resilience, capacity consumption and cloud cost optimisation.
8. Enable advanced Analytics, Data Science and AI — create the pipelines, feature engineering, data structures and MLOps capabilities required to move analytical and Machine Learning solutions reliably into production.

These responsibilities retain the central requirements of the Band 8B JD: senior responsibility for pipelines and infrastructure, technical architecture, transformation, analytical modelling, testing, automation, optimisation, Data Science enablement and collaboration with analytical teams.

Why join us?

You will work within an integrated Intelligence & Insights function supporting South East and South West London, alongside Data Engineers, Data Scientists, Population Health specialists and wider analytical teams working with complex health and care data.

Your engineering will directly support Population Health Management, strategic commissioning, planning, performance and wider analytical delivery, providing the opportunity to see how technical decisions translate into practical analytical capability.

You will be joining at an important point in the development of our data environment, with the opportunity to work across Microsoft Fabric, Snowflake and Azure while helping establish the technical foundations for faster, more reliable and scalable analytical delivery.

If you are a technically accomplished Data Platform Engineer who combines engineering depth, strategic thinking, analytical curiosity and collaboration with a determination to turn engineering excellence into measurable analytical value, we would like to hear from you.

1. Lead the design, build and optimisation of scalable data pipelines and data architecture — ensuring reliable, secure and efficient data flows from source systems through ingestion and transformation to analytical data products.
2. Lead data orchestration, transformation and integration — develop robust, automated ETL/ELT processes covering ingestion, cleansing, transformation, scheduling, dependency management, exception handling and delivery.
3. Design and assure robust analytical data models — translate complex analytical and business requirements into well-engineered, reusable dimensional, Lakehouse, Data Vault and semantic models that improve analytical consistency and performance.
4. Drive automation and analytical productivity — identify manual, repetitive and inefficient processes within Data Engineering, Analytics and the wider department and develop automated solutions that demonstrably release capacity and accelerate analytical delivery.
5. Provide senior technical leadership and engineering assurance — set and champion standards for architecture, code, pipelines, data models, testing, documentation and engineering practices, constructively challenging solutions where greater rigour or efficiency is required.
6. Establish modern engineering and deployment practices — embed Git, CI/CD, automated testing, monitoring, Infrastructure as Code, deployment pipelines and effective change control to create reliable and maintainable production services. The person specification specifically requires significant DevOps, Git and automated deployment experience.
7. Demonstrate measurable benefit from Data Engineering — establish and evidence improvements such as reduced data-processing time, fewer manual interventions, improved pipeline reliability, greater reuse of data assets and transformation logic, faster analytical turnaround and increased analytical capacity.
8. Engineer data quality, reliability and resilience into the platform — implement validation, reconciliation, monitoring, audit, recovery, backfill and exception-management processes so analytical teams can confidently rely on the data they receive.
9. Optimise Microsoft Fabric, Snowflake and Azure environments — improve performance, scalability, resilience and resource utilisation across OneLake, Lakehouse, Data Factory, Spark/Notebooks, Snowflake and associated cloud services.