Lead Data Engineer (Databricks)

Overview

In this role you will lead the design and delivery of scalable Databricks-based data platforms for near real-time river water quality monitoring. You will drive data pipelines, governance, and collaboration with scientists and analysts to produce trusted datasets for regulators and operations. You’ll shape data architecture and mentor engineers, delivering impactful environmental insights through modern data engineering practices. This is a hands-on leadership role that combines technical ownership with cross-functional collaboration.

Pay / Benefits

• Up to 10.5% annual bonus
• Pension up to 21% combined
• Private healthcare
• Share buy scheme
• Wellbeing benefits and support programmes

Responsibilities

• Design, build, and optimize scalable data pipelines using Databricks, Spark and Delta Lake
• Implement medallion architecture (bronze/silver/gold) for data quality and lineage
• Optimize Spark jobs to meet near real-time SLAs
• Develop streaming pipelines with Structured Streaming and Delta Live Tables
• Enforce data governance, schema validation, and ACID compliance
• Automate pipelines with Databricks Jobs, APIs or cloud schedulers
• Collaborate with data scientists, analysts and environmental specialists to deliver high-quality datasets
• Monitor and improve production pipelines for reliability, performance, and cost efficiency
• Promote CI/CD, testing, version control and documentation best practices
• Act as a technical leader and mentor to raise standards across the data engineering community

Key requirements

• Deep expertise in Databricks (Spark / PySpark, Delta Lake, Unity Catalog, Workflows)
• Strong experience building ETL/ELT pipelines (batch and streaming)
• Advanced Python and SQL skills
• Hands-on experience with cloud platforms (Azure, AWS or GCP)
• Solid understanding of data modelling and modern architectures
• Strong knowledge of data governance, security and compliance
• Bachelor’s degree in Computer Science, Data Engineering or related discipline
• 5+ years’ data engineering experience, including 3+ years with Databricks
• Proven ability to lead technically and mentor engineers

• collaboration
• mentoring
• problem-solving
• Databricks (Spark / PySpark, Delta Lake, Unity Catalog, Workflows)
• ETL/ELT pipelines (batch and streaming)
• Python