Lead Data Engineer

Job description:

Our client is a specialist technical consultancy delivering bespoke technology and data solutions to solve complex operational challenges. Due to continued growth, they are looking for an experienced Data Engineer to join their growing engineering team.

This is an exciting opportunity to work on high-priority projects supporting customers across Central Government, Defence, National Security, and Critical National Infrastructure .

As a Data Engineer, you will work alongside experienced technical professionals to design, build, and maintain robust and scalable data pipelines that ensure data availability, accuracy, security, and performance.

You will work closely with customers and internal stakeholders throughout the full project lifecycle, while also contributing to data engineering strategy, technical roadmaps, research, and the adoption of emerging technologies.

Location

The successful candidate can work from one of the organisation's UK offices, including Bristol, Cheltenham, or London , or work remotely from home.

The role will involve occasional travel to customer sites and other company locations when required.

Key Responsibilities

• Interpret, validate, and analyse data requirements across complex environments.
• Gather, profile, and analyse large-scale structured datasets to ensure data accuracy, quality, and completeness.
• Design and implement robust ETL/ELT frameworks for ingesting, transforming, validating, normalising, and cleansing data.
• Develop and maintain scalable data pipelines to support analytics and operational decision-making.
• Implement data quality controls and prepare datasets for reporting and visualisation.
• Design and maintain data models to support analytical and operational requirements.
• Manage and optimise cloud and data storage solutions, including Amazon S3, Azure Blob Storage, BigQuery, and Snowflake .
• Contribute to data management standards, policies, governance frameworks, and best practices.
• Support data synthesis and anonymisation activities to ensure compliance, security, and appropriate data handling.
• Develop robust data models within secure and highly controlled environments.
• Monitor and optimise data workflows to improve performance, reliability, scalability, and cost efficiency.
• Evaluate emerging data technologies and identify opportunities to improve engineering capabilities.
• Contribute to data analytics and visualisation strategies.
• Present technical findings and recommendations to customers and stakeholders.
• Work collaboratively with technical and non-technical teams throughout the project lifecycle.

About You

We are looking for an experienced Data Engineer who enjoys solving complex technical problems and working collaboratively with customers and colleagues.

You should have a strong understanding of modern data engineering principles, with experience working in complex or highly regulated environments.

The ideal candidate will have:

• Proven experience working as a Data Engineer within complex technical environments.
• Strong knowledge of data management principles, including data modelling, integration, quality, and governance.
• Strong commercial experience with Snowflake and Databricks essential.
• Experience assessing existing data management systems and recommending improvements.
• Experience with modern data architectures, including cloud-based and distributed systems.
• Strong proficiency in SQL and Python .
• Experience with data engineering and pipeline technologies such as Apache Airflow and Spark .
• Experience working with cloud platforms, particularly AWS and Azure .
• Knowledge of big data technologies and distributed data processing.
• Understanding of data analytics and visualisation.
• Awareness of data standards, interoperability, data governance, and data sharing.
• Experience working with sensitive or security-conscious data environments would be highly beneficial.
• Strong analytical, problem-solving, communication, and stakeholder-management skills.

Security Clearance

Due to the nature of the projects and industries supported by our client, the successful candidate will need to be eligible to obtain high-level UK security clearance .

To meet the current eligibility requirements, candidates must:

• Hold British citizenship ; and
• Have been permanently resident in the UK for the previous five years .

Candidates who already hold relevant UK security clearance are particularly encouraged to apply.