Senior Data Engineer

Overview

In this contract role you will lead and shape the data engineering approach for a large, global organisation. You’ll quickly grasp the existing data estate, drive high-quality engineering practices, and raise performance and efficiency across the platform. You’ll collaborate with data engineering, analytics, data science and platform teams to standardise patterns and governance while delivering impactful, scalable data solutions. This is a hands-on senior position with scope to influence architecture and delivery at scale.

Responsibilities

• Design and improve engineering patterns across Azure Databricks and Azure Data Factory
• Drive performance, caching, indexing, partitioning and cost optimisation
• Review code and act as a technical quality gate
• Improve DevOps practices across CI/CD, environment promotions, branching and release management
• Identify opportunities to automate repetitive engineering processes
• Work with advanced SQL across established patterns
• Support data quality, governance and orchestration
• Collaborate closely with data engineering, analytics, data science and platform teams
• Help establish consistent engineering standards across the wider service

Key requirements

• Azure Databricks experience
• Azure Data Factory experience
• Spark / PySpark
• Advanced SQL and performance optimisation
• CI/CD and Azure DevOps
• Delta Lake and medallion architecture
• Unity Catalog
• Data governance and orchestration
• API-based data ingestion
• Experience with high-volume operational, IoT or telemetry data (valued)

• autonomous working style
• ability to challenge existing approaches
• collaboration across cross-functional teams
• Azure Databricks
• Azure Data Factory
• Spark / PySpark