Data Engineer maintaining and supporting production data pipelines on Databricks and AWS for a large-scale UK government project. Day to day is a mix of L2/L3 production support and data engineering: owning incidents, root cause analysis, fixes and performance troubleshooting using Python, PySpark, SQL, Spark, Delta Lake, S3, IAM and CloudWatch. Permanent hybrid role based in Leeds.
Were looking for strong hands-on experience with Databricks, AWS, Python, PySpark and SQL.
We need experience with Databricks notebooks, scheduled jobs, cluster configuration, driver and executor logs, Spark UI, and performance troubleshooting.
Were looking for experience with Spark SQL, Delta Lake, Parquet, Hive metastore tables, schemas, partitions, and the relationship between table metadata and underlying files.
We need hands-on AWS experience, particularly with S3, IAM and CloudWatch, plus an understanding of role-based access, KMS encryption, and diagnosing data access failures.
Were looking for a working knowledge of Git, code reviews, CI/CD, and controlled production deployments, including testing, rollback and release validation.
Experience with AWS Glue, Lambda, Step Functions, Linux, shell scripts, Terraform, GitLab or Jenkins would be advantageous.
Responsibilities:
We need you to run and maintain production data pipelines on Databricks and AWS.
Youll keep scheduled data processing reliable, resolve incidents, apply fixes, and improve data service performance.
Youll provide a combination of L2/L3 production support and data engineering.
Youll support and maintain production data pipelines, including incident investigation, safe recovery, root cause analysis and permanent remediation.
Youll own incidents from investigation through recovery and closure, diagnosing issues in Python, PySpark, SQL, Databricks jobs and AWS integrations.
Youll provide clear progress updates and escalate issues in a timely manner.
Technologies:
AWS
AWS Glue
Lambda
CI/CD
CloudWatch
Databricks
Git
GitLab
Hive
IAM
Support
Jenkins
Linux
Python
PySpark
SQL
Spark
Terraform
UX UI Design
Cloud
More:
Were hiring a Data Engineer for a permanent, full-time role on a large-scale Government project as part of a leading global IT transformation. The position is based in Leeds with a hybrid model, primarily working from home and typically travelling to our Leeds office once or twice a month. The expected start is ASAP in October or November 2026. We offer a salary of £60,000–£65,000, depending on experience, remote/hybrid working, a pension, 22 days holiday plus bank holidays (rising to 25 days), death-in-service cover and a healthcare plan.