Location: Remote, UK, with travel to London 1-2 times per quarter
Duration: 6 months initially
Start Date: October 2026
We are looking for an experienced Data Engineer to support a major Trading Automation programme.
You will design and build modern data pipelines and platform capabilities using Databricks and Azure, providing the trusted and scalable data foundations required for AI-driven trading automation.
Key Responsibilities
- Design, build and maintain scalable data pipelines using Databricks and Azure.
- Develop ingestion and transformation processes for structured and unstructured data.
- Build reliable data products to support trading workflows and Generative AI applications.
- Implement data architectures using Delta Lake and lakehouse principles.
- Apply data quality, governance, lineage and security controls.
- Optimise pipelines and datasets for performance, reliability and scalability.
- Collaborate with AI engineers, trading teams and business stakeholders.
- Support automated deployment, monitoring and operational management of data workloads.
- Advanced experience with Databricks, Apache Spark and PySpark.
- Strong knowledge of Azure data services.
- Experience building batch and streaming data pipelines.
- Strong Python and SQL skills.
- Experience with Delta Lake and lakehouse architecture.
- Knowledge of data modelling, testing, quality and governance.
- Experience with CI/CD, Git and Infrastructure as Code.
- Comfortable working independently in a remote, client-facing environment.
- Experience within Energy Trading, commodities, Capital Markets or Financial Services.
- Experience supporting AI, machine learning or Generative AI workloads.
- Knowledge of Databricks Unity Catalog and MLflow.