Databricks Data Engineer | 12 Month+ B2B Contract | Fully Remote | €500 per day
I'm partnering with a global technology consultancy on a large data transformation programme for a well-established organisation in the insurance sector. They need an experienced, hands‑on Databricks Data Engineer to join the delivery team on a long‑term contract.
The Role
The end client is moving away from legacy data warehousing and scattered on‑premise systems to a modern, unified Lakehouse platform built on Databricks. You will sit at the centre of this transformation. You'll design and build scalable data pipelines, migrate legacy workloads, and help set the engineering standards the wider platform will be built on.
You’ll work closely with architects, analysts and business stakeholders across the underwriting, claims, policy and finance data domains.
This is a hands‑on engineering role. You should be comfortable writing production‑grade PySpark and SQL every day.
The Project
- Migrating legacy data warehouse and ETL workloads (SQL Server, SSIS, Informatica or similar) onto Databricks
- Designing and building a Medallion (Bronze, Silver, Gold) Lakehouse architecture on Delta Lake
- Rolling out Unity Catalog for central governance, lineage and access control
- Delivering curated, trusted data products for actuarial, risk, claims and regulatory reporting
- Introducing CI/CD, automated testing and infrastructure as code across the platform
- Enabling downstream BI, analytics and future AI/ML use cases
Key Responsibilities
- Design, build and maintain scalable batch and streaming pipelines on Databricks using PySpark, Spark SQL and Delta Lake
- Migrate legacy ETL processes and data models onto the Lakehouse, keeping data accurate and reconciled throughout
- Implement Medallion architecture patterns and build reusable ingestion and transformation frameworks
- Orchestrate workflows using Databricks Workflows, Delta Live Tables, Azure Data Factory or Airflow
- Work within Unity Catalog to enforce governance, security and lineage
- Optimise Spark jobs and cluster configurations for performance and cost
- Build data quality checks, monitoring and alerting into pipelines
- Contribute to CI/CD and infrastructure as code using Databricks Asset Bundles, Terraform, Azure DevOps or GitHub Actions
- Turn business requirements into robust data solutions alongside architects, analysts and insurance stakeholders
- Document solutions and contribute to engineering best practice across the programme
- 5+ years of commercial experience as a Data Engineer
- 2+ years of hands‑on Databricks delivery in production
- Advanced Python and PySpark
- Advanced SQL and data modelling (dimensional, Data Vault or similar)
- Delta Lake and Lakehouse / Medallion architecture
- Cloud platform experience, ideally Azure (AWS also considered)
- Building and orchestrating ETL/ELT pipelines at scale
- Git-based workflows and CI/CD for data engineering
- Experience on a data migration or transformation programme
- Confident communicator, comfortable working on your own in a remote team
- Unity Catalog implementation or migration
- Delta Live Tables and Structured Streaming
- Databricks Asset Bundles and Terraform
- Insurance, financial services or other regulated industry experience
- Insurance data domains such as policy, claims, underwriting and actuarial
- Exposure to regulatory reporting (Solvency II, IFRS 17)
- dbt, Azure Data Factory or Airflow
- Data quality frameworks such as Great Expectations or DLT expectations
- Previous consultancy or client‑facing delivery experience
Certification
The ideal candidate holds the Databricks Certified Data Engineer Professional certification, and certified candidates will be prioritised. Associate‑certified engineers with strong hands‑on delivery experience are also encouraged to apply.
Why Apply?
- 12 month+ initial contract with a strong chance of extension
- Fully remote working
- €500 per day on a B2B basis
- Work on a near‑greenfield build on a modern Databricks Lakehouse platform
- Real influence over engineering standards and architecture as the platform takes shape
- A high‑profile programme with a global consultancy and a major insurance client
Please note:
this role is not suited to purely BI or reporting profiles, or to architects and delivery leads who no longer code day to day. Candidates must be able to work on a B2B contract basis.