A 12-month, outside-IR35 contract at a London private equity firm to build its first Databricks environment from scratch: designing a bronze/silver/gold medallion architecture on Azure, integrating portfolio-administration data platforms, and delivering the firm's golden data source ahead of a flagship fund launch. Core tech: Databricks, Delta Lake, PySpark, Azure, Python, SQL.
Data Engineer – Databricks, Azure, Python, SQL - 12 Month Contract - Outside IR35 - Private Equity - Up to £700 p/d (dependent on experience)
A well-established private equity firm with close to two decades' track record and preparing to launch its largest flagship fund to date is hiring a Data Engineer to build its first Databricks environment from the ground up.
This is a rare opportunity to own the technical delivery of a firm's core data platform end-to-end: partnering directly with stakeholders across Finance, Portfolio Operations, General Partners, and Compliance to unlock the data quality and visibility the business needs ahead of its next fund launch.
The Role
You’ll design and build the firm's first Databricks instance, standing up a medallion architecture (bronze/silver/gold) on Azure and integrating it with the firm's existing portfolio administration and data intelligence platforms. The environment you build will become the firm's golden data source.
What We’re Looking For
Proven, hands-on delivery of Databricks / Delta Lake builds - ACID transactions, time travel, schema evolution, and performance tuning (OPTIMIZE, Z-ordering, VACUUM)
Strong command of medallion architecture and PySpark, lazy evaluation, partitioning, broadcast joins, and diagnosing partition skew
Solid Azure data platform experience; familiarity with Azure Analysis Services or a comparable semantic layer is a plus
Confidence engaging senior, non-technical stakeholders and translating between technical and business needs
Experience working within Private Equity is highly desirable - wider Financial Services is accepted
*Suitable candidates will be contacted within 24 hours