Sr. Databricks Platform Engineer

The Role

We're engaging a hands-on / Senior Databricks platform engineer/builder for our client - one of global leading strategy-led tech consultancy company. The objective of role is to lead the design and operationalisation of a governed, multi-tenant data platform on Azure Databricks - ingesting and onboarding data from multi external organisations across Databricks, Snowflake, API, and file-based sources.

Design and implement governance from the ground up, and make Unity Catalog and Delta Sharing work at scale in production, contributing to the low level design of the architecture already selected.

What You'll be doing

Unity Catalog governance — design and implement the catalog/schema structure, privilege model, data ownership, and lineage for a platform serving multiple business units and external partners

Delta Sharing — architect the provider/recipient model for external data onboarding; manage refresh strategy, access scoping, and cross-platform sharing (Databricks-to-Databricks, Databricks-to-Snowflake)

Identity & access integration — Microsoft Entra ID integration with Unity Catalog; service principal design; RBAC across environment tiers

Pipeline modernisation — refactor existing Python pipelines into a production-grade Databricks medallion architecture (bronze/silver/gold)

Platform engineering — workspace structure, environment promotion, CI/CD, cost optimisation, and capacity planning

Architecture contribution — this is a live engagement with an existing recommendation in place; we want someone who will challenge and improve it, not just implement it

What you should have:

Must-Have

Hands-on Unity Catalog implementation in a production environment — not workshop-level exposure. You should be able to talk through catalog structure, privilege model, and lineage from real deployments.

Real Delta Sharing experience — provider/recipient architecture, external data onboarding, operational challenges you've actually hit and solved

Strong Python and PySpark, with production pipeline migration experience (not just greenfield builds)

Microsoft Entra ID integration experience — service principals, RBAC, scoped access

Track record of making platform architecture decisions yourself, not just executing someone else's design

7+ years data platform engineering, 3+ years hands-on Azure Databricks

Nice to Have

Databricks Genie evaluation/implementation experience
Azure Data Factory orchestration
CI/CD and DataOps (Azure DevOps, Databricks Asset Bundles)
FinOps / platform cost optimisation
Regulated industry or enterprise governance experience

If you're an experienced builder in applying Unity Catalog & Delta Sharing to external Data integration, and looking to work alongside a top-tier global consultancy on a high visibility platform build / product-grade delivery, we'd like to hear from you.