Lead Azure Databricks Platform Engineer / Architect
Hands-on Platform Engineering | Serverless | FinOps | POSIT/RStudio Migration
6 Month contract
Inside IR35 - 500 a day
London/Hybrid
Role Purpose
We are seeking a highly experienced, hands-on Azure Databricks Platform Engineer / Architect to enhance and optimise an enterprise Data Platform. The role combines architecture with direct implementation: the successful candidate must be able to configure, develop, troubleshoot and optimise Azure Databricks rather than operate only at design or governance level.
The role is centred on three outcomes: enabling and optimising Databricks Serverless, strengthening FinOps and platform controls, and enhancing the Databricks Discovery Zone to support workloads currently delivered through POSIT/RStudio.
Key Responsibilities
1. Databricks Serverless Enablement and Optimisation
- Assess existing workloads and determine suitability for Serverless, classic, job or interactive compute based on duration, utilisation, SLA, concurrency, performance and cost.
- Enable and configure Serverless for appropriate jobs, SQL workloads, notebooks, analytical processing and data pipelines.
- Establish workload-placement guidance, including when Serverless is not economical for predictable, heavy or continuously running workloads.
- Implement compute policies, autoscaling, quotas, budget controls and operational guardrails.
- Measure cost and performance outcomes, identify idle or oversized compute, and recommend optimisation actions.
2. FinOps and Enterprise Platform Controls
- Define and embed a practical FinOps operating model covering ownership, accountability, projects, environments, teams, applications and cost centres.
- Implement mandatory tagging and integrate validation into CI/CD so non-compliant resources are prevented from being provisioned.
- Provide granular cost attribution by workspace, project, application, workload, job and team/user where technically appropriate.
- Implement budget policies, thresholds, proactive alerts and usage reporting to prevent uncontrolled spend.
- Use platform usage and billing data to identify idle compute, inefficient workloads, unnecessary storage/data movement and cost anomalies.
3. Databricks Discovery Zone and POSIT/RStudio Migration
- Enhance the Databricks Discovery Zone to support migration from POSIT/RStudio
- Enable application deployment, secure API integrations, external data ingestion, LLM integration, scheduling, BI connectivity, local IDE-based development and operational reporting.
- Define reusable onboarding and migration patterns that reduce technology sprawl while improving security, supportability and delivery speed.
4. Data Engineering and Integration
- Design and build reliable ingestion and transformation pipelines using Python, PySpark, SQL and Delta Lake.
- Implement full and incremental ingestion, CDC where appropriate, schema evolution, reconciliation, error handling and data quality controls.
- Design reusable integration patterns for REST APIs, SaaS platforms, databases, files, object storage, document repositories, enterprise applications and public/external data providers.
- Implement secure authentication and credential handling for external and internal integrations.
- Build end-to-end data flows from source through governed ingestion and curated layers to BI, ML or application consumption.
Required Hands-on Technical Skills
Deep hands-on Azure Databricks implementation and troubleshooting
Databricks Serverless and compute/workload optimisation
Azure identity, networking, security, secrets, monitoring and private connectivity
Databricks SQL, Delta Lake and performance optimisation
Python, PySpark and SQL
Jobs/workflows, incremental processing, CDC and data quality
REST/API and external data integration patterns
FinOps, cost attribution, tagging, budgets, monitoring and operational support
Experience and Candidate Profile
- Significant experience delivering enterprise Azure Databricks platforms in production environments.
- Demonstrable ability to move between architecture, implementation, debugging and optimisation without depending entirely on specialist engineering teams.
- Strong understanding of platform security, data governance, operational support and controlled delivery in regulated or complex enterprises.
- Experience working collaboratively with data engineers, data scientists, architects, security teams, platform teams and business stakeholders.
- Clear communication skills and the ability to document standards, patterns, decisions and operational guidance.
Highly Desirable but not Mandatory
- POSIT/RStudio migration or consolidation experience.
- Migration of analytical/data science workloads (convert and migrate R development/Libraries to Databricks).
- AI/ML, LLM integration, model lifecycle, RAG/vector retrieval or model-serving experience.
- Large-scale enterprise platform transformation and regulated-industry experience.
- Strong cost optimisation and FinOps delivery experience across Azure and Databricks.