Job DescriptionWe are looking for an experienced Azure Databricks Technical Consultant to design and deliver secure, scalable data and analytics solutions within a complex, highly regulated environment.
The consultant will have a strong background in Azure Databricks and modern data engineering, working across large-scale data ingestion, transformation, analytics and AI workloads. The role will involve integrating structured and unstructured data from multiple sources, including telemetry, operational datasets, OSINT and SOCMINT.
Key Responsibilities
- Design, build and maintain scalable data pipelines using Azure Databricks, Spark/PySpark and Azure Data Lake Storage
- Ingest, transform and process large volumes of structured, semi-structured and unstructured data from multiple sources
- Develop secure Lakehouse architectures and reusable data engineering frameworks within Azure
- Design and optimise data models, analytical datasets and feature stores supporting advanced analytics and AI use cases
- Integrate Databricks with Azure Machine Learning, Azure OpenAI and wider Azure data services
- Support AI/ML and agentic AI workloads including RAG, LLMs and autonomous agent architectures
- Develop analytical solutions incorporating machine learning, NLP, graph analytics, clustering, entity resolution and pattern analysis
- Implement MLOps, AIOps and SecML processes covering model monitoring, governance, audit logging, explainability, drift detection and rollback
- Work with graph technologies including Neo4j, Azure Cosmos DB Gremlin API and Microsoft Graph Data Connect
- Build reporting, dashboards and analytical outputs using Power BI and associated Microsoft technologies
- Ensure data platforms meet stringent requirements around security, governance, Zero Trust, compliance and Responsible AI
- Conduct functional, system, performance and non-functional testing across data and AI solutions
- Produce solution designs, technical documentation, implementation plans and operational support documentation
- Collaborate with data scientists, cloud architects, software engineers, analysts and business stakeholders within Agile delivery teams
Essential Skills & Experience
- Strong hands-on experience with Azure Databricks
- Excellent Python/PySpark and SQL data engineering skills
- Strong experience designing and developing ETL/ELT and large-scale data pipelines
- Experience with Apache Spark and Databricks Lakehouse architectures
- Strong knowledge of Azure Data Lake Storage (ADLS)
- Experience integrating Databricks with wider Azure data and analytics services
- Understanding of data modelling, data quality, governance and performance optimisation
- Experience working with Azure Machine Learning and/or Azure OpenAI
- Strong understanding of cloud security, , access controls and Zero Trust principles
- Experience working within Agile/Scrum delivery environments
Desirable Experience
- Agentic AI, autonomous agents, RAG and Large Models
- MLOps and productionisation of machine learning workloads
- Graph databases and graph analytics including Neo4j or Cosmos DB Gremlin API
- Power BI and analytical visualisation
- Feature stores and ML data pipelines
- NLP, clustering, entity resolution and causal inference
- Responsible AI and AI governance frameworks
- Previous experience within Defence, Government, Security or similarly regulated environments
This is an Inside IR35 PAYE contract role so you must be in the UK to be considered for this role and you must hold ACTIVE SC CLEARANCE