Data Engineering Manager - Databricks

We are implementing a modern data and analytics platform to strengthen data quality, governance, automation, and insight generation across our insurance and broader enterprise reporting functions. We operate in a regulated, high-sensitivity environment and are building sustainable internal capability to operate and extend our data solutions over time.

This is a hands-on technical leadership role responsible for ensuring our Databricks platform is implemented the right way. You will bridge engineering, architecture, data governance, and delivery leadership, working across actuarial, IT, security, and implementation partners.

This position requires you to be an Australian Citizen & living in Sydney *

The Responsibilities

• Lead day-to-day technical coordination across actuarial, business, IT, and vendor stakeholders.
• Contribute directly to Databricks implementation: pipelines, notebooks, Unity Catalog, orchestration, and performance optimisation.
• Support migration of priority data assets, processing logic, and analytics code into Databricks.
• Establish reusable patterns that support maintainability, governance, auditability, and performance.
• Implement governance practices: version control, code review, CI/CD, access control, data quality checks, and documentation standards.
• Define and enforce standards for open source, libraries, and third-party components including security and licensing compliance.
• Review and assure key technical artefacts from the implementation partner; manage technical debt and BAU readiness.
• Build internal capability through coaching, paired delivery, and knowledge transfer to reduce reliance on external vendors.

The Requirements

• 8+ years hands-on data engineering / platform engineering experience delivering production-grade solutions end-to-end.
• Strong Databricks / Spark delivery in production: pipeline build, orchestration, performance tuning, and operationalisation.
• Proven data governance implementation: RBAC, audit logging, lineage, data quality gates, and change control.
• Experience in a regulated or high-sensitivity data environment with strong security discipline.
• Demonstrated ability to lead co-delivery with vendors: set acceptance criteria, review designs and code, manage technical debt.
• Expert SQL and strong Python (including PySpark); strong software engineering discipline including Git, PR/code review, and CI/CD.

Desirable

• Databricks Certified Data Engineer