Data Engineer
Source Point Strategic Sourcing · The Rocks, Sydney ·
- Employment
- Contract
- Category
- Data engineering
8-week contract (strong extension chance) for an experienced Databricks Data Engineer in Sydney to complete a Databricks Proof of Technology and stand up a simulated MVP environment, supporting the Global Optimisation Platform project's optimisation and bidding solution.
Contract position (8 weeks) Strong chance of extension
Our client is seeking an experienced Databricks Data Engineer for an 8-week contract to bring the Databricks Proof of Technology (PoT) to completion and stand up a simulated MVP environment. This role supports the Global Optimisation Platform Project, which is creating an innovative optimisation and bidding solution.
About the role:- Be part of a core Data Engineering team, this engagement adds the specialist Databricks capacity to close that gap across the team.
- Contributing direct Databricks engineering effort within the existing team, with the goal of clearing the PoT gate on schedule and handing Quant Research a workable simulated MVP.
- Stand up the core Databricks environment for the simulated MVP, with security built in through Unity Catalog, access controls, and compute policies.
- Create simulated asset telemetry and align the market data, giving the Quant Research team a complete, end-to-end modelling pipeline
- Use real effort from reference pipelines as a baseline to refine the delivery-phase estimate for the data ingestion and integration workstream.
- Over five years of data engineering experience, including a minimum of two years building and running Azure Databricks platforms in production.
- Hands‑on experience designing and governing Unity Catalog, covering catalog structure, access permissions, external locations, storage credentials and system tables.
- Deep knowledge of Azure Databricks security, including VNet injection, Private Link, Entra ID with SCIM provisioning, service principals and Key Vault-backed secret scopes.
- Proven experience building real‑time pipelines with Structured Streaming on Event Hubs or Kafka, along with Delta Lake and medallion architecture.
- Practical experience deploying with Databricks Asset Bundles and running CI/CD pipelines in Azure DevOps or GitHub, plus working with Git folders.
- Advanced Python skills, including building wheel packages and managing library dependencies in privately networked environments.