Job Description
About the role
As a Data Engineer, you’ll bring a well-rounded understanding of data-driven practices to support senior team members and stakeholders in enhancing regulatory processes. From uncovering insights through data profiling and analytics to building robust models and engineering solutions, you’ll play a key role in how data is governed, interpreted, and transformed into actionable intelligence.
Key responsibilities of this role are:
- Collaborate within a high-performing Scrum team to deliver BAU work and Project Roadmap milestones that support the Data Science and Analytics Strategy. This includes interpreting requirements, estimating effort, and raising risks and impediments early.
- Interpret business requirements, estimate workloads, prioritise tasks, and manage time in the delivery of quality solutions to meet deadlines.
- Design, build, and maintain scalable data pipelines in Databricks using Python, PySpark, and Spark SQL, primarily within our metadata-driven ingestion and transformation framework.
- Develop and operate Databricks notebooks and jobs on classic compute. Optimize performance and cost through cluster configuration, cluster policies, and Spark and Delta Lake tuning.
- Extend and improve the framework's configuration, patterns, and reusable components so that new data sources can be onboarded quickly and consistently.
- Contribute to the platform's evolution toward Lakeflow declarative pipelines and modern Databricks capabilities.
- Deliver changes through modern CI/CD practices, including source control, automated testing, peer code review, and CAB-approved releases.
- Support, maintain, and upgrade existing workflows, deployment pipelines, and environments across the full application lifecycle.
- Implement and manage data security with Unity Catalog, including catalog, schema, and table permissions, row filters, column masks, service principals, and secrets management.
- Identify, classify, and protect sensitive and personal (PII) data. Apply tagging, masking, and access controls in line with privacy obligations and organizational policy.
- Participate in a rotating incident management roster. This includes troubleshooting and resolving issues, performing root cause analyses, configuring monitoring and alerting, and writing troubleshooting guides.
- Proactively remediate and document operational incidents, project risks, and impediments where possible or otherwise surface these items as appropriate to project managers and senior team members.
NHVR’s Data & Technology teams are currently undertaking growth and considerable transformation, and this is an exciting time to join the organisation.
To learn more, the position description can be found on the NHVR Careers site - https://careers.nhvr.gov.au/jobs/search
About You
You’ll have at least four years of data engineering experience, including significant hands-on work with Databricks or equivalent cloud data platform. A bachelor's degree in a relevant field is desirable. You’ll also be experienced with Agile Scrum, the SDLC, and data lifecycle processes.
Your toolkit for success:
- Hands-on Databricks experience, including notebooks, Jobs/Workflows, cluster configuration on classic compute, and Spark performance tuning.
- Strong Python and PySpark data pipeline development skills, with advanced SQL (Spark SQL and T-SQL or similar).
- Practical experience with Delta Lake, including MERGE and upserts, schema evolution, OPTIMIZE and VACUUM, and partitioning or clustering strategies.
- Experience building or working within metadata-driven or configuration-driven pipeline frameworks.
- Working knowledge of Unity Catalog security and governance, including the securable model, grants and ownership, external locations, storage credentials, and lineage.
- Experience identifying and handling sensitive and PII data, including classification, masking, and least-privilege access.
- Solid data modelling knowledge (relational, dimensional, and medallion architecture) and an understanding of RDBMS concepts.
- Software engineering discipline, including modular and reusable code, unit testing, Git-based workflows, CI/CD, and debugging.
- Experience with Azure data services such as ADLS Gen2, and Key Vault.
- Understanding of batch and streaming data concepts, structured and unstructured data, and data governance
Ideally you will have:
- Experience with Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables), Lakeflow Connect, Auto Loader, and data quality expectations.
- Relevant certifications including Professional or Associate Databricks Data Engineer, and Azure Certification.
- Experience with data visualization tools such as Microsoft PowerBI, Tableau.
- Familiarity with supporting machine learning workloads, such as MLflow.
Why join us?
The National Heavy Vehicle Regulator (NHVR) commenced operations in 2013 as the national entity responsible for regulating all vehicles in Australia over 4.5 tonnes gross vehicle mass. The NHVR minimises the compliance burden on the heavy vehicle transport industry, reduces duplication and inconsistences across state and territory borders, and ultimately for the heavy vehicle business with government in Australia.
We are headquartered in Brisbane and employ more than 1000 people across the ACT, New South Wales, Queensland, South Australia, Tasmania and Victoria.
We offer
- Salary package of $135,995.90 + superannuation
- Monthly RDO and paid leave over Christmas closure.
- Hybrid working environment – up to 2 days from home.
- Up to15.25% superannuation through a co-contribution scheme
- Leave loading 17.5%
- Wellbeing programs – discounted fitness through Fitness Passport, Discount health insurance, flu vaccine, financial programs including discounted banking products, finance information sessions.
- Salary packaging options including novated lease.
- Discounted IT products including Microsoft and free family password security plan.
- Ongoing training and professional development support
- Professional membership
Successful applicants will be required to complete standard pre-employment checks including a national police and work rights check.
We are committed to being an inclusive, diverse and flexible workplace where differences are valued.We welcome applications from people of all backgrounds, experience and abilities.
#J-18808-Ljbffr