Data Architecture Melbourne Australia Regular / Full time
Description
About the Role
Vericent is partnering with a leading global technology consulting organisation to recruit an experienced Data Engineer. This is an exciting opportunity to work on enterprise-scale data initiatives, building modern data pipelines and cloud-based data solutions that enable analytics, reporting, and AI-driven outcomes.
In this role, you'll collaborate with cross-functional teams, including data scientists, software engineers, product owners, and business stakeholders, to design, develop, and optimise scalable data integration and transformation solutions. You'll also play a key role in ensuring data quality, reliability, security, and performance across the entire data lifecycle.
Key Responsibilities
- Design, build and maintain batch and near-real-time data pipelines.
- Develop ingestion and transformation processes for structured and semi-structured data.
- Build reusable data components and curated datasets for reporting, analytics and machine learning.
- Develop solutions using SQL, Python, dbt, PySpark or equivalent technologies.
- Integrate data from enterprise applications, APIs, event streams and external sources.
- Optimise data pipelines for reliability, performance and cost.
- Support migration and modernisation of legacy data processes.
Data Modelling and Quality
- Develop logical and physical data models aligned to business requirements.
- Build dimensional, relational and analytical data structures.
- Implement data-quality rules, reconciliations and validation controls.
- Investigate data defects and coordinate remediation with source-system and product teams.
- Maintain data definitions, lineage, metadata and technical documentation.
- Apply appropriate handling for sensitive, customer and commercially restricted data.
Engineering and DevOps Practices
- Develop maintainable, tested and version-controlled data solutions.
- Implement automated testing across ingestion, transformation and data-quality processes.
- Support CI/CD pipelines and controlled deployment across development, test and production.
- Participate in code reviews and apply engineering standards.
- Implement monitoring, alerting and operational logging.
- Reduce manual deployment and support activities through automation.
Production Support and Operations
- Monitor production pipelines, schedules and data-processing workloads.
- Investigate and resolve failed jobs, delayed data and data-quality incidents.
- Participate in incident, problem and change-management processes.
- Perform root-cause analysis and implement permanent remediation.
- Maintain operational runbooks, support procedures and recovery steps.
- Work with Platform Operations to ensure data products meet availability and service-level requirements.
Collaboration and Delivery
- Work with product owners and stakeholders to define data requirements and acceptance criteria.
- Collaborate with data scientists to provide model-ready datasets and features.
- Work with software engineers to support application and API data requirements.
- Contribute to technical design, estimation and delivery planning.
- Communicate data risks, dependencies and design decisions clearly.
Required Skills and Experience
- Advanced SQL skills and strong understanding of data structures and query performance.
- Experience with at least two of the following:
- Snowflake
- dbt
- Databricks and PySpark
- Experience building automated data-ingestion and transformation pipelines.
- Strong understanding of data modelling, warehousing and analytical data patterns.
- Experience implementing data-quality controls and reconciliation processes.
- Experience with Git, code reviews and CI/CD practices.
- Understanding of cloud security, access control and data-protection requirements.
- Experience supporting production data pipelines and resolving operational incidents.
- Strong technical documentation and stakeholder-communication skills.
- Tertiary qualification in computer science, engineering, information systems or a related discipline, or equivalent practical experience.
Key Deliverables and Success Measures
- Reliable and scalable data pipelines and data products.
- Achievement of agreed data availability and processing service levels.
- Reduction in pipeline failures and recurring data-quality issues.
- Accurate, traceable and well-documented datasets.
- Automated testing, deployment, monitoring and recovery processes.
- Improved pipeline performance and cloud-cost efficiency.
- Compliance with security, data-governance and engineering standards.
- Positive engagement with product, platform, analytics and business stakeholders.