Big Data Engineering Lead
Valce Talent Solutions ·
- Seniority
- Lead
- Category
- Data engineering
- Experience
- 3+ years
Valce Talent Solutions ·
EXL · Pune, Maharashtra, India
challenger · Sydney CBD, NSW
lgtcp · Pfäffikon, Switzerland
Latitude IT · Sydney, Australia
Leads a team building and maintaining enterprise data pipelines, data products, and canonical data models to support analytics, AI, and reporting, ensuring scalable, reliable, and high-quality data solutions.
Big Data Engineering Lead
Remote
Advanced english
Role Summary
The Data Engineering Lead is responsible for the technical leadership, engineering delivery, and quality of the Data Foundation platform.
This role leads the team responsible for building and maintaining enterprise data pipelines, data products, canonical data models, and data services that enable analytics, AI, integration, and operational reporting. The Data Engineering Lead partners closely with the Service Owner, Architecture Team, and Platform Engineering Lead to deliver scalable, trusted, and maintainable data solutions aligned with enterprise standards and strategic objectives.
The ideal candidate is a hands-on technical leader with deep expertise in data engineering, modern data architectures, distributed processing, and engineering excellence. They are accountable for engineering execution, solution quality, operational reliability, and the development of a high-performing data engineering team.
Required Qualifications
Experience
8+ years of experience in data engineering, software engineering, or enterprise data platform development.
3+ years of technical leadership experience leading engineering teams or large-scale data initiatives.
Proven experience designing and delivering enterprise data platforms, pipelines, and data products.
Experience operating within agile software development environments.
Technical Skills
Advanced SQL and data modeling expertise.
Strong Python and PySpark development experience.
Deep understanding of ETL and ELT architecture patterns.
Experience implementing Master Data Management solutions.
Experience building RESTful APIs and data services.
Familiarity with GraphQL and modern integration approaches.
Understanding of microservices-based architectures.
Experience implementing event-driven and real-time data processing solutions.
Strong understanding of data quality frameworks, metadata management, and data lineage concepts.
Leadership Competencies
Strong technical leadership and coaching capabilities.
Excellent problem-solving and decision-making skills.
Ability to balance delivery commitments with long-term platform sustainability.
Strong collaboration and stakeholder engagement skills.
Excellent written and verbal communication abilities.
Passion for engineering excellence and continuous improvement.
Success Measures
The Data Engineering Lead will be measured on:
Successful delivery of Data Foundation engineering commitments.
Data pipeline reliability, performance, and scalability.
Data quality improvements across enterprise data assets.
Adoption of engineering standards and development practices.
Reduction of technical debt and engineering risk.
Improvement in operational stability and incident reduction.
Engineering team engagement, capability growth, and retention.
Increased reuse of canonical data models, data products, and engineering frameworks.