Data Engineering Lead

Support a transforming, AWS-based Data Product Platform that runs ML workloads and enables data scientists to self-serve. As the Data Engineering Lead, you will lead and coach a team of data engineers, provide technical leadership, and contribute ideas and solution design that align with the client's required architecture practice and governance. While the platform is owned by the client, you are trusted to shape engineering direction, uplift the team, and ensure quality delivery. This role also offers the opportunity to gain exposure to emerging AI and GenAI initiatives - a great way to grow into the next wave of data and AI work.

Responsibilities:

• Lead, coach and mentor a team of data engineers, growing their technical skills and delivery capability.
• Provide technical leadership and hands- on guidance, setting engineering standards, reviewing designs and code, and driving quality across the team.
• Contribute solution ideas and detailed design that align with the client's required architecture practice, patterns and governance.
• Present and articulate technical solutions to client stakeholders and internal teams, clearly explaining design decisions and trade-offs.
• Guide the team in building and optimising Spark on EMR pipelines for batch and streaming data processing.
• Design and guide the handling of streaming API endpoints - real-time data ingestion, consumption and serving through APIs.
• Enable data scientists to self-serve through curated data products and governed access, advising on the best delivery approach.
• Champion good engineering practice around MLOps - supporting model deployment, monitoring and lifecycle management for ML models on the platform.
• Support delivery of lightweight front-end and service components using React and Node.js in a BFF (Backend-for-Frontend) structure.
• Partner with client stakeholders, data scientists and platform teams to translate requirements into robust, secure and cost-effective solutions.
• Promote good practice in data quality, integrity, lineage and security across the data lifecycle.
• Opportunity to gain exposure to AI / GenAI initiatives and contribute ideas as they emerge.

Requirement:

• Bachelor's degree in Computer Science, Engineering, Data Science or a related technical field, or equivalent industrial experience.
• 6+ years of data engineering experience, with a proven track record leading and coaching engineering teams.
• Strong technical leadership skills - able to mentor engineers, set standards, and review designs and code.
• Ability to present and articulate technical solutions clearly to both technical and non-technical stakeholders.
• Hands- on expertise in Apache Spark on AWS EMR (key skill) - pipeline design, performance tuning and cost optimisation.
• Experience designing and handling streaming API endpoints - real-time data ingestion, consumption and API- based serving.
• Solid AWS data platform experience: S3, Glue, Lake Formation, Athena, Redshift, Step Functions and Lambda.
• Proficiency in Python; a strong secondary skill set beyond Python is preferred - React / Node.js in a BFF structure.
• Ability to contribute solution ideas and design within a client's required architecture practice, patterns and governance.
• Understanding of MLOps concepts and experience enabling data scientist self- service.
• Streaming and messaging experience (e.g. Kafka, Spark Streaming) and strong SQL / data modelling skills.
• Interest in or exposure to AI / GenAI initiatives is a plus (not the primary focus of the role).
• Excellent communication, stakeholder management and coaching skills.
• Fluent in Cantonese and English; Mandarin is a plus.
• Previous IT experience in a consulting firm or services vendor is preferred.

Researching careers? Find all the information and tips you need on career advice.