Lead Data Engineer

Overview

In this senior, hands-on role you will lead the design, build, and operation of production-grade data platforms and AI-enabled solutions within a large enterprise. You’ll act as a technical anchor for data engineering and AI initiatives, guiding delivery across complex, cloud-based environments. The role emphasizes robust production systems, governance, and operational support to enable safe, scalable data and AI capabilities. You’ll work with cross-functional teams to translate problems into scalable data/AI solutions and shape platform strategy.

Responsibilities

• Design, build, and enhance enterprise-scale data platforms, pipelines, and services
• Lead end-to-end data engineering work from problem definition through deployment and ops
• Develop cloud-native data platforms supporting analytics, ML, and AI workloads
• Create robust data pipelines for traditional analytics and AI/ML use cases
• Translate operational problems into scalable data/AI solutions with stakeholders
• Design data models, integration patterns, and storage structures for maintainability
• Establish engineering frameworks for AI-led development including CI/CD, governance, and deployment standards
• Operationalise ML/AI solutions into production environments
• Improve data onboarding, interoperability, and data sharing across teams
• Evaluate and adopt new tools/tech where value-added
• Build robust, production-grade solutions with strong metadata, observability, governance, and security
• Troubleshoot complex data/platform issues in enterprise environments
• Foster a quality-focused, delivery-driven engineering culture

Key requirements

• Significant hands-on experience as a Senior or Lead Data Engineer on complex, enterprise-scale systems
• Strong Python and Spark skills with production data pipelines experience
• Full data engineering lifecycle expertise: ingestion, transformation, storage, serving, reuse
• Experience designing integrations across diverse sources, platforms, and legacy environments
• Cloud-based data platforms experience, preferably AWS
• Proven ability to design scalable data models and architectures
• Solid understanding of governance, security, compliance, and operational controls
• Experience with modern engineering practices: source control, automated testing, CI/CD, IaC, deployment automation
• Effective communication with technical and non-technical stakeholders
• Experience supporting ML/AI workloads through production-grade data engineering solutions

• Effective communicator with diverse stakeholders
• Hands-on pragmatism and delivery focus
• Collaborative mindset and willingness to mentor through example
• Python
• Spark
• AWS (S3, Glue, EMR, Redshift, Athena, Lambda, SageMaker, Bedrock)