Ai Data Engineering Lead

Our client is seeking an experienced AI Data Engineering Lead to design, build, and lead data engineering solutions that support Artificial Intelligence (AI), Machine Learning (ML), and Generative AI (GenAI) initiatives.

Responsibilities

• Lead the design and development of scalable data platforms and pipelines to support AI, ML, analytics, and GenAI applications.
• Build and maintain data pipelines for structured, semi-structured, and unstructured data from enterprise systems.
• Develop batch and real-time data processing solutions to support AI model training, inference, and evaluation.
• Design data ingestion and processing workflows for GenAI applications, including Retrieval-Augmented Generation (RAG), document processing, embedding generation, and vector search.
• Integrate enterprise data platforms with AI services, search engines, vector databases, and other relevant technologies.
• Establish data engineering standards, reusable frameworks, automated testing, CI/CD, and DataOps practices to improve development efficiency and solution quality.
• Implement data quality checks, data lineage, metadata management, monitoring, and error-handling mechanisms to ensure data accuracy and reliability.
• Ensure data engineering solutions comply with data governance, information security, privacy, and regulatory requirements.
• Work with security, risk, compliance, and data governance teams to implement appropriate access controls and safeguards for sensitive data.
• Lead and mentor data engineers, conduct code and architecture reviews, and provide technical guidance throughout the development lifecycle.
• Collaborate with enterprise architects, AI engineers, platform teams, and business stakeholders to translate requirements into effective technical solutions.
• Continuously improve data pipeline performance, platform scalability, operational resilience, and infrastructure cost efficiency.

Requirements

• At least 8 years of experience in data engineering, data platforms, or related technology roles, with demonstrated technical leadership experience.
• Practical understanding of AI/ML data preparation, Generative AI architectures, RAG pipelines, document processing, embeddings, and vector search.
• Bachelor's or Master's degree in Computer Science, Data Engineering, Information Technology, or a related discipline.
• Strong programming skills in Python and SQL, with hands-on experience in Apache Spark, PySpark, or equivalent distributed processing technologies.
• Proven experience designing and implementing enterprise data pipelines, data lakes, lakehouses, data warehouses, and data integration solutions.
• Experience with cloud platforms such as Microsoft Azure, AWS, or Google Cloud, and workflow orchestration tools such as Apache Airflow or equivalent.
• Experience with CI/CD, automated testing, monitoring, and production support for data platforms.
• Strong understanding of data quality, data governance, security controls, data lineage, and access management.
• Experience in banking, financial services, or another regulated industry is a plus.
• Familiarity with technologies such as Databricks, Snowflake, Kafka, Azure AI Search, MLOps, or LLMOps is an advantage.