Dear Applicant,
If you or someone you know is interested, please send the CV directly to [email protected] (most preferred, as I may overlook some CVs due to the high volume).
Please note that visa sponsorship is not available at this time.
Key Responsibilities
- Implement and operationalize enterprise-scale Lakehouse platforms, data products, and data marketplace capabilities.
- Design, develop, test, and maintain scalable batch, streaming, CDC, and API-based data ingestion pipelines.
- Develop multimodal data ingestion pipelines, including:
o Content extraction from various file formats.
o Regex-based extraction of specific fields.
o Content extraction from embedded images.
o Frame extraction from video files.
o Transcript extraction from audio files.
- Build and maintain foundation and business data products with defined data contracts, SLAs, data quality controls, and governance standards.
- Implement and work with modern open table formats, including Iceberg, Hudi, and Delta Lake.
- Develop and support data pipelines for RAG, vector search, Generative AI (GenAI), and agentic AI use cases.
- Perform performance tuning, optimization, production support, troubleshooting, and root cause analysis across data platforms and pipelines.
- Implement data ingestion, transformation, reconciliation, and data quality frameworks.
- Develop data architectures supporting NLP and AI-driven analytics, including the ingestion, curation, governance, and management of structured and unstructured data.
- Support ML platforms and workflows, including model development, deployment, and operationalization.
- Develop internal engineering tools and full-stack applications using Python, shell scripting, and modern web frameworks.
- Expose and integrate data through APIs, event streams, dashboards, and BI platforms.
- Create and maintain technical documentation, deployment guides, and operational runbooks.
- Ensure compliance with engineering standards, DevSecOps controls, CI/CD practices, security requirements, and software delivery standards.
- Collaborate effectively with distributed engineering, data, technology, and business teams across multiple projects.
- Drive continuous improvement in data platform reliability, scalability, automation, and operational excellence.
Key Requirements
- Bachelor’s degree in Computer Science, Engineering, Information Technology, or a related discipline.
- 8–12 years of experience in Data Engineering, Big Data, Data Lake, Lakehouse, or large-scale data platform implementations.
- Strong hands-on experience with enterprise data platforms such as Databricks, Snowflake, Cloudera, Azure, AWS, GCP, Huawei, or Alibaba Cloud.
- Hands-on experience designing and developing data products and data marketplace capabilities.
- Strong expertise in Spark, PySpark, SQL, Python, and/or Scala, with strong overall programming skills.
- Proven experience building scalable data ingestion, transformation, streaming, CDC, reconciliation, and data quality pipelines.
- Hands-on experience with Iceberg, Hudi, Delta Lake, and object storage platforms.
- Experience with modern data technologies such as Kafka, Flink, Spark Streaming, Airflow, Trino, Dremio, Hive, and/or Impala.
- Strong hands-on experience with Kubernetes, OpenShift, Docker, Terraform, Jenkins, Git, CI/CD, MLflow, and observability tools.
- Experience designing data architectures and pipelines supporting NLP, AI-driven analytics, RAG, vector search, GenAI, and agentic AI use cases.
- Experience handling and governing unstructured and multimodal data, including documents, images, audio, and video.
- Experience with ML platforms and frameworks such as CML, Spark MLlib, scikit-learn, and XGBoost, including model deployment.
- Strong knowledge of data modelling, metadata management, data lineage, data governance, and data quality.
- Experience exposing data through APIs, event streams, dashboards, and BI platforms.
- Experience with full-stack/internal engineering tool development using Python, shell scripting, Flask, React, or similar technologies is advantageous.
- Knowledge or experience with Teradata, Netezza, Greenplum, or MPP migration programmes is advantageous.
- Strong engineering, automation, troubleshooting, and performance optimization mindset.
- Strong communication and stakeholder management skills, with the ability to work effectively across distributed teams and multiple projects.
- Experience working in Agile delivery environments and enterprise-scale technology platforms.
- Strong commitment to quality, operational excellence, automation, and continuous improvement.
- Relevant certifications such as Databricks Certified Data Engineer, Azure Data Engineer Associate, AWS Data Analytics Specialty, Google Professional Data Engineer, SnowPro, or DAMA CDMP would be an advantage.