Senior Big Data Platform Engineer
HCLTech · Kuala Lumpur, Malaysia ·
- Seniority
- Senior
- Category
- Data engineering
Designs, builds, and maintains scalable big data pipelines for enterprise analytics using Python, Apache Spark, and complex SQL in a Linux environment, with ETL/ELT and data quality focus. Day to day involves Airflow orchestration, Kubernetes deployments, MinIO/S3 storage, and partnering with DevOps on CI/CD and runbooks.
HCLTech is seeking a Big Data Platform Engineer to design, build, and maintain scalable data pipelines using Python and Apache Spark in a Linux-based environment. The role focuses on ETL/ELT processes, complex SQL transformations, and ensuring data quality for enterprise analytics.
You will work with Airflow for orchestration, Kubernetes for deployment, and MinIO/S3 storage, while collaborating with DevOps to implement CI/CD pipelines and runbooks.