We are looking for an experienced Senior Data Engineer with 8–12 years of experience in Data Engineering, Big Data, Data Lake, and Lakehouse platforms. The ideal candidate will have strong hands‑on expertise in Databricks, Spark, Pyspark, Python, SQL, cloud data platforms, data products, multimodal data pipelines, and modern data architectures.
Key Responsibilities
- Implement and operationalize enterprise Lakehouse platforms, Data Products, and Data Marketplace capabilities.
- Design and develop scalable batch, streaming, CDC, and API-based data ingestion pipelines.
- Build multimodal data ingestion pipelines supporting
- Build, test, and maintain foundation and business data products with defined data contracts, SLAs, and data quality controls.
- Implement and work with open table formats such as Apache Iceberg, Apache Hudi, and Delta Lake.
- Develop and support data pipelines for RAG, vector search, Generative AI, and agentic AI use cases.
- Perform performance tuning, optimization, production support, troubleshooting, and root cause analysis.
- Develop technical documentation, deployment guides, operational runbooks, and support procedures.
- Ensure compliance with engineering standards, DevSecOps controls, CI/CD practices, and software delivery standards.
- Collaborate with distributed engineering, architecture, data science, and business teams across multiple projects.
Key Technical Requirements
- 8–12 years of experience in Data Engineering, Big Data, Data Lake, or Lakehouse implementations.
- Hands‑on experience with Databricks, Snowflake, Cloudera, and cloud data platforms such as Azure, AWS, or GCP.
- Strong experience building Data Products and Data Marketplace capabilities.
- Excellent programming skills in Python, Scala, Java, and SQL.
- Strong hands‑on expertise with Spark / PySpark.
- Experience with Iceberg, Hudi, Delta Lake, and object storage platforms.
- Strong experience with data ingestion, transformation, reconciliation, and data quality frameworks.
- Experience with Kafka, Flink, Spark Streaming, Airflow, Trino, Dremio, Hive, and Impala.
- Hands‑on experience with Kubernetes, OpenShift, Docker, Terraform, Jenkins, Git, and CI/CD pipelines.
- Experience with MLflow and observability platforms/tools.
- Ability to design data architectures for NLP, AI/ML, GenAI, and unstructured data.
- Experience with ML platforms and libraries such as CML, Spark MLlib, scikit‑learn, and XGBoost, including model deployment.
- Experience developing internal engineering tools and applications using Python, Shell scripting, Flask, React, or similar modern frameworks.
- Knowledge of data modeling, metadata management, data lineage, governance, APIs, event streams, dashboards, and BI platforms.
- Experience with Teradata, Netezza, Greenplum, or MPP migration programs is an advantage.