Senior Data Engineer - Databricks

Position Overview:

Datakulture, Sedin's Data Analytics & Business Intelligence division, is looking for a highly experienced Senior Data Engineer with 8+ years of hands-on expertise, with a strong focus on Databricks, to join our Chennai team in an in-office capacity. In this role, you will architect and build enterprise-scale data platforms, Lakehouse environments, and complex data pipelines that power analytics and AI workloads for global clients.

Key Responsibilities:

- Architect, design, and build large-scale data pipelines and Lakehouse solutions on Databricks using PySpark, Spark SQL, and Delta Lake, following Medallion Architecture principles.

- Lead the design and implementation of batch and streaming data pipelines using Spark Structured Streaming, Apache Kafka, AWS Kinesis, or Azure Event Hubs.

- Develop and manage Databricks Workflows, Unity Catalog, Delta Live Tables (DLT), and cluster configurations for optimal performance and governance.

- Engineer robust data ingestion frameworks from diverse sources including RDBMS, REST APIs, flat files, NoSQL stores, and event streams.

- Design and optimize data models for analytical workloads; implement advanced performance tuning including Z-ordering, partitioning, compaction, and caching strategies.

- Build and maintain cloud data infrastructure on AWS and/or Azure alongside Databricks.

- Implement CI/CD pipelines for data engineering workflows; enforce code review standards and Git-based development practices.

- Define and enforce data quality frameworks, data observability standards, and data governance policies.

- Collaborate with data scientists, ML engineers, and BI developers to deliver feature stores, analytical datasets, and ML-ready data pipelines.

- Lead technical discussions, architecture reviews, and stakeholder engagements; mentor and guide junior and mid-level data engineers.

- Drive adoption of Infrastructure as Code (IaC) using Terraform or CloudFormation; manage cost optimization and security best practices.

Required Qualifications & Skills:

- B.E. / B.Tech / M.Tech / M.Sc. in Computer Science, Information Technology, Data Engineering, or a related discipline.

- 8+ years of proven experience in data engineering, with at least 3 - 4 years of deep, hands-on Databricks expertise in production environments.

- Databricks (Expert): PySpark, Spark SQL, Delta Lake, Unity Catalog, Delta Live Tables, Databricks Workflows, cluster tuning, and Databricks SQL.

- Cloud Platforms: Strong proficiency in AWS (S3, Glue, Athena, EMR, Redshift, Kinesis, Lambda, Step Functions) and/or Azure (ADLS Gen2, ADF, Synapse Analytics, Azure SQL, Event Hubs).

- Programming: Advanced Python (pandas, PySpark) and SQL (complex queries, window functions, performance optimization).

- Strong communication and leadership skills; demonstrated ability to drive technical decision-making, engage with senior stakeholders, and mentor engineering teams.

- Data Architecture: Deep understanding of Medallion Architecture, Lakehouse design, Data Vault, star/snowflake schema, and streaming data patterns.

- DevOps & IaC: Proficiency in CI/CD (GitHub Actions, Azure DevOps, Jenkins), Terraform or CloudFormation, and Git-based development workflows.

- Data Governance: Experience implementing data quality frameworks, data observability, and Unity Catalog-based governance.

Preferred Qualifications:

- Databricks Certified Data Engineer Professional or Associate; AWS Certified Data Engineer or Azure Data Engineer Associate.

- Experience delivering data platforms across BFSI, retail, healthcare, manufacturing, or other data-intensive industries.

- Exposure to dbt, Apache Iceberg, Apache Flink, Airflow/Prefect, or ML platform tooling (MLflow, Feature Store).