Job Title: Data Engineer - Databricks (AWS) | PySpark | Delta Lake
Location: Bengaluru, Noida, Gandhi Nagar, Pune, Chennai
Experience: 4+ Years
Job Summary
We are looking for a talented Data Engineer with strong expertise in Databricks on AWS, PySpark, Delta Lake, and modern data engineering practices. The ideal candidate will be responsible for building scalable data pipelines, implementing data governance frameworks, optimizing platform performance, and supporting enterprise analytics initiatives within the Healthcare/Life Sciences domain.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks on AWS.
- Build and support batch and streaming data ingestion frameworks.
- Implement data quality, governance, lineage, and security controls using Unity Catalog and Databricks best practices.
- Develop and optimize ETL/ELT processes using PySpark and SQL.
- Design and implement Medallion Architecture (Bronze, Silver, Gold) data solutions.
- Build and maintain CDC-based ingestion pipelines using AWS and Databricks services.
- Optimize Spark performance, cluster utilization, and platform costs.
- Implement and support Delta Lake features to ensure data reliability and governance.
- Manage Structured Streaming, Auto Loader, and real-time data processing requirements.
- Develop CI/CD processes using Git, Databricks Asset Bundles, and deployment automation tools.
- Monitor production environments, troubleshoot issues, and ensure SLA compliance.
- Collaborate with business stakeholders, analysts, and engineering teams to deliver high-quality data solutions.
- Mentor junior engineers and contribute to data engineering best practices.
Required Skills & Qualifications
Technical Skills
- 4+ years of experience in Data Engineering.
- 4+ years of hands-on experience with Databricks on AWS.
- Strong expertise in PySpark and Spark optimization techniques.
- Experience with Delta Lake and Lakehouse architecture.
- Strong SQL programming skills.
- Experience with Unity Catalog and Databricks governance frameworks.
- Knowledge of dimensional data modeling and Medallion Architecture.
- Hands-on experience with:
- Structured Streaming
- Auto Loader
- Change Data Capture (CDC)
- Experience with AWS services:
- Amazon S3
- IAM
- AWS DMS
- Kinesis and/or MSK
- AWS Glue
- Experience with Git, CI/CD pipelines, and Databricks Asset Bundles.
- Experience supporting production environments, performance tuning, and cost optimization.
Domain Experience
- Healthcare or Life Sciences domain experience preferred.
- Understanding of data governance, analytics, privacy, and regulatory compliance requirements.
Education
- Bachelor's or Master's Degree in Computer Science, Information Technology, Engineering, or related field.
- Flexible for candidates with strong relevant experience.
Certifications (Preferred)
- Databricks Certified Data Engineer Associate/Professional
- AWS Certified Data Engineer
- AWS Certified Solutions Architect
- Microsoft Azure Certified Developer
- Google Professional Cloud Developer