Lead Data Engineer - Medallion Architecture
Square Root Consulting · Bangalore ·
- Work mode
- Onsite
- Seniority
- Lead
- Category
- Data engineering
- Experience
- 7+ years
Square Root Consulting · Bangalore ·
Were looking for a hands-on Lead Data Engineer with strong expertise in Databricks Lakehouse architecture. Youll design and optimize scalable data platforms, lead small teams, and drive innovative solutions for complex data challenges.
Role : Lead Data Engineer
Location : Bangalore (Onsite WFO 5 Days) (India)
Experience : 7 to 12 years
Certification : Databricks Certified Data Engineer Associate (mandatory); Professional preferred
What Youll Do :
- Build batch & streaming pipelines using PySpark, Spark SQL, Structured Streaming
- Implement Medallion architecture (Bronze, Silver, Gold layers)
- Work with Delta Lake features (Time Travel, Schema Evolution, MERGE, CDC, Data Quality)
- Apply Unity Catalog for governance, lineage, and security
- Optimize Spark jobs for performance and cost efficiency
- Lead and mentor engineers, review code, and advise stakeholders
- Automate deployments with CI/CD, Asset Bundles, CLI, REST APIs, Git
Must-Have Skills :
- Databricks (Workflows, Delta Live Tables, Autoloader, Unity Catalog, SQL, Asset Bundles)
- Apache Spark (PySpark, Spark SQL, Structured Streaming, performance tuning)
- Advanced Python & SQL
- AWS Databricks + S3
Nice-to-Have Skills :
- MLflow, Kafka, Airflow, Snowflake, dbt
- Experience with AdTech/MarTech data problems (identity resolution, campaign attribution, name normalization)
- Generative AI pipelines for data engineering
Soft Skills :
- Strong problem-solving & design thinking
- Excellent communication & stakeholder management
- Agile/Scrum experience