Senior Data Engineer - Databricks
Melonleaf Consulting · India ·
- Seniority
- Senior
- Category
- Data engineering
- Experience
- 3+ years
Melonleaf Consulting · India ·
About the Role :
As the Senior Data Engineer, you will design and implement a greenfield Databricks Lakehouse for our CRM domain, connecting Salesforce and other key data sources. You will establish scalable pipelines and the ML infrastructure needed to drive advanced customer forecasting and strategic business decisions.
You will act as the technical design authority and driver for the end-to-end data pipelines (ETL/ELT), transforming raw data from Salesforce, Redshift, and Datalake into a high-quality Medallion architecture.
You will apply Infrastructure as Code (Terraform) to provision, configure, and secure the Databricks workspace and Unity Catalog to enforce strict GDPR and PII compliance policies for partner data.
You will lead the implementation of data monitoring dashboards and BI solutions (Databricks SQL/MicroStrategy) and architect the bi-directional data flow required for CRM activation (Reverse ETL) and advanced customer forecasting.
You will be the go-to technical expert, mentoring colleagues, reviewing code, and ensuring engineering best practices are defined and followed for operational excellence (e.g., alerting, performance optimization).
Requirements :
- 4+ years of experience as a Data Engineer, with 3+ years specifically working with Databricks, Spark, and Delta Lake in a production environment.
- Proven experience in designing and driving the implementation of systems with cross-component scope, particularly integrating enterprise CRM systems.
- Proficiency in Infrastructure as Code (Terraform), advanced Python programming, and CI/CD for building scalable ETL/ELT pipelines and ML infrastructure (MLflow, Databricks Genie).
- Expert knowledge in data governance, security, and quality management principles (e.g., Unity Catalog, PII handling) to establish standards for highly sensitive customer data.
- Proven hands-on experience designing, developing, and publishing dashboards and reports using Databricks SQL, including integration with BI tools such as MicroStrategy.
Responsibilities :
- Design and develop data-monitoring dashboards and business reports using Databricks SQL and MicroStrategy, including defining KPIs, building reporting datasets, optimizing queries, and implementing access controls.