Senior/Lead Databricks Engineer
EXL Services · Pune; Bangalore; Delhi ·
- Seniority
- Lead
- Employment
- Full time
- Experience
- 3+ years
EXL Services · Pune; Bangalore; Delhi ·
Job Title : Databricks Engineer
Department : Client Delivery / Data & AI Engineering
Experience Required : 10 - 12 Years (overall) | 5+ Years Databricks hands-on
Employment Type : Full-Time
Location : Hybrid (as per business requirement)
Reporting To : Delivery Manager / Technical Project Manager
Notice Period : Immediate to 15 Days
Key Responsibilities :
Databricks Platform Engineering :
- Design, build, and maintain Databricks workspaces, clusters, and compute pools across dev/test/prod environments.
- Configure and manage Databricks Unity Catalog for data governance, access control, fine-grained permissions, and data lineage.
- Optimize cluster configurations instance types, auto-scaling policies, spot/preemptible nodes for cost and performance.
- Implement workspace-level best practices: folder structures, access controls, secret management (Databricks Secrets / Azure Key Vault / AWS Secrets Manager).
- Manage Databricks jobs, workflows, and multi-task job orchestration with dependency management.
Delta Lake & Lakehouse Architecture :
- Design and implement Delta Lake tables with appropriate partitioning, Z-ordering, and file compaction (OPTIMIZE / VACUUM).
- Build Medallion Architecture (Bronze / Silver / Gold) layers for structured data lake organization.
- Implement Delta Live Tables (DLT) pipelines for declarative, reliable ETL/ELT with built-in data quality expectations.
- Manage schema evolution, table versioning, time travel, and Change Data Feed (CDF) for incremental processing.
- Design data lakehouse patterns integrating Delta Lake with external systems (Kafka, ADLS, S3, GCS).
Data Pipeline Development (PySpark / SQL) :
- Develop scalable batch and streaming data pipelines using PySpark, Spark SQL, and Delta Lake.
- Build structured streaming pipelines for real-time ingestion from Kafka, Event Hubs, and Kinesis into Delta tables.
- Write optimized PySpark transformations leveraging broadcast joins, adaptive query execution (AQE), and dynamic partition pruning.
- Create reusable transformation libraries, utility frameworks, and pipeline templates for team productivity.
- Implement robust error handling, retry logic, and dead-letter queue patterns in production pipelines.
MLflow & AI/ML Workloads :
- Set up and manage MLflow tracking servers, experiment registries, and model lifecycle management on Databricks.
- Support data scientists and ML engineers in deploying model training and inference workloads on Databricks clusters and GPU instances.
- Build feature engineering pipelines using Databricks Feature Store for reusable, versioned ML features.
- Enable GenAI workloads LLM fine-tuning, RAG pipeline development, and vector search (Databricks Vector Search / Mosaic AI).
- Implement MLOps practices: model versioning, A/B testing, model serving via Databricks Model Serving endpoints.
Cloud Integration & DevOps :
- Integrate Databricks with cloud-native services: Azure Data Lake Storage (ADLS).
- Build and maintain CI/CD pipelines for Databricks notebooks and jobs using Azure DevOps, GitHub Actions, or GitLab CI.
- Implement Databricks Asset Bundles (DABs) or Terraform for infrastructure-as-code (IaC) deployment of Databricks resources.
- Manage data ingestion using Auto Loader, COPY INTO, and partner integrations (Fivetran, dbt, Airbyte).
- Monitor pipeline health, cluster utilization, and costs using Databricks system tables and cloud cost management tools.
Governance, Security & Optimization :
- Implement row-level security, column masking, and dynamic data views using Unity Catalog policies.
- Ensure data quality enforcement using Delta Live Tables expectations and Great Expectations integrations.
- Conduct performance tuning query plan analysis, caching strategies, Photon engine enablement.
- Maintain data cataloging, metadata management, and data lineage tracking within Unity Catalog.
- Document architecture decisions, runbooks, and operational guides for Databricks workloads.
Required Qualifications :
Education :
- Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or related field.
Experience :
- 10+ years of total experience in data engineering or software engineering.
- 3+ years of dedicated hands-on experience with the Databricks platform in production environments.
- Strong background in big data engineering, cloud data platforms, and distributed computing.
Databricks Platform :
- Deep expertise in Databricks Workspaces, Clusters, Jobs, Workflows, and Repos.
- Proficiency with Unity Catalog metastore setup, catalog/schema/table management, access controls, and data lineage.
- Hands-on experience with Delta Live Tables (DLT) pipeline development, expectations, and monitoring.
- Strong command of Delta Lake internals transaction log, ACID guarantees, file layout, and optimization techniques.
- Experience with Databricks SQL Warehouses, SQL Analytics, and dashboard creation.
- Knowledge of Databricks Photon engine, serverless compute, and cost optimization strategies.
PySpark & SQL :
- 4+ years of PySpark development DataFrames, Datasets, Spark SQL, RDD operations.
- Expert-level SQL window functions, lateral joins, CTEs, recursive queries, and analytical functions.
- Experience with Spark performance tuning AQE, query plans (EXPLAIN), partitioning, and caching.
- Proficiency with Python for pipeline development, utilities, and automation.
Cloud Platforms :
- Hands-on experience with at least one : Azure (ADLS Gen2, ADF, Azure Databricks), AWS (S3, EMR, Glue, AWS Databricks), or GCP (GCS, BigQuery, Dataproc).
- Experience with cloud networking for Databricks : VNet/VPC injection, private endpoints, and firewall configurations.
- Familiarity with IAM roles, managed identities, and service principal authentication for Databricks.
MLflow & ML Engineering (Nice to Have) :
- Working knowledge of MLflow experiment tracking, model registry, and deployment.
- Experience supporting ML pipelines on Databricks for training, evaluation, and serving.
- Exposure to Databricks Feature Store and Mosaic AI / GenAI capabilities.
Preferred Certifications :
- Databricks Certified Associate Developer for Apache Spark (PySpark or Scala).
- Databricks Certified Data Engineer Associate / Professional strongly preferred.
- Databricks Certified Machine Learning Associate / Professional.
- Azure Data Engineer Associate (DP-203) / AWS Data Analytics Specialty / GCP Professional Data Engineer.
- dbt Analytics Engineer Certification.
Preferred Qualifications :
- Experience with dbt (data build tool) for SQL-based transformation on Databricks SQL.
- Knowledge of Apache Kafka / Confluent for real-time streaming into Databricks.
- Familiarity with Terraform or Pulumi for Databricks infrastructure-as-code.
- Exposure to Apache Iceberg or Apache Hudi in addition to Delta Lake.
- Experience with BI tool integration: Power BI, Tableau, or Looker connected to Databricks SQL.
- Knowledge of data mesh principles and federated data governance at scale.
- Domain experience in BFSI, Healthcare, Retail, or Manufacturing data programs.