Lead Data Engineer (Databricks & Spark)

We are building a next-generation analytics platform and need a Lead Data Engineer to shape scalable ingestion, transformation, and governed data products on Databricks. You will combine hands-on delivery with technical leadership and architecture to enable self-service analytics and AI-assisted engineering.

Responsibilities

• Design scalable data ingestion and transformation frameworks on Databricks
• Define and enforce Databricks best practices for medallion layers and operational excellence
• Build batch and streaming pipelines using Apache Spark and PySpark
• Deliver end-to-end data products from ingestion through governed consumption layers
• Develop trusted analytical datasets and semantic models for reporting and self-service analytics
• Implement data quality, lineage, monitoring, and observability standards
• Lead proof-of-concepts and evaluate emerging capabilities in the Databricks ecosystem
• Mentor engineers and drive reusable frameworks adopted across teams

Requirements

• 5+ years of data engineering experience with production-grade platforms
• Strong Databricks experience in enterprise production environments
• Lead-level technical leadership skills to drive standards and mentor engineers
• End-to-end delivery experience across ingestion, transformation, semantic layers, and reporting
• Advanced Apache Spark and PySpark expertise for scalable pipeline development
• Advanced SQL skills for transformation, modeling, and performance tuning
• Upper-Intermediate English proficiency (B2) for leading technical discussions

Nice to have

• Data architecture experience for enterprise lakehouse patterns
• Data solution architecture skills to define scalable platform designs
• Scala proficiency for Spark development and optimization

Benefits

• International projects with top brands
• Work with global teams of highly skilled, diverse peers
• Healthcare benefits
• Employee financial programs
• Paid time off and sick leave
• Upskilling, reskilling and certification courses
• Unlimited access to the LinkedIn Learning library and 22,000+ courses
• Global career opportunities
• Volunteer and community involvement opportunities
• EPAM Employee Groups
• Award-winning culture recognized by Glassdoor, Newsweek and LinkedIn