We are seeking a highly experienced Senior Data Engineer to design, build, and operate robust Databricks and Lakehouse data platforms. This role will focus on enabling analytics, AI, and Generative AI applications by delivering high-quality, governed, and scalable data assets. Working within product-aligned squads, you will collaborate with AI Engineers, Product Owners, and analytics teams to support cutting‑edge GenAI use cases, including Retrieval Augmented Generation (RAG), and contribute to the development of AI Engineers.
Key Responsibilities
- Design, build, and operate data solutions using Databricks components such as Delta Lake, Databricks Jobs and Workflows, and Unity Catalog.
- Develop production‑grade data pipelines using Python, SQL, and Apache Spark.
- Implement automated testing, CI/CD practices, and ensure data solutions are observable, resilient, performant, and cost‑efficient.
Data Enablement for AI/GenAI
- Enable data consumption for Generative AI use cases (RAG, AI services, agent workflows), analytics, reporting tools, and downstream systems.
- Support feature‑style and curated data access patterns required for AI and GenAI workloads.
- Build data pipelines to feed Generative AI applications, including curated knowledge datasets, structured/semi‑structured data, and metadata/lineage for AI consumption.
- Implement data patterns essential for GenAI, such as RAG, context/prompt data preparation, and model input/output/feedback data flows.
Collaboration and Governance
- Work as a senior individual contributor within a cross‑functional product squad.
- Collaborate closely with Product Owners, AI/ML Engineers, Analytics teams, and platform/security teams.
- Provide engineering input into design discussions and delivery decisions, supporting peer reviews and shared engineering standards.
- Ensure data solutions comply with enterprise security, risk, and governance standards.
- Support operational stability, participate in incident resolution, root cause analysis, and maintain documentation.
Required Skills and Qualifications
- 10-15 years of industry experience.
- Proven experience as a Senior/Lead Data Engineer (5+ years).
- Hands‑on experience with Databricks environments (2+ years).
- Strong understanding of enterprise data lake and lake house architecture (5+ years).
- Proficiency in Python, SQL, and Apache Spark (5+ years).
- Experience building and operating production‑grade data platforms (3+ years).
- Experience working in enterprise or regulated environments (5+ years).
Preferred Qualifications
- Experience enabling AI, ML, or Generative AI use cases from a data engineering perspective.
- Familiarity with RAG data patterns, feature‑style or AI‑serving datasets, and vector or embedding‑ready data workflows.
- Experience working in Agile, product‑aligned squads.
- Exposure to cloud‑native data platforms (AWS or Azure).
Skills
Databricks Delta Lake Unity Catalog Python SQL apache spark CI/CD Generative AI RAG Al Services Agent Workflows Data Warehousing Machine Learning Cloud‑native data platforms AWS Azure