Senior Data Engineer - Databricks & AI Solutions
Okda Solutions · India ·
- Seniority
- Senior
- Category
- Data engineering
- Experience
- 6+ years
Okda Solutions · India ·
Job Summary :
We are looking for a Senior Data Engineer with 6+ years of experience in Databricks, real-time data processing, Lakehouse architecture, and AI-driven solutions. The role involves designing scalable batch and streaming data platforms and delivering high-quality, governed data and AI solutions.
Key Responsibilities :
- Design, develop, and maintain batch and streaming data pipelines using Databricks and PySpark.
- Build, optimize, and manage Delta Lake tables within Unity Catalog.
- Develop and support Lakeflow pipelines with incremental processing, CDC, and data quality validation.
- Integrate SharePoint and other external enterprise data sources into the Databricks Lakehouse.
- Ensure data quality, governance, security, reliability, and operational excellence.
- Monitor, troubleshoot, and optimize data workflows for performance and reliability.
- Collaborate with business and technical stakeholders to deliver scalable solutions.
- Build AI-powered data solutions, copilots, and intelligent agents using LLMs, RAG, and vector search.
- Implement CI/CD and automated deployment using Databricks Repos and Asset Bundles.
- Drive deliverables independently and proactively resolve technical challenges.
Mandatory Skills :
- 6+ years of Data Engineering experience.
- Strong hands-on experience with Databricks and modern Lakehouse architecture.
- Strong proficiency in PySpark, Spark Structured Streaming, and SQL.
- Experience with Delta Lake, Unity Catalog, Databricks Workflows, and job orchestration.
- Experience with Bronze, Silver, and Gold Lakehouse architecture.
- Practical experience with Databricks Lakeflow, Declarative Pipelines, Lakeflow Connect, CDC, data quality, and incremental processing.
- Experience integrating enterprise data platforms with AI-driven applications.
- Experience with LLMs, RAG, vector search, and prompt engineering.
- Strong understanding of automated deployment and release processes.
- Experience with Git-based source control and CI/CD.
Data Governance & Security :
- Unity Catalog governance and security.
- Role-Based Access Control (RBAC).
- Data masking and data protection policies.
- Audit logging, data lineage, and compliance.
DevOps & Deployment :
- Git-based source control and CI/CD.
- Databricks Repos.
- Databricks Asset Bundles (DABs).
- Automated deployment and release processes.
Preferred Skills :
- Experience with large-scale enterprise data platforms.
- Strong understanding of Lakehouse and Data Mesh principles.
- Exposure to Azure cloud services and enterprise integration patterns.
- Experience with NoSQL databases.
- Strong problem-solving, communication, and stakeholder management skills.