AI Data Engineer
SG Analytics Pvt. Ltd. · Mumbai ·
- Category
- Data engineering
- Experience
- 4+ years
SG Analytics Pvt. Ltd. · Mumbai ·
Roles and Responsibilities :
- Architect and scale enterprise lakehouse solutions using Azure Databricks, ADLS Gen2, Delta Lake, and medallion architecture (Bronze, Silver, Gold).
- Design, build, and maintain production-grade batch and streaming ELT/ETL pipelines using PySpark, Spark SQL, Azure Data Factory, and Databricks Workflows.
- Develop configuration-driven, metadata-based ingestion frameworks to rapidly onboard structured, semi-structured, and unstructured data sources.
- Curate and optimize gold-layer data products, vector search indexes, and embeddings via Azure AI Search to support Retrieval-Augmented Generation (RAG) and AI agents.
- Enforce end-to-end data governance, access management, and data lineage utilizing Databricks Unity Catalog and Microsoft Purview to ensure strict compliance (Section 7216, PCAOB, NIST AI RMF).
- Implement performance tuning, Liquid Clustering, Z-Ordering, file compaction, and right-sized compute strategies to minimize platform compute costs.
- Establish CI/CD pipelines, automated data quality testing, data contracts, and proactive observability (freshness, volume, schema drift).
- Collaborate with AI developers, architects, and cross-functional teams to integrate point solutions into shared, scalable platform services.
Technical Expertise :
- Core Cloud/Data : Azure Databricks, ADLS Gen2, Azure Data Factory, Synapse.
- Languages & Querying : Strong proficiency in Python, PySpark, Spark SQL, and SQL.
- Storage Formats : Delta Lake, Parquet, Avro, JSON.
- AI Grounding : Vector Search, RAG Pipelines, Azure AI Search, Embeddings, Chunking.
- Governance & DevOps : Databricks Unity Catalog, Microsoft Purview, CI/CD (Azure DevOps/GitHub Actions), Infrastructure-as-Code (Terraform/Bicep).
Requirements :
- 4+ years of hands-on experience building production data pipelines and data platforms.
- Proven track record designing lakehouse architectures on Azure cloud infrastructure.
- Candidate must be willing to relocate near to Goregaon, Mumbai Location.