Senior Data Engineer - PySpark
SG Analytics Pvt. Ltd. · Mumbai ·
- Seniority
- Senior
- Category
- Data engineering
SG Analytics Pvt. Ltd. · Mumbai ·
Roles and Responsibilities:
- Design, construct, and manage scalable Lakehouse architecture on ADLS Gen2 and Azure Databricks following medallion pattern (Bronze, Silver, Gold).
- Develop metadata-driven ETL and ELT ingestion pipelines using Azure Data Factory, Databricks Workflows, and Structured Streaming.
- Create vector-ready data products and embeddings via Azure AI Search to enable Retrieval-Augmented Generation (RAG) and LLM applications.
- Standardize open table formats on Delta Lake while optimizing performance using partitioning, Z-Ordering, liquid clustering, and file compaction.
- Implement automated data governance, cataloging, lineage tracking, and role-based access control using Unity Catalog and Microsoft Purview.
- Ensure strict adherence to regulatory standards, data privacy, and security frameworks including PCAOB, NIST AI RMF, and PII/PHI protection.
- Optimize platform compute, storage, and pipeline execution costs to balance scale, reliability, and cost-per-workload efficiency.
- Build CI/CD data pipelines, automated testing, and observability monitors for pipeline freshness, schema drift, and data quality checks.
Preferred Profile:
- Work Experience: Minimum 4+ years of hands-on experience in building production-grade data engineering platforms, pipeline architecture, and data lakes.
Technical Expertise:
- Core Technologies: Azure Databricks, Azure Data Factory, ADLS Gen2, Python, PySpark, SQL, Delta Lake.
- Governance & Security: Unity Catalog, Microsoft Purview, PII segregation.
- AI & Advanced Data: Azure AI Search, Vector Databases, Embeddings, RAG architecture.
- Bonus Skills: dbt, Delta Live Tables, Terraform, Kafka, Event Hubs, Apache Iceberg.
- Educational Qualification: Bachelors or Masters degree in Computer Science, Data Science, Engineering, or relevant technical field.
- Domain Preference: Professional services, consulting, fintech, or regulated industries handling complex audit/financial datasets.