Senior Data Engineer

Job Description

Location : (Remote / Hybrid — Kolkata, Hyderabad, Bangalore) Type : Full-time Level : Senior About the role We're looking for a Senior Data Engineer to design, build, and operate the data platform that powers our analytics and reporting. You'll own data pipelines and models end to end — from source ingestion through a Lakehouse medallion architecture to the semantic layer that business users rely on. This is a long-term build-and-run role: you'll ship new data products, keep existing ones healthy, and continuously raise the quality and performance bar of the platform. We run on Microsoft Fabric (Lakehouse, Delta, DirectLake Power BI), so Fabric experience is a strong plus — but we care more about deep, transferable data engineering fundamentals than any single vendor stack. What you'll do Design and build batch and incremental data pipelines in PySpark across Bronze/Silver/Gold layers. Model data for analytics — dimensional / star schemas, slowly changing dimensions, conformed dimensions, and well-partitioned Delta tables. Own the semantic layer: build and maintain Power BI models and DAX measures (DirectLake), and partner with analysts on what the business needs. Build data quality and validation into everything — row/column parity, schema checks, freshness, null/range/dedup rules — so issues are caught before stakeholders see them. Tune performance and cost — partitioning, file compaction, broadcast joins, and eliminating common Spark anti-patterns. Operate what you build: monitoring, logging, alerting, incident response, and clean CI/CD via Azure DevOps. Collaborate in a shared codebase with strong Git discipline — branches, PRs, code review — and mentor more junior engineers. Translate ambiguous business asks into reliable, documented, maintainable data products. What we're looking for Required 3+ years in data engineering, with significant Spark / PySpark at production scale. Strong SQL and dimensional data modeling (Kimball-style star schemas, SCD patterns, surrogate keys). Solid grasp of lakehouse / medallion architecture — what belongs in each layer and why — and Delta Lake (MERGE/upserts, time travel, OPTIMIZE, partitioning). Comfort building incremental, idempotent pipelines rather than full-reload jobs. Python engineering fundamentals — testing, modularity, reusable libraries. Git-based collaboration and CI/CD (Azure DevOps, GitHub Actions, or similar). A quality- and ownership-first mindset: you instrument, validate, and monitor your own work. Git-based collaboration and CI/CD (Azure DevOps, GitHub Actions, or similar). A quality- and ownership-first mindset: you instrument, validate, and monitor your own work. Preferred Microsoft Certified: Fabric Data Engineer Associate (DP-700) — strongly preferred. Hands-on Microsoft Fabric (Lakehouse, OneLake, Notebooks, Data Pipelines) and Power BI / DirectLake + DAX. Agentic / AI-assisted development — productive with LLM coding agents and tooling (e.g. Claude Code, MCP servers, custom agents/skills) to accelerate engineering work while keeping a human-in-the-loop quality bar. Python data-quality and testing tooling — pytest, chispa, Great Expectations. Spark performance tuning at scale. Streaming or near-real-time ingestion experience. Exposure to healthcare / pharmacy / pricing data domains. What success looks like You ship reliable data products that pass review on the first or second pass and need little rework. The pipelines you own are observable, well-tested, and rarely page anyone at night. You leave the platform cleaner than you found it — better patterns, fewer anti-patterns, faster onboarding for the next engineer. Tech you'll work with Microsoft Fabric (Lakehouse, OneLake, Notebooks, Data Pipelines) · PySpark · Delta Lake · Power BI / DirectLake · DAX · SQL · Python (pytest, chispa, Great Expectations) · Azure DevOps