Lead Data Engineer, AWS Data Platform
Hiring immediately. We are looking for someone who can start now or on a short notice period.
Experience: 5+ years in data engineering, 1 to 2 of those leading Type: Full-time Location: Remote (India) Working hours: 5:00 PM to 2:00 AM IST. Hard requirement, not a preference.
Stack: AWS (S3, Glue, Redshift, Athena, Lake Formation, SNS/SQS, RDS Postgres), Apache Airflow, a metadata-driven ingestion framework, SQL, Python, GitHub, VS Code, GitHub Copilot
About the role
We run a modern data platform on AWS that feeds analytics and AI/ML teams at an enterprise client. Data moves from a raw landing zone through curated layers into Redshift, driven by a metadata framework rather than hand-written one-off pipelines, and it runs across both non-production and production environments. We want one person accountable for the design, the standards, and the roadmap of that platform. Senior enough to say no to a bad model before it ships, and calm enough to run a backfill without turning it into a war room.
This is an immediate opening on a live project, so we are moving fast on interviews and offers. If you are available on a short notice period, say so in your application.
The evening shift is real and it is the reason this role is hard to fill. Your day overlaps almost entirely with a US-based stakeholder group, and this role talks to them directly rather than through a proxy. If that overlap does not work for you long term, please do not apply. We would rather lose you now than in month three.
What you will own
- The metadata-driven ingestion framework. Extending it, hardening it, and making it the default path for new sources instead of another bespoke pipeline.
- The raw to land to Redshift flow. Layer contracts, partitioning, file formats, compaction, and load patterns that stay reliable as source count grows.
- The Redshift warehouse. Dimensional models, distribution and sort keys, WLM behaviour, and a real answer for cost per query.
- Orchestration in Airflow. DAG standards, retries, SLAs, alerting, dependency design, and backfills that do not become incidents.
- Non-prod and prod parity. Promotion process, environment configuration, and the discipline that keeps the two from drifting apart.
- Governance in Lake Formation. Fine-grained access, row and column controls, and a defensible answer to "who can see what and who approved it".
- Event-driven flows on SNS and SQS, including the parts people skip: DLQs, replay, ordering, idempotency.
- RDS Postgres. Schema design, indexing, and query performance for the operational and metadata layers.
- The contract with analytics and ML. Curated datasets and models those teams can build on without messaging you first.
- The engineering bar. Code review, modelling and naming standards, documentation, and mentoring 2 to 4 engineers.
Must have
- 5+ years building production data platforms, including 1 to 2 years as a lead or tech lead accountable for other engineers' output, not just your own.
- Hands-on AWS data stack: S3, Glue (jobs, crawlers, catalog), Redshift, Athena, Lake Formation, and RDS Postgres. Ownership, not exposure.
- SNS and SQS used inside real data or event-driven pipelines, including failure handling and replay.
- Experience with metadata-driven or config-driven ingestion frameworks, and a clear view of why they beat one-off pipelines at scale.
- Airflow in production. You have authored DAGs, run backfills, and been the person paged when they failed.
- SQL you can defend line by line, plus genuine dimensional modelling background (Kimball or equivalent).
- Redshift performance work: distribution and sort key decisions, load strategy, and diagnosing a slow query rather than throwing compute at it.
- Comfortable working across separate non-production and production environments with a controlled promotion process.
- Python for pipeline and framework development, with Git-based collaboration and code review.
- Proven remote leadership: async written communication, running reviews, and holding a technical position with stakeholders you have never met in person.
- Ability to work 5:00 PM to 2:00 AM IST consistently.
Nice to have
- Curated data layers or a feature store built for ML teams.
- Infrastructure-as-code for data infra: Terraform, CDK, or CloudFormation.
- Cost governance: Redshift concurrency scaling and WLM tuning, Glue DPU sizing, S3 lifecycle policies.
- dbt or a comparable transformation framework.
- Working fluently with AI coding assistants such as GitHub Copilot.
We review applications daily and respond within 48 hours. If you fit the brief and can start soon, apply now.