Title: Senior Data Engineer
Location: Chennai
Work Mode: Hybrid
Job purpose
We are seeking an experienced Data Engineer to support a large-scale Global Capital Markets platform-modernization program. The role focuses on designing, building and optimizing scalable data pipelines and analytical workflows using Databricks, Python, PySpark and AWS cloud services, underpinning the stabilization of Front Office Trading systems and the build-out of target-state capabilities across event-driven integration, data-lake / Databricks platforms, reporting, migration and the front-to-back trade lifecycle for FX Cash, Cleared IRS and future product onboarding. The ideal candidate combines strong hands-on data-engineering and distributed-processing expertise with a solid grasp of ETL/ELT and data lake architectures, and partners closely with Front Office, Product Owners, Risk, Quantitative Analysts and Technology teams to deliver production-ready, high-performance data solutions in a complex capital-markets setting.
Key responsibilities
- Partner with Front Office, Product Owners, Quantitative Analysts, Risk and Technology teams to understand business requirements and transform them into scalable data and analytics solutions.
- Design, develop and maintain data pipelines and analytical workflows using Databricks, Python, PySpark and AWS cloud services.
- Implement scalable ETL/ELT solutions leveraging AWS services such as S3, Glue, Lambda, CloudWatch and Databricks, and develop reusable data models, transformation frameworks and data-quality controls.
- Develop and optimize PySpark applications and distributed-processing workloads to improve performance, scalability and operational efficiency.
- Develop APIs and data services using Python and FastAPI to enable consumption of analytics and business data across multiple applications.
- Analyze structured and unstructured datasets to derive actionable insights and support business decision-making, aligned to business and regulatory requirements.
- Support migration and modernization initiatives involving data platforms, cloud transformation and distributed-processing architectures, optimizing data architecture, processing performance and cloud resource utilization.
- Collaborate with stakeholders to define data-quality standards, reconciliation processes, governance controls and monitoring frameworks.
- Contribute to CI/CD, DevOps, testing and release-management activities, and participate in Agile delivery including estimation, sprint planning, refinement, deployment and production support.
- Identify risks, dependencies, bottlenecks and performance challenges and proactively recommend solutions; contribute to documentation, knowledge sharing and operational readiness.
Key competencies
- Bachelor’s / Master’s degree with 8+ years of experience in Data Engineering, Data Science or Analytics Engineering roles.
- Strong hands-on experience with Databricks on AWS and distributed data processing frameworks, with a strong understanding of Spark architecture, performance tuning and optimization.
- Advanced programming skills in Python and PySpark.
- Experience implementing data-engineering solutions using AWS services including S3, Glue, Lambda, IAM, CloudWatch and API Gateway.
- Expertise in building scalable ETL/ELT pipelines and modern data lake architectures, with strong data-warehousing, dimensional-modeling and large-scale data-management knowledge.
- Strong SQL skills across relational and NoSQL databases, and experience developing APIs using FastAPI or similar Python frameworks.
- Experience with CI/CD pipelines, Git, DevOps and automated deployment practices, working in Agile environments with JIRA and sprint-based execution.
- Excellent analytical, problem-solving and troubleshooting skills, with strong stakeholder-management and communication skills across business and technology teams.
Preferred qualifications
- Capital Markets experience covering Equities & Derivatives.
- Databricks and/or AWS data-engineering certifications.
- Prior experience in regulated, security-controlled financial-services environments and platform-modernization / migration programs.