Senior Data Engineer

Role and Responsibilities:

• Design, develop, and maintain scalable data pipelines and data assets on Databricks and the Microsoft Azure Data Platform to support Business Intelligence and Advanced Analytics objectives.
• Lead the technical migration of legacy databases and on-premises data systems to a cloud-based lakehouse architecture using Delta Lake, including data modeling, transformation logic, and post-migration data validation.
• Develop distributed data processing applications using PySpark, Scala Spark, Python, and SQL for batch, incremental(CDC),and real-time streaming workloads.
• Build and customize data integration tools, data warehouses, data marts, and data lakes using Azure Data Factory, ADLS Gen2, and Kafka.
• Analyze business and system requirements and translate them into technical designs, data models, and software specifications aligned with enterprise architecture.
• Implement data governance, security, and data quality controls, including access management, lineage tracking, and automated data validation frameworks.
• Develop CI/CD pipelines and infrastructure-as-code for automated testing and deployment of data solutions using Databricks Asset Bundles and Git-based version control.
• Optimize performance, reliability, and cost of Spark workloads and cloud infrastructure through job tuning, cluster configuration, and storage optimization.
• Create tools and automation that help client engineering teams derive value from their data.
• Perform hands-on development, lead code reviews and testing, and establish engineering best practices across the team.
• Review solution approaches and provide technical guidance to onshore and offshore engineers to ensure delivery quality.
• Collaborate with client stakeholders to understand data needs and support timely delivery of data platform initiatives