Job Summary
We are looking for a highly skilled Senior Data Engineer with 6 to 9 years of hands-on experience in SQL, Python for data engineering, Azure data engineering services, and Snowflake analytical layer development. The ideal candidate should have strong expertise in designing, developing, optimizing, and maintaining enterprise-grade data pipelines, scalable data warehouses, data marts, and analytical models.
The candidate should be capable of leading technical implementations, mentoring junior engineers, collaborating with architects and business stakeholders, and driving best practices across data engineering initiatives. Experience with DBT, CI/CD, DevOps practices, and performance optimization is highly preferred.
Key Responsibilities
Core Duties
- Understand business requirements, architecture documents, and translate them into scalable technical solutions.
- Lead end-to-end development of Azure Data Factory (ADF) pipelines and enterprise data workflows.
- Design and develop scalable ELT/ETL frameworks using Azure services and Snowflake.
- Build and optimize analytical layers, dimensional models, data marts, and semantic data structures in Snowflake.
- Create reusable, maintainable, and performance-optimized SQL and Python-based data transformation frameworks.
- Drive data quality, validation, reconciliation, and governance processes across data platforms.
- Conduct detailed code reviews, design reviews, and ensure engineering best practices are followed.
- Troubleshoot complex production issues, optimize pipeline performance, and reduce data processing latency.
- Participate actively in solution architecture discussions and provide technical recommendations.
- Lead technical demos, walkthroughs, sprint reviews, and stakeholder presentations.
- Mentor junior and mid-level engineers and guide them in technical problem-solving.
- Contribute to framework development, automation initiatives, and process standardization.
- Support release management, CI/CD deployment processes, and environment migrations.
- Collaborate with cross-functional teams including business analysts, architects, QA teams, DevOps, and product owners.
Technical Skills or Knowledge Areas
Mandatory Skills
- Advanced SQL (complex joins, CTEs, window functions, query optimization, stored procedures)
- Python for data engineering and automation
- Azure Data Engineering Services:
- Azure Data Factory (ADF)
- Azure Data Bricks
- Azure Data Lake
- Azure Functions
- Azure Synapse Analytics
- Logic Apps
- Key Vault
- Snowflake Data Warehouse Development
- Data Modeling (Star Schema, Snowflake Schema, Dimensional Modeling)
- Performance tuning and optimization
- CI/CD and DevOps for data engineering
- Git/version control systems
Preferred Skills
- DBT (Data Build Tool)
- Apache Spark / PySpark
- Airflow
- Terraform or Infrastructure as Code
- Data governance and metadata management
- API integration and real-time data ingestion
- Monitoring and observability frameworks
Collaboration & Leadership
- Strong experience working in offshore-onshore delivery models.
- Ability to lead technical discussions and coordinate across multiple teams.
- Familiarity with Agile/Scrum methodologies.
- Hands-on experience with:
- Slack
- JIRA/Kanban
- Confluence
- Azure DevOps
- Ability to communicate effectively with technical and non-technical stakeholders.
- Provide mentorship and technical guidance to junior team members.
- Drive collaboration and foster a positive engineering culture.
Process Improvement
- Identify opportunities for automation, optimization, and engineering standardization.
- Improve reusable framework components and reduce operational overhead.
- Participate in architecture reviews and suggest scalable design improvements.
- Handle change requests, impact analysis, and production support activities efficiently.
- Contribute to engineering documentation standards and knowledge-sharing initiatives.
Compliance & Documentation
- Create and maintain High-Level Design (HLD) and Low-Level Design (LLD) documents.
- Prepare technical architecture diagrams and data flow documentation.
- Create operational runbooks, support documentation, and stakeholder training materials.
- Ensure compliance with enterprise security, governance, and audit requirements.
- Maintain documentation for data lineage, transformations, and business rules.
Required Qualifications
Education & Certifications
- BE / BTech / MCA / MTech in Computer Science, Information Technology, or related field.
- Azure Data Engineering certifications preferred:
- Microsoft Certified: Azure Data Engineer Associate
- Snowflake Certifications are a plus.
Experience
- 6 to 9 years of experience in Data Engineering.
- Strong enterprise experience in:
- SQL
- Python
- Azure Data Engineering ecosystem
- Snowflake Data Warehousing
- Experience handling large-scale enterprise data platforms and complex transformation pipelines.
Performance Metrics
- Timely and high-quality delivery of user stories and technical deliverables.
- Ability to independently own medium to high-complexity technical implementations.
- Ability to lead solutioning discussions and provide scalable recommendations.
- Strong troubleshooting and problem-solving capabilities.
- Effective stakeholder communication and status reporting.
- Contribution towards process optimization and engineering improvements.
- Mentoring effectiveness and team collaboration.
- Ability to adapt quickly in fast-changing technical environments.
- Ensure high reliability, scalability, and performance of data solutions.
Additional Expectations for Senior Role
- Ability to act as technical lead for modules/projects.
- Drive engineering best practices and coding standards.
- Participate in hiring and technical interviews.
- Guide architecture decisions and technology evaluations.
- Ownership mindset with proactive risk identification and mitigation.
- Strong business understanding and ability to align engineering solutions with business goals.