- Build Scalable Pipelines: Develop and implement robust, highly scalable data pipelines using Azure Databricks, Azure Data Factory, and other modern Azure tools.
- Design Medallion Architecture: Design and optimize enterprise data storage solutions by implementing a Medallion Architecture (Bronze, Silver, Gold) within Azure Data Lake to efficiently support complex enterprise reporting.
- Execute Advanced Data Transformations: Perform data modeling and create highly efficient data architectures. Competency in PySpark and Spark SQL is a must-have requirement.
- Collaborate Across Teams: Proven experience collaborating with cross-functional teams to translate business data requirements into seamless, integrated technical solutions.
- Production Support & Monitoring: Demonstrate the strong ability to support and monitor ETL/ELT processes in a live production environment, proactively troubleshooting pipelines to ensure accuracy, reliability, and minimal downtime.
- Ensure Data Security: Implement stringent data security measures and access controls to protect sensitive enterprise information.
- Mentor & Document: Document technical processes, establish standard operating procedures, and provide knowledge-sharing and training sessions to the team.
- Drive Innovation: Stay continuously updated with the latest Azure technologies, Databricks advancements, and best practices in modern data engineering.
WHAT IS THE JOB LIKE?
This role acts as the core architect and protector of the company’s data infrastructure. The day-to-day work breaks down into four main areas:
1. The Builder (Focus: ~40%)
The engineer uses Azure Data Factory to extract messy data from various sources and leverages the computing power of Azure Databricks (using PySpark) to clean and transform it. They organize this data into structured, highly efficient architectures—often using a Bronze, Silver, and Gold Medallion setup—within Azure Data Lake or Azure SQL Database. Building this efficiently ensures analysts can run fast enterprise reports while keeping the company's computing costs low.
2. The Firefighter (Focus: ~30%)
Data pipelines occasionally break due to source changes, server timeouts, or corrupted files. The engineer monitors these automated systems, investigates errors, and quickly fixes broken pipelines so business dashboards stay updated.
3. The Translator (Focus: ~15%)
The engineer collaborates regularly with business stakeholders and analysts. They listen to business goals (like tracking real-time sales) and design the technical steps required to find, move, and deliver that specific data seamlessly.
4. The Guardian & Teacher (Focus: ~15%)
The engineer protects the company's data by setting up strict access controls and encrypting sensitive information. As a teacher, they also document technical processes, provide knowledge transfer, and train team members on data systems and best practices.
WHO ARE YOU?
- Experience & Education: Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field (or equivalent practical experience).
- 5+ years of proven experience working as a Data Engineer, with a strong focus on the Microsoft Azure ecosystem.
- Core Technical & Architectural SkillsMust-Have: Deep, hands-on competency in PySpark and Spark SQL for complex data processing and transformation.
- Extensive experience developing and orchestrating scalable data pipelines using Azure Databricks and Azure Data Factory (ADF).
- Proven track record of designing and implementing a Medallion Architecture (Bronze, Silver, Gold) within Azure Data Lake to support downstream enterprise reporting and analytics.
- Solid proficiency in data modeling and optimizing database solutions, including Azure SQL Database.
Operational Expertise
- Demonstrated ability to actively monitor, support, and troubleshoot ETL/ELT pipelines in a live production environment.
- Experience identifying performance bottlenecks, analyzing error logs, and ensuring pipeline reliability and accuracy with minimal downtime.
- Strong understanding of data security best practices, including implementing role-based access control (RBAC) and protecting sensitive enterprise data.
- Worked on SAP systems preferred but not required
- Proven experience collaborating with cross-functional teams (business stakeholders, analysts, and software engineers) to translate business requirements into robust technical solutions.
- Excellent written and verbal communication skills, with the ability to document complex technical processes clearly.
- A collaborative mindset with a willingness to lead knowledge-sharing sessions and mentor team members.