Position Description
We are seeking a Senior Data Engineer to lead the design and implementation of end-to-end data solutions within a modern Azure-based cloud environment. This hands-on, technical role will partner closely with data scientists, analysts, and business stakeholders to ensure that data is accurate, accessible, and optimized for analytics, reporting, and operational needs. You will play a key role in building Data Lakehouse, scalable data pipelines and integrations while championing best practices, mentoring junior engineers, and leading strategic data initiatives across teams.
Key Responsibilities
- Architect and implement secure, scalable data pipelines using Azure Data Factory, Azure Functions, and Azure Data Lake Storage
- Design, develop, and maintain scalable ETL/ELT pipelines using Azure Databricks (PySpark/Spark SQL) to ingest, transform, and process large volumes of structured and unstructured data
- Build and manage data pipelines using Azure services such as Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage (ADLS Gen2), Azure Event Hubs, and Azure SQL Database
- Implement data quality, validation, and monitoring frameworks to ensure reliability and accuracy of data pipelines
- Collaborate cross-functionally with data scientists, analysts, and business stakeholders to understand data requirements and deliver fit-for-purpose data solutions
- Lead modernization and optimization efforts, improving pipeline performance, maintainability, and scalability
- Support CI/CD practices for data pipelines using tools such as Azure DevOps, Git, and Databricks Repos
- Detect and resolve data quality issues; implement automated audits and monitoring processes
- Troubleshoot and resolve production data pipeline issues, ensuring high availability and minimal downtime
- Act as a technical leader on schema design, performance tuning, and Azure data architecture
- Mentor and support junior engineers across data engineering, analytics, and BI teams
- Participate in and lead code reviews, promoting clean, well-documented, and testable code
- Stay current on trends in data engineering and cloud technologies, identifying opportunities to innovate
Minimum Requirements
- 8+ years of hands-on experience in data engineering, including designing and implementing enterprise-scale data solutions.
- 2+ years of experience developing and operating Azure cloud-native data platforms.
Critical Skills
- Expertise in MS SQL Server, Python (pandas, PySpark), Azure Data Factory, Azure Functions and Azure Data Lake Storage.
- Strong expertise with Azure Databricks, including Spark (PySpark/Scala), Delta Lake, and cluster/job optimization.
- Solid understanding and hands-on experience building Data Lakehouse architecture
- Experience working with a variety of file formats (e.g., CSV, JSON, XML, Parquet).
- Experience with version control (Git) and CI/CD pipelines for data engineering workflows
- Familiarity using REST APIs for data extraction and integration.
- Proven experience designing and implementing data solutions.
- Strong understanding of cloud architecture, data warehousing and modern data stack components.
Additional Skills & Qualifications
- Demonstrated ability to perform root cause analysis on data and processing issues
- Strong problem-solving skills with the ability to explain technical concepts to non-technical audiences
- A successful history of manipulating, processing and extracting value from large disparate datasets
- Experience with Big Data technologies such as Databricks, Spark, or Azure Synapse
- Knowledge of CI/CD workflows, version control, and agile development practices
- Familiarity with data governance, privacy, and compliance frameworks
- Experience with data warehousing, analytics tools, and BI platforms
- Familiarity with streaming data technologies (Azure Event Hubs, Kafka, Structured Streaming)
Education
- 4-year degree in computer science, engineering or other related IT field of study, or equivalent professional work experience
Physical Requirements
- General office demands
- Prolonged periods of sitting at a desk and working on a computer.
- Frequent reaching, handling, and fine manipulation for using office equipment, filing, and managing paperwork.
- Manual dexterity sufficient to operate a keyboard, mouse, and other office tools.
- Occasional standing, walking, and bending.
- Ability to lift up to 10-20 pounds occasionally.
- Vision abilities required include close vision for computer work and reading documents.
- Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.