Senior Data Engineer

We are looking for a Senior Data Engineer to join a fast-paced, data-driven environment on a contract basis. This role is ideal for an engineer with deep hands-on expertise across the Azure and Databricks ecosystem who can design, build, and optimize enterprise-scale data platforms. You'll play a key role in modernizing ETL/ELT processes, strengthening data governance, and driving cost and performance efficiency across the Databricks Lakehouse.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Azure and Databricks to support enterprise data and analytics needs
• Build and manage Delta Live Tables (DLT) pipelines for reliable, automated data transformation workflows
• Implement and maintain Unity Catalog for centralized data governance, access control, and lineage tracking across the Lakehouse
• Design and develop semantic layers to support consistent, business-friendly access to curated data for analytics and BI consumption
• Build and maintain CI/CD pipelines for data engineering workflows, ensuring automated testing, deployment, and version control best practices
• Own FinOps practices within Databricks — monitoring, forecasting, and optimizing compute/storage costs across clusters and jobs
• Conduct performance tuning and optimization of Databricks workloads, including cluster configuration, query optimization, and job orchestration
• Work across core Azure data services (e.g., Azure Data Factory, Azure Data Lake Storage, Azure Synapse) to integrate and orchestrate data pipelines
• Partner with data architects, analysts, and business stakeholders to translate requirements into robust, scalable data solutions
• Ensure data quality, security, and compliance standards are upheld across all pipelines and platforms
• Troubleshoot and resolve pipeline failures, performance bottlenecks, and data quality issues in a timely manner

Required Skills & Experience

• Proven experience as a Senior Data Engineer, with a strong track record delivering production-grade data pipelines
• Strong hands-on experience with Microsoft Azure (Azure Data Factory, ADLS, Azure Synapse, or similar)
• Deep expertise in Databricks, including:
• Unity Catalog
• Cluster and job performance optimization
• FinOps / cost management within Databricks
• Strong experience building and maintaining CI/CD pipelines for data engineering workflows (e.g., Azure DevOps, GitHub Actions)
• Experience designing and implementing semantic layers for analytics/BI consumption
• Solid background in ETL/ELT pipeline design, development, and optimization
• Proficiency in Python and/or SQL for data transformation and pipeline development
• Strong understanding of data governance, security, and lineage best practices
• Excellent problem-solving skills and ability to work independently in a contract setting
• Strong communication skills, with the ability to work cross-functionally with technical and business stakeholders

Nice to Have

• Databricks certifications (e.g., Databricks Certified Data Engineer Professional)
• Experience with Terraform or other infrastructure-as-code tools in an Azure/Databricks context
• Experience in a regulated or enterprise-scale industry
• Familiarity with Spark internals and advanced performance tuning