DATA ENGINEER - AZURE DATABRICKS & PYSPARK

Duncan & Ross is hiring a Data Engineer for a leading client in Dubai. You'll design, build and run scalable data pipelines and data warehouse solutions on Azure and Databricks, turning raw data from across the business into trusted, analytics-ready data.

What you'll do

• Design, build and maintain ETL/ELT pipelines that ingest, transform and load data from multiple source systems into Azure and Databricks.
• Develop data transformations in Python and PySpark, optimising Spark jobs for performance, scalability and cost.
• Write efficient, well-structured SQL for data modelling, transformation and analysis across relational databases.
• Build and manage data warehouse and lakehouse solutions on Azure and Databricks, including Delta Lake and medallion (bronze, silver, gold) architectures.
• Orchestrate and schedule data workflows using tools such as Azure Data Factory, Databricks Workflows or Apache Airflow, with monitoring and alerting.
• Put data quality checks, validation rules and reconciliation in place so data is accurate, complete and reliable.
• Apply data governance standards, including cataloguing, lineage, access control and security, using tools such as Unity Catalog or Microsoft Purview.
• Work with data analysts, data scientists and business stakeholders to understand requirements and deliver data products.
• Troubleshoot pipeline failures and performance issues, and continuously improve reliability.
• Document pipelines, data models and processes, and follow CI/CD and version control best practices.

What you'll bring

• 3-5+ years of hands-on data engineering experience.
• Data pipelines: proven experience building ETL/ELT pipelines for batch and incremental loads.
• SQL and relational databases: advanced SQL and strong experience with databases such as SQL Server, Azure SQL, PostgreSQL or Oracle.
• Big data: hands-on Apache Spark experience, including performance tuning and handling large datasets.
• Cloud platforms: strong experience with Microsoft Azure (Data Lake Storage, Data Factory, Synapse) and Databricks.
• Data warehousing: experience designing data warehouses or lakehouses on Azure and Databricks, including dimensional modelling (star and snowflake schemas) and Delta Lake.
• Programming: strong Python and PySpark skills.
• Orchestration: experience with Azure Data Factory, Databricks Workflows, Airflow or similar.
• Data quality and governance: experience implementing data quality frameworks, data lineage, metadata management and access controls.
• Familiarity with Git, CI/CD pipelines (Azure DevOps or GitHub Actions) and Agile delivery.
• Bachelor's degree in computer science, Engineering, Information Systems or a related field.
• Fluent written and spoken English.

Nice to have

• Databricks certification (Data Engineer Associate or Professional) or Microsoft Azure Data Engineer Associate (DP-203).
• Experience with streaming data (Kafka, Event Hubs, Spark Structured Streaming).
• Experience with Unity Catalog, Microsoft Purview or Microsoft Fabric.
• Infrastructure as code (Terraform or Bicep).
• Exposure to BI tools such as Power BI.
• Previous experience in the UAE or wider GCC.

Vertical:

Technology