Data Engineer

Job Description

Location UAE Dubai Abu Dhabi Employment Type Full-Time Contract Work Mode Onsite Hybrid As per client requirement

Role Overview

We are looking for an experienced Data Engineer with expertise in PySpark Python SQL Databricks and cloud data platforms to design develop and maintain scalable data pipelines and data processing solutions The ideal candidate should have hands-on experience working with large datasets developing ETL ELT pipelines implementing data transformations optimizing Spark workloads and building cloud-based data platforms

Key Responsibilities

Design develop and maintain scalable data pipelines and ETL ELT workflows
Develop high-performance data processing solutions using PySpark and Apache Spark
Build data pipelines to ingest data from databases APIs files applications and other sources
Perform data cleansing transformation aggregation and validation
Develop and optimize Spark SQL and PySpark jobs
Work with structured semi-structured and unstructured data
Implement data pipelines using Databricks and cloud-based data platforms
Optimize Spark jobs for performance scalability memory utilization and processing time
Implement data quality validation monitoring and error-handling mechanisms
Work with data lakes lakehouses and data warehouse environments
Develop reusable and scalable data engineering frameworks
Collaborate with Data Architects Data Scientists Business Analysts and application teams
Troubleshoot production data pipeline issues and perform root-cause analysis
Participate in code reviews testing deployment and production support
Maintain technical documentation for data pipelines and data architecture

Mandatory Skills

4 8 years of experience in Data Engineering
Strong hands-on experience with PySpark and Apache Spark
Strong programming experience in Python
Strong proficiency in SQL
Experience developing ETL ELT pipelines
Hands-on experience with Databricks
Strong understanding of Spark architecture DataFrames Spark SQL transformations and actions
Experience working with large-volume datasets
Good understanding of data warehousing and data lake concepts
Experience with relational and NoSQL databases
Knowledge of data pipeline performance tuning and optimization
Experience working with at least one major cloud platform Azure AWS or GCP
Cloud Data Technologies
Candidates with experience in one or more of the following will be preferred Azure Data Factory Azure Databricks Azure Data Lake Storage Azure Synapse Microsoft Fabric AWS Glue Amazon EMR Amazon S3 Redshift Kinesis GCP BigQuery Dataflow Dataproc Cloud Storage

Good-to-Have Skills

Delta Lake Lakehouse architecture Apache Kafka Apache Airflow Snowflake dbt Hadoop Hive Data modelling Data governance and data quality Docker and Kubernetes CI CD Git GitHub GitLab Terraform Experience with real-time streaming data pipelines Exposure to Generative AI ML data pipelines

Preferred Technology Stack

Programming Python SQL Big Data Apache Spark PySpark Kafka Data Platform Databricks Delta Lake Snowflake Cloud Azure AWS GCP Orchestration Airflow Azure Data Factory DevOps Git CI CD Docker Kubernetes

Candidate Profile

The ideal candidate should Have strong analytical and problem-solving skills Be experienced in designing scalable data solutions Have strong debugging and troubleshooting capabilities Understand data architecture and data lifecycle concepts Be comfortable working with large and complex datasets Have good communication and stakeholder-management skills Be able to work independently as well as collaboratively in a multicultural environment Have experience working in enterprise or large-scale data environments

Education

Bachelor s or Master s degree in Computer Science Information Technology Engineering Data Science or a related discipline

Relevant Job Titles for Sourcing

Data Engineer Senior Data Engineer PySpark Developer PySpark Data Engineer Databricks Data Engineer Big Data Engineer Cloud Data Engineer Data Platform Engineer ETL Developer Data Engineering Specialist