Data Engineer (Databricks)

Syntax Consultancy Ltd · Pride Park, Derby · 3 days ago

Category: Data engineering

aws cloudwatch databricks delta-lake hive iam parquet pyspark python s3 spark sql

An onsite Data Engineer role at Pride Park, Derby, focused on building and troubleshooting big-data pipelines in Databricks. Day to day involves PySpark/Spark SQL development, Delta Lake and Hive metastore table work, cluster configuration and performance tuning, plus hands-on AWS (S3, IAM, CloudWatch).

Salary: £? - ? per year Requirements: Strong hands-on experience with Databricks, PySpark, AWS, SQL, and Python. Databricks experience with notebooks, scheduled jobs, cluster configuration, driver and executor logs, Spark UI, and performance troubleshooting. Experience with Spark SQL, Delta Lake, Parquet, and Hive metastore tables, including schemas, partitions, and the relationship between table metadata and underlying files. Hands-on AWS experience, particularly with S3, IAM, and CloudWatch; …

Similar jobs

Data Engineer

Talent Sure Recruitment Limited · Oxford Circus, Central London

1 week ago

airflowawsci-cddata-modeling

Data Engineer - AWS, Databricks & Informatica

Synapxe · Singapore, Singapore

1 month ago

Contractanalyticsawsdata-qualitydatabricks