Senior Data Engineer : Databricks & Delta

Overview

Senior Data Engineer to design and implement scalable data pipelines in a world-leading energy trading environment. You will enable real-time and batch analytics using Databricks, Delta Lake, and streaming architectures, collaborating with cross-functional teams to deliver robust data solutions. The role focuses on handling SAP data sources and various ingestion tools to ensure data quality and reliable analytics. You will shape data platforms to support trading decisions and operational insights, contributing to best practices and documentation.

Responsibilities

• Design and implement scalable data pipelines using Databricks and Delta Live Tables (DLT)
• Develop and manage streaming data solutions for real-time analytics
• Ingest data from SAP systems using Qlik Replicate and ADF to ensure data quality and consistency
• Perform data modelling and optimize data structures for analytics and reporting
• Collaborate with cross-functional teams to understand data requirements and deliver robust solutions
• Monitor, troubleshoot, and enhance data workflows for performance and reliability
• Maintain technical documentation and contribute to data engineering best practices

Key requirements

• Strong experience with Databricks- DLT, Delta Lake concepts, UC governance
• Solid understanding of streaming technologies (e.g., Spark Structured Streaming, Autoloader)
• Programming skills in Python, SQL, or Scala
• Proficiency in data modelling, ETL/ELT processes, and data architecture
• Experience in MS Copilot, AI Foundry, Databricks (MosiacAI, MLflow, Agentbricks, Genie)
• Strong analytical background with problem-solving skills
• Performance tuning concepts like watermarking, late data handling, parallelism & checkpointing
• Hands-on expertise in ADF, and Qlik Replicate for data ingestion and replication
• Experience working in Azure cloud environments

• analytical mindset
• collaboration
• problem-solving
• Databricks
• Delta Lake
• Delta Live Tables (DLT)