We are looking for a Senior Data Engineer with strong hands‑on engineering skills and a solid understanding of modern data architecture.
About the Role
You will be responsible for designing, building, and improving scalable data platforms and pipelines using Databricks and AWS. In addition to implementation, you will contribute to architectural decisions around data modelling, ingestion, storage, processing, governance, reliability, and platform scalability. This role is suitable for someone who enjoys building production systems but is also comfortable stepping back to think about the broader data platform architecture.
Responsibilities
- Design, build, and maintain scalable batch and streaming data pipelines using Databricks.
- Develop reliable ETL/ELT pipelines for ingesting data from databases, APIs, Kafka/event streams, files, and other enterprise systems.
- Design and maintain data lake / lakehouse architecture on AWS and Databricks.
- Build and optimise Bronze, Silver, and Gold data layers using Delta Lake and Medallion Architecture principles.
- Work with stakeholders and engineering teams to define appropriate data models, schemas, and data contracts.
- Participate in architectural discussions and help define standards for: Data ingestion, Data storage, Data modelling, Streaming architecture, Data quality, Data governance, Security and access control, Scalability and performance
- Design solutions that balance performance, maintainability, cost, and operational complexity.
- Build reusable data frameworks, libraries, and platform components rather than one-off pipelines.
- Improve pipeline performance through partitioning, clustering, query optimisation, caching, and efficient Spark processing.
- Implement monitoring, alerting, logging, lineage, and data quality checks across data pipelines.
- Troubleshoot production data issues and perform root-cause analysis.
- Ensure data platforms are resilient, observable, and recoverable.
- Work closely with Software Engineers, Data Analysts, BI teams, DevOps/Platform Engineers, and Solution Architects.
- Contribute to technical design reviews and provide guidance to other Data Engineers.
- Support CI/CD, infrastructure automation, testing, and release practices for data workloads.
- Continuously evaluate improvements to the data platform, engineering standards, and architecture.
Qualifications
- 5+ years of experience in Data Engineering, Data Platform Engineering, or a similar role.
- Strong hands‑on experience with Databricks and Apache Spark / PySpark.
- Strong SQL and Python development skills.
- Experience building production-grade ETL/ELT pipelines at scale.
- Strong understanding of Delta Lake and Lakehouse architecture.
- Good understanding of data modelling and data warehouse concepts.
- Experience working with AWS data services, particularly S3, IAM, Glue, and related services.
- Good understanding of distributed data processing and its performance considerations.
- Experience with orchestration, scheduling, dependency management, monitoring, and failure recovery.
- Understanding of data quality, lineage, governance, and access‑control concepts.
- Experience implementing CI/CD and software engineering practices for data pipelines.
- Ability to independently analyse technical problems and propose scalable solutions.
- Ability to explain architectural decisions and their trade‑offs clearly.
Required Skills
- Strong technical skills and remains hands‑on.
- Thinks beyond individual pipelines and considers the overall platform.
- Understands the trade‑offs between different architectural approaches.
- Builds systems that other engineers can maintain and extend.
- Takes ownership of production reliability and data quality.
- Can work independently on complex technical problems.Is comfortable challenging existing designs and proposing better approaches.
- Can translate business requirements into practical, scalable data solutions.
- Communicates effectively with both technical and non‑technical stakeholders.
Preferred Skills
- Experience with every technology is not mandatory, but strong experience with Databricks, Spark, Python, SQL, and AWS is expected.