Senior Data Engineer Role (Python / Spark / Databricks) - Hybrid - Contract - Banking
Role - Senior Data Engineer
Rate - £850 p/d (Inside IR35)
Duration - 6 months with very likely extension
Location - Hybrid / Liverpool Street (London) - 3 days per week in a Liverpool Street office
About the Role
Global financial institution seeking a highly skilled Senior Data Engineer to join a growing data engineering function responsible for delivering critical data platforms and analytics capabilities across the business.
This is a genuinely hands-on engineering role, requiring strong expertise in Python, Databricks, Spark/PySpark and SQL. The successful candidate will be actively involved in the design, development, optimisation and support of enterprise-scale data platforms, working alongside engineers, architects and business stakeholders to deliver robust and scalable solutions.
The role offers the opportunity to work on complex data challenges within a modern cloud-based environment, leveraging the latest data engineering technologies and best practices.
Key Responsibilities
Data Engineering & Development
- Design, develop and maintain scalable data pipelines using Python, Databricks and PySpark
- Build and optimise ETL/ELT solutions processing large-scale datasets
- Develop high-quality, reusable code and engineering frameworks
- Deliver robust, performant and maintainable data solutions
- Troubleshoot and resolve complex production and performance issues
- Support the ongoing enhancement and modernisation of enterprise data platforms
Databricks & Data Platform Engineering
- Develop and support Databricks-based data solutions and Lakehouse architectures
- Utilise Delta Lake and Spark technologies to deliver scalable data processing capabilities
- Optimise distributed data processing workloads for performance and reliability
- Implement data quality controls and monitoring solutions
- Contribute to platform stability, scalability and operational excellence
Solution Delivery
- Work closely with business and technology stakeholders to understand requirements and deliver effective solutions
- Participate in Agile delivery processes including sprint planning, refinement and estimation
- Support the delivery of strategic data initiatives across multiple business areas
- Collaborate with architects, engineers and product teams to deliver high-quality outcomes
- Identify and resolve technical risks and implementation challenges
Engineering Best Practice
- Promote software engineering best practices including testing, code reviews and version control
- Support CI/CD adoption and deployment automation
- Contribute to coding standards and engineering governance
- Drive continuous improvement across development and support processes
- Mentor junior engineers and contribute to knowledge sharing within the team
Required Skills & Experience
- Strong commercial experience as a Data Engineer or Senior Data Engineer
- Expert-level Python development skills
- Extensive hands-on experience with Databricks in enterprise environments
- Strong Apache Spark and PySpark expertise
- Advanced SQL development and optimisation skills
- Experience designing and building large-scale ETL and data integration solutions
- Strong understanding of distributed computing concepts and data processing frameworks
- Experience working with modern cloud platforms including Azure, AWS or GCP
- Knowledge of software engineering principles, testing and code quality practices
- Experience working within Agile development environments
- Excellent problem-solving and troubleshooting capabilities
Desirable Experience
- Experience within investment banking, financial services or other highly regulated environments
- Exposure to risk, trading, treasury or regulatory data domains
- Experience with Kafka, streaming technologies or real-time data processing
- Databricks certifications
- Experience with Airflow, Databricks Workflows or similar orchestration tools
- CI/CD pipeline development and DevOps practices
- Docker and Kubernetes
- Experience supporting data science or machine learning workloads
This role will suit a Senior Data Engineer who enjoys remaining close to the technology and has strong recent experience building production-grade solutions using Python, Databricks, Spark/PySpark and SQL in enterprise environments.