Senior Data Engineer - Azure Databricks
Join Euroclear
Euroclear is a financial services company that specializes in the settlement of securities transactions, as well as the safekeeping and asset servicing of these securities. We are located in Brussels and several major cities in Europe and around the world. We are deeply convinced that diversity of talents, backgrounds and opinions is a key to success, by encouraging engagement, energy and innovation.
Our IT Division
The Data & AI division ensures Euroclear's competitiveness by delivering reliable and sustainable IT solutions for the financial securities markets.
Within Data & AI division, our Data Tribe supports advanced analytics, reporting, operational data products, and AI-driven capabilities across the Euroclear Group. We help transform data into business value by leveraging modern cloud technologies, data engineering practices, and innovative analytical solutions.
Our scope can be summarized as follows:
- Operating and evolving the Data & Analytics Platform (DAP), including the Data Lake, Group Data Warehouse, and strategic Azure-based data platforms.
- Delivering data products and analytics capabilities for key business domains.
- Supporting business transformation initiatives through modern data engineering and cloud technologies.
- Providing expertise in data management, reporting, analytics, and AI.
Our tribe is working for both internal and external clients.
Your Role as a Senior Data Engineer
As a Senior Data Engineer, you will play a key role in designing, building, and evolving modern cloud-based data solutions supporting Euroclear's business domains.
You will work closely with business stakeholders, architects, product owners, and fellow engineers to deliver scalable, reliable, and high-performing data products that support operational processes, reporting, analytics, and future AI initiatives.
Key Responsibilities
- Design, develop, test, and maintain scalable data engineering solutions on Microsoft Azure and Databricks.
- Build robust and reusable ETL/ELT pipelines using Python, PySpark, and SQL.
- Develop data products and applications supporting data ingestion, transformation, enrichment, validation, and distribution.
- Implement reliable batch and streaming data workflows using Azure Databricks, Delta Lake, and modern Lakehouse architectures.
- Contribute to the modernization and migration of strategic data platforms to Azure Cloud.
- Build and maintain datasets supporting operational, analytical, management, and regulatory reporting.
- Partner with business and technology teams to translate requirements into scalable, maintainable solutions.
- Promote engineering excellence through code quality, automation, performance optimization, and operational reliability.
- Mentor and support junior engineers through coaching, code reviews, and knowledge sharing.
Azure / Databricks Data Platform
Develop solutions leveraging Azure and Databricks services, including:
- Azure Databricks
- Delta Lake / Lakehouse Architecture
- Azure Data Lake Storage
- Azure Data Factory / Synapse Pipelines
- Azure Event Hubs and Streaming Ingestion Patterns
- Azure Key Vault
- Azure DevOps
Key Responsibilities
- Design and implement curated data layers supporting analytics, reporting, operational processes, and AI use cases.
- Build and maintain scalable data models
- Apply best practices for data quality, lineage, observability, performance optimization, and operational monitoring.
- Ensure data solutions are secure, scalable, resilient, and aligned with Enterprise Architecture standards.
- Collaborate with platform, infrastructure, and architecture teams to continuously improve the data ecosystem.
ETL, Data Pipelines & Software Engineering
- Develop and maintain production-grade data pipelines using Python, PySpark, and SQL.
- Design reusable ingestion, transformation, validation, and monitoring frameworks.
- Implement automated testing, validation, logging, error handling, and monitoring.
- Optimize Spark jobs for performance, scalability, reliability, and cost efficiency.
- Apply software engineering best practices including clean code, modular design, version control, code reviews, and documentation.
- Build CI/CD pipelines using Azure DevOps YAML pipelines.
- Package, deploy, and maintain data engineering solutions across multiple environments.
- Contribute to operational support activities and continuous service improvement.
Infrastructure & DevOps Collaboration
- Collaborate with Cloud, DevOps, Platform, and Security teams to ensure operational readiness of delivered solutions.
- Contribute to infrastructure automation initiatives where required.
- Support the integration of data applications with cloud services, security controls, and deployment pipelines.
- Follow DevOps practices for release management, environment promotion, monitoring, and production support.
- Contribute to the adoption of Infrastructure-as-Code principles and cloud engineering best practices.
Collaboration & Technical Leadership
- Work closely with Business Analysts, Product Owners, Architects, Data Scientists, and Engineering teams.
- Translate business challenges into robust and scalable data engineering solutions.
- Provide technical leadership on Databricks, Spark, Cloud Data Platforms, and Data Engineering best practices.
- Contribute to architecture discussions, design standards, and reusable engineering patterns across the tribe.
- Support and coach less experienced engineers through mentoring, knowledge sharing, and technical guidance.
- Foster a culture of continuous improvement, collaboration, and engineering excellence.
Required Qualifications
- 5+ years of hands-on experience in data engineering, cloud data platform development, or large-scale data solutions.
- Strong hands-on development experience with:
- Python
- PySpark
- SQL
- ETL/ELT development
- Nice to have: Scala
- Proven experience working with the Azure Databricks ecosystem, including:
- Databricks notebooks and jobs
- Spark clusters
- Delta Lake
- Lakehouse Architecture
- Performance tuning and optimization
- Experience designing, developing, and operating production-grade data pipelines.
- Strong understanding of data modelling concepts for operational and analytical platforms.
- Experience with Azure services including:
- Azure Data Lake Storage
- Azure Data Factory
- Azure Key Vault
- Azure DevOps
- Azure monitoring and observability solutions
- Experience implementing CI/CD pipelines, preferably using Azure DevOps YAML pipelines.
- Strong understanding of software engineering principles including automated testing, version control, code reviews, and modular design.
- Knowledge of cloud security, data governance, observability, and cost-aware engineering practices.
- Experience working in complex enterprise environments with multiple stakeholders and delivery teams.
Preferred Qualifications
- Experience with large-scale distributed data processing using Apache Spark.
- Experience with streaming architectures and real-time data processing.
- Experience building reusable data engineering frameworks, libraries, and shared components.
- Knowledge of data governance, metadata management, lineage, and data quality frameworks.
- Experience supporting analytics, reporting, AI, or machine learning use cases.
- Experience contributing to cloud migration and data modernization initiatives.
- Familiarity with Infrastructure-as-Code concepts (Terraform or equivalent).
Preferred Certifications
Azure Certifications
- Microsoft Certified: Azure Data Engineer Associate
- Microsoft Certified: Azure Developer Associate
- Microsoft Certified: Azure Solutions Architect Expert
- Microsoft Certified: Azure DevOps Engineer Expert
Databricks Certifications
- Databricks Certified Data Engineer Associate
- Databricks Certified Data Engineer Professional
- Databricks Certified Developer for Apache Spark
Soft Skills
- Strong communication and documentation skills.
- Ability to translate business requirements into practical and sustainable technical solutions.
- Strong analytical and problem-solving mindset.
- Ownership, accountability, and proactive attitude.
- Team-oriented and comfortable working in a collaborative, cross-functional environment.
- Ability to influence technical decisions and drive continuous improvement.
- Passion for learning and adopting modern data engineering technologies and practices.
- Strong mentoring and coaching capabilities.
Why Join Us?
You will have the opportunity to work on some of Euroclear's most strategic data initiatives, helping modernize our data platforms and enabling the next generation of data, analytics, and AI capabilities that support key business domains across Settlement, Collateral, and Banking.
#LI-MJ1