Data Engineer - Azure Data Marketplace Platform
Belvedere · Poland ·
- Employment
- Contract
- Category
- Data engineering
- Experience
- 8+ years
Belvedere · Poland ·
Belvedere Recruitment · Warszawa, Poland
Belvedere Recruitment · Warszawa, Poland
Belvedere Recruitment · Warszawa, Poland
Zohorecruit · Netherlands
A 3-month, hybrid contract in Warsaw for a Manager-level Data Engineer to build and maintain data ingestion pipelines for an internal AI data marketplace platform. The core stack is Azure Data Factory, Azure Databricks, Apache Spark/PySpark, Python and SQL, focused on stable, fault-tolerant ETL.
Location: Warsaw
Contract Type: 3 Months
Job type: Hybrid
Work on cutting-edge AI data marketplace technology at Manager level
Build world-class data products using Azure Data Factory and Databricks
Join cross-functional teams focused on stability, optimisation and innovation
Leverage 8+ years of experience in a role that values resilience and scalability
Our client is a global professional services organisation operating at the forefront of data engineering and AI technology. They are building an advanced data marketplace platform designed to deliver robust, scalable data assets to consumers across the business. With a focus on innovation, collaboration and technical excellence, they provide an environment where experienced engineers can contribute to meaningful projects that shape the future of data-driven decision making.
They are seeking an experienced Data Engineer at Manager level to join their team in Warsaw. This contract role offers the opportunity to work on sophisticated data ingestion pipelines within their AI Data Marketplace, collaborating with talented professionals to build resilient, fault-tolerant systems.
As a Data Engineer, you will play a central role in developing and maintaining the data ingestion pipelines that power their AI Data Marketplace platform. You will work closely with cross-functional teams to design, build and optimise data assets, ensuring stability, traceability and high availability. Your expertise in Azure technologies, big data processing and ETL will directly contribute to delivering world-class data products and services to internal consumers.
Identify opportunities to improve data assets and services within the AI Data Marketplace
Design and develop data ingestion pipelines with focus on stability and optimisation
Build, test and maintain robust data pipelines and processes
Write resilient, fault-tolerant and modular code for high availability systems
Research and implement new methods to enhance data products
Collaborate across teams to ensure smooth transitions through ingestion and consumption phases
Manage Azure Data Factory pipelines, activities, triggers and data flows
Process large datasets using Azure Databricks, Apache Spark and PySpark
Transform and integrate data from various sources using ETL processes
8-10 years of experience as a Data Engineer
Expert knowledge of Azure Data Factory, including pipeline development and management
Proficiency in Azure Databricks for data processing with notebooks, jobs and clusters
Strong programming skills in Python, PySpark and SQL
Experience with Apache Spark for big data processing
Solid understanding of ETL processes and data transformation
Familiarity with Azure storage solutions including Azure SQL Database, Blob Storage and Data Lake Storage
Knowledge of data warehousing principles and scalable data modelling techniques
Experience integrating multiple data sources in cloud environments
Ability to write modular, testable code for production systems
While specific benefits have not been disclosed, you will join a collaborative environment where technical excellence is valued and encouraged. You'll work alongside skilled professionals on meaningful projects that advance data engineering capabilities, with opportunities to research and implement innovative solutions that shape the future of their data marketplace platform.
Dataeconomy · Hyderabad, India