Data engineer in GiG's Data department in Barcelona, building and maintaining real-time big data ETL pipelines, integrations, and high-performance services to process and analyse large datasets, plus troubleshooting live production systems. Core stack: Spark, Kafka, Nifi, ClickHouse, SQL, and Python/Java.
GiG’s Data department develops and supports real-time big data projects, using information to support strategic decision-making and organisational growth.
Задачи:
Design, create, and maintain ETL processes, data integrations, and data pipelines
Ensure seamless handling of large volumes of data
Develop high-performance services to process, store, and analyse large datasets using optimised code
Design and implement data validation and quality checks tailored to business needs
Troubleshoot live production systems and resolve issues
Test and implement new technologies relevant to current and future projects
Review code, promote best practices, and implement robust solutions across big data projects
Требования:
Strong expertise in big data technologies such as Spark, Kafka, Nifi, ClickHouse, and big data datastores
Strong understanding of schema design methodologies and data models
Expertise in extracting large datasets using SQL and familiarity with SQL databases
Object-oriented programming skills, ideally in Python and/or Java
Strong experience with cloud computing and/or distributed systems