Data Engineer
Synthesis Software Technologies · Johannesburg, South Africa ·
- Seniority
- Lead
- Category
- Data engineering
- Experience
- 2+ years
Synthesis Software Technologies · Johannesburg, South Africa ·
Careerbox · Umhlanga Rocks, South Africa
BFN Engineering Services · Philippines
TymblHub · Negros Occidental, Philippines
Amgen · India - Hyderabad
Epignosis · Athens, Attica, Greece
Builds and supports scalable, secure batch and real-time data pipelines on AWS (EMR, EC2, S3), using Talend, Python, and Spark/PySpark to ingest, process, and visualise large datasets. Works as a technical lead in an agile team, architecting big data and BI solutions and moving data from on-premise to the cloud.
The Data Engineer’s role entails building and supporting data pipelines of which must be scalable, repeatable and secure. This role functions as a core member of an agile team whereby these professionals are responsible for the infrastructure that provides insights from raw data, handling and integrating diverse sources of data seamlessly. They enable solutions, by handling large volumes of data in batch and real-time by leveraging emerging technologies from both the big data and cloud spaces. Additional responsibilities include developing proof of concepts and implements complex big data solutions with a focus on collecting, parsing, managing, analysing and visualising large datasets. They know how to apply technologies to solve the problems of working with large volumes of data in diverse formats to deliver innovative solutions. Data Engineering is a technical job that requires substantial expertise in a broad range of software development and programming fields. These professionals have a knowledge of data analysis, end user requirements and business requirements analysis to develop a clear understanding of the business need and to incorporate these needs into a technical solution. They have a solid understanding of physical database design and the systems development lifecycle.