The Data Engineer plays a key role in designing, building and optimising data solutions. This role leads and guides a team of data engineers across data architecture, ETL pipelines and business rule transformation. Expertise in data integration, processing and visualization ensures the delivery of high-quality data solutions.
Key Responsibilities:
- Lead and guide a team of data engineers as part of a project.
- Design data architecture (e.g. data warehouse, data lake) and ETL pipelines.
- Analyse business rules with users and translate them into mapping and functional specification documents.
- Implement, and lead a team in, the following:
o Developing data ingestion, integration and extraction pipelines with the data lake.
o Writing clear and concise documentation, including the business logic implemented for each required transformation.
o Troubleshooting and resolving issues with users on the implemented solution.
o Enforcing and adhering to defined processes, procedures, best practices and standards.
o Creating visualizations using modern platforms (e.g. Tableau, Microsoft Power BI), where required.
Requirements:
- Minimum 3 to 5 years of experience in data engineering or a related field.
- Bachelor's degree in Computer Science, Information Technology, Data Science or a related discipline.
- Hands-on ETL experience across design, mapping and development, adhering to standard practices.
- Talend certification is required(e.g. Qlik Talend Core Certified Data Integration Developer, or Talend Data Integration Certified Developer using Talend Studio) and exposure to Talend ETL tool compulsory to have.
- Experienced and able to perform ETL and data pipelines design and development using Talend solution.
Skills & Competencies:
- Strong grounding in database structures, theories, principles and practices, including data warehouse design.
- Preferred: experience with a major cloud or data platform (AWS, Azure, Google Cloud, Alibaba Cloud, Snowflake, Databricks).
- Preferred: knowledge of big data querying tools such as Hive and Hue.
- Preferred: experience with the Talend ETL tool.
- Good to have: Python, data APIs, unstructureddata and the Hadoop ecosystem.
- Have a diverse and thorough awareness/understanding of key enterprise data management best practices and guidelines.
- Have entry level understanding of state of the art market ETL and data management solutions such as but not limited to : Informatica, AWS Glue, Alation, Talend Data Catalog, Precisely, IBM Data Stage, Qlik Replicate etc.