Data Engineer at Boardroom Appointments – Cape Town | Jobyssey

Hybrid position

Minimum requirements:

Relevant 3-year tertiary qualification BSC In Computer Science / Information Systems
2-4 Years Senior Data Engineering experience

In depth knowledge and understanding of:

Azure Databricks
Data governance and data management frameworks
BI and Warehouse design
Movement of data into the Cloud using the following tools Apache Kafka, Storm, Flume for ingesting data or Amazon Web Services (AWS) Cloud Development Kit (CDK)
Operating with real-time streams, data warehouse queries, JSON, CSV, raw data
Scripting Data Pipelines for scheduled movement of data
Designing and developing ETL/ELT processes and data pipelines Experience working with Azure DevOps
Data visualization including PowerBI / Tableau
Exploratory Data Analysis - EDA
SQL
Microsoft SQL Server
NoSQL
Microsoft SQL Server
Python (PySpark)
C# , Json- calling APIs
PowerShell
Apache Spark
Kafka
Scala
Splunk
Elk Stack
Data Modelling
Data warehousing solutions
Data pipelines
Data Cleansing
Terraform

Responsibilities:

Work as part of an agile data engineering team
Ensure that data systems meet the companys regulatory commitments, and that data is reliable and efficient
Support best business capabilities for high performance database solutioning
Actively participate in team, cross-discipline and vendor-driven collaboration sessions or forums to increase understanding of the working environment by contribution and participation in the relevant Data Governance Frameworks
Partner with Technology, Business and other stakeholders to ensure that database structures, programmes and applications meet business delivery requirements
Monitor adherence to processes which support the prescribed data architectural frameworks and ensure development/delivery teams align to the required standards and methodologies
Design and implement scalable end-to-end database solutions including:
Addressing issues of data migration i.e. validation, clean-up and mapping and consistently apply data dictionaries
Data cleanse using databricks
Component design and development
Identification and resolution of production and application development constraints
Integration of new technologies and software into the existing landscape
Development data set processes for data modelling, mining and production
Optimal utilization of big data
Documentation management and disaster recovery

Data Engineer

Similar jobs