Job Title
Data Platform Engineer — Sandton, Johannesburg (3 days per week onsite)
Duration: 12 months (renewal)
Description
Purpose: Support the design, build, and operation of the Data Platform, blending infrastructure engineering with data engineering practices to keep the Databricks and Azure environment reliable, automated, and self-service for platform users.
Responsibilities
- Design, implement and maintain CI/CD pipelines (GitHub Actions, Azure DevOps or similar) for infrastructure and application deployments.
- Write and maintain Terraform infrastructure-as-code for Azure services and Azure Databricks, including Unity Catalog, workspaces, networking, and compute policies.
- Develop and support platform tooling and automation using Python and/or .NET.
- Build and maintain shared libraries or modules consumed by data engineers and analysts across the platform.
- Design and implement management pipelines and processes for platform concerns such as cost tracking, governance, provisioning, and monitoring.
- Collaborate with data engineers, analysts, security, and infrastructure teams to translate requirements into platform capabilities and secure designs.
- Troubleshoot infrastructure, pipeline, and integration issues, including networking, authentication, and connectivity problems.
- Contribute to platform documentation, runbooks, and best-practice standards to enable self-service for platform users.
Qualifications
- Solid hands-on experience with Microsoft Azure infrastructure components (Entra ID, Key Vault, networking, storage).
- Practical experience with Azure Databricks, including Unity Catalog and cluster/job management.
- Proven experience authoring Terraform in multi-environment setups for reproducible deployments.
- Proficiency in Python; working knowledge of SQL. .NET experience is a strong advantage.
- Experience building and maintaining CI/CD pipelines using GitHub Actions, Azure DevOps, or similar tools.
- Good understanding of data platform governance, cost optimization, security controls, and monitoring practices.
- Strong troubleshooting skills across networking, authentication, and pipeline integrations.
- Excellent collaboration and documentation skills for cross-functional platform delivery.
Skills
- Microsoft Azure
- Entra ID
- Azure Key Vault
- Azure Networking
- Azure Storage
- Azure Databricks
- Unity Catalog
- Terraform
- Infrastructure as Code
- Python
- .NET
- SQL
- GitHub Actions
- Azure DevOps
- CI/CD
- DevOps
- Monitoring
- Cost Management
- Governance
- Networking troubleshooting
- Authentication and Authorization
- Automation
. Skillset Required: Microsoft Azure, Entra ID, Azure Key Vault, Azure Networking, Azure Storage, Azure Databricks, Unity Catalog, Terraform, Infrastructure as Code, Python, .NET, SQL, GitHub Actions, Azure DevOps, CI/CD, DevOps, Monitoring, Cost Management, Governance, Networking troubleshooting, Authentication and Authorization, Automation