Microsoft Fabric Specialist / Data Engineer

About the Role

An established insurance company is seeking to hire a highly skilled and experienced Data Engineer to join their team. The role is critical for building and maintaining the robust data infrastructure that powers AI and machine learning initiatives. The ideal candidate possesses a strong background in data engineering principles, experience with big data technologies, and a passion for enabling data-driven innovation through reliable and efficient data systems.

About the Role

Our client is seeking a meticulous and experienced AI Data Engineer to join their team in Soweto. This role is critical for building and maintaining the robust data infrastructure that powers our AI and machine learning initiatives. You will be responsible for designing, developing, and optimizing data pipelines, ensuring data quality, availability, and accessibility for data scientists and ML engineers.

Key Responsibilities

• Design and build scalable and reliable data pipelines for AI/ML workloads.
• Develop ETL/ELT processes to ingest, transform, and load data from various sources.
• Ensure data quality, integrity, and security across all data platforms.
• Collaborate with data scientists and ML engineers to understand data requirements and provide solutions.
• Optimize data infrastructure for performance, cost, and scalability.
• Implement data governance policies and best practices.

Responsibilities

• Build and maintain ETL pipelines supporting a multi-tenant data platform, ingesting data from APIs, databases, and event sources.
• Build and maintain Data Platform APIs that allow teams to ingest, process, and access data easily and reliably.
• Implemented tenant-specific logic by following existing configuration and naming conventions.
• Apply tenant-level data isolation using schemas, partitions, or access controls.
• Build models from existing templates used for financial and operational reporting. Develop models for analytics and reporting, maintaining consistency with shared data models.
• Monitor scheduled pipelines, investigate failures, and resolve data quality issues and inconsistencies.
• Maintain daily and incremental data loads into the data warehouse.
• Assist with onboarding new clients by validating source data and testing pipeline outputs.
• Work closely with senior data engineers to learn patterns for multi-tenant data isolation.
• Collaborate with analytics, product, and customer facing teams to understand reporting needs.
• Support strict regulatory and audit requirements by following data handling, retention, and audit guidelines.
• Handle financial and sensitive data (PII) according to company policies and regulatory standards (e.g., POPIA).
• Apply least-privilege access and role‑based access controls, and support data protection through masking, encryption, and established security standards.

Requirements

• Bachelor's degree in Computer Science, Engineering, Statistics, or a related quantitative field.
• 3+ years of experience in data engineering, with a focus on big data technologies.
• Proficiency in programming languages such as Python, SQL, or Scala.
• Experience with big data platforms like Spark, Hadoop, or similar.
• Familiarity with cloud data services (AWS, Azure, GCP) and data warehousing concepts.
• Strong understanding of data modeling, database design, and data architecture.
• Relevant Microsoft Fabric and/or Azure data certification.
• Strong hands‑on experience with Microsoft Fabric, including OneLake, Lakehouse, Warehouse, Data Pipelines/Data Factory, notebooks and Dataflows Gen2.
• Strong SQL Server and T‑SQL capability, including complex query development, schema design, indexing and performance optimisation.
• Practical experience developing, maintaining and supporting production ETL/ELT pipelines.
• Experience integrating and extracting data from REST/SOAP APIs, databases, flat files and other structured or unstructured data sources.
• Proficiency in data transformation using SQL and Python/PySpark, with an understanding of scalable data processing practices.
• Practical experience in data warehousing, dimensional modelling, incremental loading, orchestration and schema evolution.
• Experience with troubleshooting pipeline failures, data quality issues and performance bottlenecks, including the ability to restore service efficiently.
• Experience with source control, CI/CD and deployment practices using Git, Azure DevOps or equivalent tools.
• Experience supporting Power BI and other downstream analytical or reporting requirements.
• Demonstrated ability to take ownership of an existing technical environment with limited hand‑holding.
• Strong documentation, communication and stakeholder engagement skills.

Benefits

• Competitive salary and comprehensive benefits package.
• Opportunities for professional growth and training in AI and data technologies.
• Access to modern data infrastructure and cutting‑edge tools.
• A collaborative and innovative work environment in Soweto.
• Hybrid work model offering flexibility.

Benefits

• Competitive salary and performance-based bonuses.
• Comprehensive medical aid and retirement benefits.
• Opportunities for professional development and training in cutting‑edge data technologies.
• Hybrid work model offering flexibility and team collaboration.
• Contribute to critical data infrastructure that powers advanced AI capabilities.

Benefits

• Competitive salary and performance bonuses.
• Comprehensive health and retirement benefits.
• Generous paid time off and leave policies.
• Hybrid work environment offering flexibility.
• Professional development opportunities and access to training in Vereeniging.

Benefits

• Competitive salary and comprehensive benefits package.
• Opportunity to work on impactful AI and data initiatives.
• Professional development and continuous learning opportunities.
• Hybrid work model providing flexibility and work‑life balance.
• Dynamic team environment in the heart of Sandton.

Benefits

• Competitive salary and comprehensive benefits package.
• Opportunity to work on impactful AI and data initiatives.
• Professional development and continuous learning opportunities.
• Hybrid work model providing flexibility and work‑life balance.
• Dynamic team environment in the heart of Sandton.