Senior Data Engineer

Key Responsibilities

Data Engineering & Development

• Design and develop scalable ETL/ELT pipelines using Databricks, PySpark and SQL.

• Build batch, incremental and event-driven ingestion pipelines from multiple source systems.

• Design and implement solutions using Medallion Architecture (Bronze, Silver and Gold).

• Develop and optimise Delta Lake and Parquet-based data solutions.

• Implement reusable engineering patterns, frameworks and common components.

• Perform performance tuning and cost optimisation across Spark jobs, clusters and pipelines.

• Build and manage CI / CD processes on Azure DevOps

Data Quality & Reconciliation

• Design and implement automated data reconciliation and control frameworks.

• Implement completeness, accuracy, balancing and data-quality controls across source-to-target pipelines.

• Investigate data discrepancies and perform root-cause analysis.

• Ensure appropriate auditability, lineage and traceability throughout the data lifecycle.

Solution Design & Architecture

• Translate business requirements into scalable technical designs and data products.

• Contribute to solution architecture, data standards and engineering patterns.

• Evaluate existing solutions and define appropriate as-is / to-be designs.

• Ensure solutions align with enterprise architecture, information security, governance and control standards.

Operational Excellence

• Monitor production pipelines and proactively resolve failures, data incidents and performance issues.

• Perform root-cause analysis and implement permanent remediation.

• Support production releases through established change-management and deployment processes.

• Develop and maintain monitoring, alerting, logging, runbooks and operational documentation.

• Manage and support Databricks job schedules to ensure critical business processes run successfully and issues are resolved timeously.

Technical Leadership

• Provide technical leadership and mentorship to Data Engineers.

• Conduct code reviews and enforce engineering standards, testing and development best practices.

• Promote reusable solutions rather than point-to-point development.

• Support technical design reviews and challenge solutions where appropriate.

• Drive continuous improvement across engineering practices, automation and platform efficiency.

Stakeholder & Business Engagement

• Work with business and technical stakeholders to understand data and reporting requirements.

• Translate insurance business requirements into appropriate data models and engineering solutions.

• Communicate technical issues, dependencies, risks and delivery status clearly to stakeholders.

• Partner with architecture, governance, security, operations, finance and business teams.

Minimum Requirements

Bachelor's degree in Computer Science, Computer Engineering, Information Technology, Data Science or a related discipline. Relevant Databricks and cloud certifications are advantageous.

Experience

Ideally 5+ years of Data Engineering / ETL development experience, with demonstrated experience delivering enterprise-scale data solutions and operating production data pipelines.

Core Technical Skills

Advanced PySpark and SQL; strong Databricks experience; Delta Lake and Parquet; Medallion Architecture; ETL/ELT and ingestion pipelines; data modelling; reconciliation and data-quality frameworks; Git/Azure DevOps or equivalent CI/CD; cloud data platforms, preferably AWS; performance optimisation; and production troubleshooting.Advantageous Skills

Databricks Workflows/Jobs, Unity Catalog, Auto Loader, CDC or similar incremental ingestion patterns, orchestration frameworks, CI/CD automation, AWS services, and exposure to AI/ML data workloads.

Industry Experience

Insurance industry experience is highly advantageous, particularly exposure to policy, claims, premium, finance / GL, actuarial or regulatory data.

Business & Behavioural Competencies

• Strong problem-solving and analytical capability.

• Engineering discipline, ownership and accountability.

• Ability to work independently in a complex enterprise environment.

• Strong stakeholder management and communication skills.

• Ability to coach and uplift the capability of less experienced engineers.

• Commercial awareness and a focus on sustainable, cost-effective engineering solutions.