Principal Data Engineer

Responsibilities

• Act as the leading technical authority and Subject Matter Expert for Databricks Data Engineering, representing the Engineering team to stakeholders.
• Define and own the Data Architecture for the creation of speciality insurance analytics data products.
• Design Conceptual, Logical, and Physical Data Models for the business area, aligned to broader/standardised data models.
• Establish engineering standards, patterns, and best practices for AI-assisted pipeline development, orchestration, processing, storage, and observability.
• Design, oversee, and contribute to the development of large-scale, fault-tolerant data pipelines and orchestration jobs across medallion layers in Databricks.
• Ensure pipelines are designed to meet SLAs for reliability, latency, cost, and data quality.
• Implement processes for the management of metadata, lineage, and observability in the delivery of high-quality and dependable Data Products.
• Partner with security teams on data privacy and compliance, implementing data governance with particular focus on managing sensitive, business critical data.
• Mentor senior and mid-level engineers and drive Engineering Excellence across AI Assisted Development, Code quality, Testing, and Documentation.
• Lead design reviews and architecture discussions, encouraging collaboration and innovation to promote a culture of continuous improvement.

Requirements

• Databricks and AWS Certifications
• BSc/MSc degree in Computer Science, Engineering or a related subject
• Data Management certifications (e.g. DAMA) an advantage
• Extensive experience in Data Engineering roles
• Proven experience leading or mentoring engineering teams
• Experience delivering large-scale cloud-based data initiatives
• Previous experience working within regulated environments such as Insurance highly desirable
• Experience with Databricks Lakehouse architecture preferred