Senior Data Engineer - Databricks Lakehouse & AI Pipelines

• Act as the leading technical authority and Subject Matter Expert for Databricks Data Engineering, representing the Engineering team to stakeholders.
• Define and own the Data Architecture for the creation of speciality insurance analytics data products.
• Design Conceptual, Logical, and Physical Data Models for the business area, aligned to broader/standardised data models.
• Establish engineering standards, patterns, and best practices for AI-assisted pipeline development, orchestration, processing, storage, and observability.
• Design, oversee, and contribute to the development of large-scale, fault-tolerant data pipelines and orchestration jobs across medallion layers in Databricks.
• Ensure pipelines are designed to meet SLAs for reliability, latency, cost, and data quality.
• Implement processes for the management of metadata, lineage, and observability in the delivery of high-quality and dependable Data Products.
• Partner with security teams on data privacy and compliance, implementing data governance with particular focus on managing sensitive, business critical data.
• Mentor senior and mid-level engineers and drive Engineering Excellence across AI Assisted Development, Code quality, Testing, and Documentation.
• Lead design reviews and architecture discussions, encouraging collaboration and innovation to promote a culture of continuous improvement.

Requirements

• Databricks and AWS Certifications
• BSc/MSc degree in Computer Science, Engineering or a related subject
• Data Management certifications (e.g. DAMA) an advantage
• Extensive experience in Data Engineering roles
• Proven experience leading or mentoring engineering teams
• Experience delivering large-scale cloud-based data initiatives
• Previous experience working within regulated environments such as Insurance highly desirable
• Experience with Databricks Lakehouse architecture preferred

Core Competencies

Demonstrates expertise in Databricks Data Engineering and Data Architecture, with a strong focus on designing and implementing large-scale data pipelines and ensuring data quality and compliance. Proven ability to mentor engineering teams and establish best practices for AI-assisted development.

Highest-signal resume keywords

• Databricks Data Engineering
• Data Architecture
• Large-Scale Data Pipelines
• AWS Certifications
• Data Management Certifications

ATS Optimization Keywords

Hard Skills

• Data Modeling
• Data Pipeline Development
• Data Governance
• Metadata Management
• Data Quality Assurance

Soft Skills

• Mentoring
• Collaboration
• Continuous Improvement

Certifications & Qualifications

• Databricks Certification
• AWS Certification
• Data Management Certification

Industry Keywords

• Speciality Insurance
• Regulated Environments
• Data Privacy
• Compliance

Tools & Technologies

• Databricks Lakehouse
• AI-Assisted Development
• Cloud-Based Data Solutions