Lead Data Engineer (Azure/Databricks)

Overview

Location: Sheffield (2 days per week/ Hybrid)

Reporting to: VP of Engineering

The Team: 2 Data Scientists, 1 Data Engineer

IntelliAM is an intelligent asset management company specialising in industrial IoT data. We ingest high-frequency factory and sensor data, apply machine learning techniques, and present high value actionable insights to analysts and customers.

We are building a cutting-edge UNS aligned Industrial IoT data platform with a highly scalable architecture to handle our next phase of growth. We require a delivery-focused technical lead to own the data engineering domain, re-architect and build our pipelines from the ground up, and mentor a small but talented team.

The Mission

You will not be maintaining a legacy system; you will be architecting the new one. You will take ownership of the data flows within the platform, designing a robust architecture that supports both real-time operational views and deep historical analysis.

The Tech Stack

Primary Objectives (First 6–12 Months)

• Re-platforming: Own the redesign and rebuild of the data transformation layer (post-event bus) in Databricks. Move from ad-hoc scripts to software engineering standards (CI/CD, testing, modular code).
• Modelling Implementation: Support the implementation of the “Unified Name Space” (UNS) across the data estate, to include schema and path standardisation; machine hierarchies and semantic relationships; and methodologies for validation and processing of data against contracts.
• Enablement: Establish a stable data serving layer for Grafana and Analytics (via Databricks/MLflow), unblocking the Data Science team.
• Mentorship: Upskill the existing team in good practices (dbt, git workflows, SQL optimisation, data modelling).

Experience & Background

• Pragmatic Architect: Capable of making trade-offs between “academic perfection” and “business value”. They must align architecture to the 80/20 rule.
• Technical Authority: Comfortable discussing and constructively challenging architectural decisions with the Head of Software and VP of Engineering. Must be able to justify changes to the core architecture and challenge existing thinking.
• Mentor vs. Manager: Willing to sit and pair-program with junior engineers. They should measure their success by the team’s outcomes, not simply their own. A full “Team Manager” is not required; this is not a pastoral-focused hire.
• Business Translator: Able to explain to non-technical stakeholders why changes are required, and what their impacts may be. Similarly, the ability to provide a bridge between technical approaches and limitations, and business requirements.
• Industrial / IoT Context: Highly desirable. You should be comfortable with the chaotic nature of sensor data, time-series continuity, and the physical reality of the machines we are modelling.
• Alternative: Experience in high-volume data systems that face similar challenges (duplicated or missing data, spiky load, variable schemas, etc.) would be beneficial.

Skills

• Advanced Data Engineering: Proven experience building data platforms on Azure. You know Databricks inside out – not just how to write a notebook, but how to architect a Lakehouse, manage clusters, and optimise costs.
• Data Modelling Proficiency: You understand that data engineering is more than moving JSON blobs. You have strong opinions on schema design, data standardisation, and ideally have worked with graph data or hierarchical models.
• Code Quality: You treat data pipelines as software. You use git, CI/CD, and automated testing. You can teach these practices to a junior engineer.
• Communication: You can explain to a business stakeholder why architectural decisions matter for their bottom line, and you can debate architectural trade-offs with Software Engineers.

Why Join?

• Autonomy: You have the support of leadership to make architectural decisions. You aren’t fighting for permission to fix tech debt; you are being hired specifically to fix it.
• Impact: Your work directly enables the Data Science team to stop cleaning data and start building better predictive models.
• Modern Stack: We are moving toward a cutting-edge IIoT stack (UNS + Digital Twins), offering you the chance to work on industry-leading patterns.

What you’ll get

• 5 weeks paid annual leave, plus UK bank holidays
• Tax efficient stock options (subject to scheme rules and eligibility)
• Workplace pension with employer contributions, in line with UK auto-enrolment legislation (subject to eligibility and scheme terms)
• Salary Sacrifice EV Scheme (subject to eligibility)
• Training and professional development opportunities

This is a full-time role. Due to the collaborative and mentoring nature of this role, regular in-person attendance is required. The team collaborates in person from our Sheffield office twice weekly for planning, workshops and team development, providing valuable opportunities for mentoring, feedback and connection. We recognise the importance of flexibility and are open to discussing reasonable flexible working arrangements in line with business needs.

Join the Team!

If your background doesn’t exactly align with the job description, but you possess transferable skills or experience that could be a strong match, we encourage you to highlight this in a cover letter. We’re committed to personal growth as our company evolves, so if you’re excited to be part of that journey, we’d love to hear from you.

We are an equal opportunities employer and are committed to creating an inclusive workplace. We welcome applications from all suitably qualified candidates regardless of age, disability, gender reassignment, marriage or civil partnership, pregnancy or maternity, race, religion or belief, sex, or sexual orientation.

Applicants must have the right to work in the UK.

We will process your personal data as part of the recruitment process in accordance with applicable data protection laws. Please refer to our Recruitment Privacy Notice for further information on how we collect, use, and store candidate information.