Senior Data Engineer

Join us at Social Security Scotland as a Senior Data Engineer and be at the forefront of our ambitious digital transformation. This role offers an exciting opportunity to work with a talented team and turn data into a driving force for positive change, helping us create a system that supports the people of Scotland for decades to come.

The Digital, Delivery & Change division is responsible for delivering the infrastructure, capability and digital solutions that will support Social Security Scotland. As a Senior Data Engineer within the division, you will play a crucial role in developing robust data infrastructure and systems that underpin our diverse digital initiatives. Collaborating with a growing team of digital experts across Engineering & Cloud Services, Architecture, Applications, and more, you will design and implement data flows that facilitate analytics and business intelligence.

Your expertise will ensure the delivery of high-quality data products that meet the needs of our stakeholders and enable effective decision-making. We are committed to fostering a positive and inclusive culture that empowers all team members to thrive, making this an exciting opportunity to contribute to meaningful societal change.

A Senior Data Engineer designs and leads the implementation of data flows to connect operational systems, data for analytics and business intelligence (BI) systems.

At This Role Level, You Will

• Recognise opportunities to reuse existing data flows.
• Lead the build of data streaming systems.
• Optimise the code to ensure processes perform optimally.

Responsibilities

• Design, develop and support real-time data streaming solutions using Apache Kafka to deliver scalable, reliable data integration.
• Design and implement scalable Python-based solutions for data processing, integration and automation.
• Lead technical discussions during sprint planning, story refinement and other agile ceremonies, working closely with stakeholders.
• Analyse, profile and assess data from multiple complex source systems to understand structure, quality and business value.
• Design, build and maintain complex, large-scale data products and integration services.
• Consolidate data from multiple sources into conformed models that support analysis, reporting and decision-making.
• Create and maintain data models, including relational, dimensional, graph, metadata and subject-area models.
• Select and implement appropriate technologies from the approved portfolio to deliver scalable, secure and future‑ready data solutions.
• Develop, test, document and maintain complex programs, scripts and data pipelines in line with agreed standards, specifications and user requirements.
• Perform data transformation, cleansing and normalisation to ensure data is fit for analytical and reporting purposes.
• Lead the development and adoption of reusable design patterns, standards and best practices.
• Present data insights and technical findings clearly to colleagues and stakeholders to support informed decision‑making.
• Monitor databases, data products and services, proactively identifying issues, trends and risks, and implementing corrective and preventative actions.
• Review requirements and specifications, define test conditions, analyse test results and produce synthetic test data where required.
• Build capability within teams by providing technical leadership, mentoring and support across data integration activities.
• Proactively identify opportunities to maximise value from data and drive innovation across services and products.
• Maintain and apply up‑to‑date knowledge of relevant technologies, frameworks and industry best practices.
• Contribute to the wider Government Digital & Data community through knowledge sharing, technical peer support, code reviews, pair programming and specialist consultancy.

Success Profiles

We use an assessment framework called ‘Success Profiles’ which lists the elements we test and provides detailed descriptions of each. Find out more about the framework here.

For this post, the following Success Profile elements will be assessed:

Experience

• Experience of designing, developing, testing, documenting and maintaining automated, repeatable data solutions using agreed standards, tools and best practices.
• Experience of monitoring, troubleshooting and resolving issues across data, data processes, data products and services, implementing preventative measures where appropriate.

Behaviours

• Leadership (Level 4)
• Working Together (Level 4)

You can find out more about Success Profiles Behaviours here.

Technical Skills

This role is aligned to Senior Data Engineer within the Government Digital and Data Profession.

These skills will be tested during the Technical Assessment if you are successful at sift stage. They will not be assessed at application stage.

Please review the following to understand the skill expectations: Data engineer - Government Digital and Data Profession Capability Framework

Information Session

We will be hosting a candidate information session for this role on Wednesday 26th August from 12noon - 1pm where we will be talking about:

• The Senior Data Engineer role and team
• About Social Security Scotland
• Our recruitment process
• Q&A with the hiring manager

Expected Timeline

Sift - week commencing 31st August

Interview – week commencing 21st September

Location - In Person in either Dundee or Glasgow

Reserve List

In the event that there are more successful candidates than posts available, a reserve list will be kept for up to 12 months.

About Us

Social Security Scotland is an Executive Agency of the Scottish Government. Our benefits help people from all walks of life in Scotland. We offer rewarding careers and employ people across Scotland in a wide range of professions and roles. We are committed to recruiting a diverse workforce that is representative of the clients we serve. Find more about us here.

We offer a supportive and inclusive working environment along with a wide range of employee benefits. Find out more about what we offer.

As part of the UK Civil Service, we uphold the Civil Service Nationality Rules.

GDD Pay Supplement

This post is part of the Government Digital and Data (GDD) profession and currently attracts a £4,000 annual GDD pay supplement, which is paid monthly. Pay supplements are reviewed regularly.

Working Pattern

Our standard hours are 35 hours per week and we offer a range of flexible working options, depending on the needs of the role. We embrace a hybrid working style where all colleagues will spend time in either our Glasgow or Dundee offices. There is an expectation of a minimum 2 days per week in your assigned location, which will be either Glasgow or Dundee. If you have specific questions about the role you are applying for, please contact us.

Security Checks

Successful candidates must complete the Baseline Personnel Security Standard (BPSS), before they can be appointed. BPSS is comprised of four main pre‑employment checks – Identity, Right to work, Employment History and a Criminal Record check (unspent convictions).

You can find out more about BPSS on the UK Government website, or read about the different levels of security checks in our Candidate Guide.

Equality Statement

Social Security Scotland are committed to equality and inclusion, and we aim to recruit a diverse workforce that reflects the population of our nation.

Social Security Scotland are a Disability Confident Employer. We will consider and implement any reasonable adjustments you may require throughout the recruitment process and during the course of your employment, should you be successful in securing a post. If you feel you may require assistance with any part of our recruitment process, please contact us at [email protected].

Find out more about our commitment to diversity and how we offer and support recruitment adjustments for anyone who needs them.

Right to Work in the UK

Social Security Scotland is an approved sponsor under the UK Visa and Immigration (UKVI) Skilled Worker route. Please note that UK immigration guidance, including skill and salary thresholds and eligible occupations, is reviewed regularly and subject to change. If you require visa sponsorship, you should check the latest criteria to confirm whether this role meets current requirements before applying. You can find further advice on Gov.UK - Skilled Worker visa: Overview - GOV.UK

Further Information

Social Security Scotland’s recruitment processes are underpinned by the recruitment principles of the Civil Service Commissioner, which outline that selection for appointment be made on merit on the basis of fair and open competition - Recruitment - Civil Service Commission

If you feel at any time your application has not been treated in accordance with the values in the Civil Service Code and/or if you feel the recruitment has been conducted in such a way that conflicts with the Civil Service Commissioner's Recruitment Principles, you can make a complaint, by contacting Social Security Scotland at [email protected] in the first instance. If you are not satisfied with the response you receive you can contact the Civil Service Commissioner.

The successful candidate will be expected to remain in post for a minimum of 3 years unless successful in gaining promotion to a higher Band or Grade.

Find out more about our organisation, what we offer staff members and how to apply on our Careers Website.

Read our Candidate Guide for further information on our recruitment and application processes.

If you experience any difficulties accessing our website or completing the online application form, please contact the Resourcing Team via [email protected]