Software Quality Assurance Engineer – GenAI / LLM

Primary Focus: Software Quality Assurance / Software Testing / Test Automation – GenAI, LLM & Agentic AI
Secondary Exposure: Solution Analysis / Technology Solution Design / Enterprise Integration
Domain / Project: Global Markets, Capital Markets Banking Technology & Market Risk Technology

Role Overview

We are looking for a Senior GenAI Quality Engineer / Solution Analyst to design, analyse, test and validate production-grade Generative AI (GenAI), Large Language Model (LLM), RAG and Agentic AI applications within a complex enterprise environment.

This is not a traditional manual QA or software testing role.

The role combines:

• Software Quality Engineering

• GenAI / LLM Testing & Evaluation

• Agentic AI / AI Agent Testing

• UI & API Testing

• Test Automation

• Solution Analysis

• Enterprise Integration Testing

• Observability & Troubleshooting

You will work across discovery, solution design, development, testing and release, translating business requirements into clear application behaviours and validating end-to-end application quality across user interfaces, APIs, data flows, LLMs, RAG components, AI agents and enterprise integrations.

Key Responsibilities

GenAI / LLM Quality Engineering

• Define and execute end-to-end quality engineering and test strategies covering:

  • Web / UI workflows

  • REST APIs

  • Backend services

  • Enterprise integrations

  • GenAI applications

  • LLM workflows

  • RAG pipelines

  • Agentic AI / AI Agent interfaces

• Perform GenAI / LLM testing and evaluation covering:

  • Response quality

  • Task completion

  • Grounding

  • Faithfulness

  • Relevance

  • Consistency

  • Citation accuracy

  • Hallucination risk

  • Safe failure behaviour

• Test non-deterministic / probabilistic AI systems using:

  • Evaluation datasets

  • Repeat testing

  • Quality thresholds

  • Acceptance criteria

  • Regression evaluation

• Validate RAG / Retrieval-Augmented Generation solutions, including retrieval quality, grounding and response accuracy.

Agentic AI / AI Agent Testing

Test end-to-end Agentic AI and AI Agent workflows, including:

• Multi-turn conversations

• Context handling

• Agent planning

• Tool selection

• Tool calling / function calling

• Tool inputs and outputs

• State transitions

• Memory and state

• Human-in-the-loop approvals

• Handoffs

• Retries

• Timeouts

• Fallback behaviour

• Error recovery

• Termination conditions

• Partial failures

Validate that AI agents behave correctly across both successful and failure scenarios.

Software & API Quality Engineering

Perform:

• Functional Testing

• Integration Testing

• API Testing

• Regression Testing

• Exploratory Testing

• Negative Testing

• Resilience Testing

• Basic Performance Testing

• End-to-End Testing

Design comprehensive REST API tests covering:

• API contracts

• Authentication

• Authorisation

• Input validation

• Error handling

• Idempotency

• Rate limits

• Downstream system failures

Test web application behaviour across browsers and realistic end-user journeys, including:

• Loading states

• Interrupted sessions

• Error messages

• Feedback capture

• Accessibility fundamentals

Test Automation

Develop and maintain risk-based test automation that reduces:

• Regression testing time

• Manual testing effort

• Release cycle time

• Production risk

Use automation frameworks and tools such as:

• Playwright

• Cypress

• Selenium

• pytest

• REST Assured

• Postman

• Equivalent UI / API automation frameworks

Apply pragmatic automation principles by prioritising stable, high-value and frequently executed test scenarios.

GenAI Evaluation & AI Safety Testing

Validate LLM and GenAI applications for:

• Grounded responses

• Hallucinations

• Retrieval quality

• Citation accuracy

• Prompt behaviour

• Prompt injection

• Unsupported requests

• Restricted content handling

• Safe failure behaviour

• Adversarial scenarios

Support AI evaluation / LLM evaluation using appropriate evaluation datasets, quality metrics and repeatable evaluation approaches.

Exposure to AI Red Teaming / Adversarial Testing would be advantageous.

Observability & Troubleshooting

Use application and GenAI observability to identify the source of defects across:

• Application

• LLM / Model

• RAG / Retrieval

• Data

• API / Integration

• Platform

Analyse:

• Logs

• Distributed traces

• API requests / responses

• Payloads

• Network calls

• Database records

• Agent execution traces

Exposure to observability and LLM evaluation tools such as:

• Langfuse

• LangSmith

• OpenTelemetry

• Elastic / Elasticsearch

• Splunk

is advantageous.

Solution Analysis & Design

The role also acts as a hands-on Solution Analyst for GenAI applications.

Responsibilities include:

• Partner with product owners, business users, architects, engineers and GenAI specialists during discovery and solution design.

• Analyse proposed GenAI use cases and determine whether the requirement should use:

  • Conventional application logic

  • Deterministic business rules

  • Search / retrieval

  • RAG

  • Workflow automation

  • Agentic AI

  • Human approval

• Translate business requirements into:

  • Functional requirements

  • End-to-end solution flows

  • User journeys

  • Acceptance criteria

  • Interface behaviour

  • Decision rules

  • Non-functional requirements

• Map interactions across:

  • User Interfaces

  • APIs

  • LLMs / Models

  • Prompts

  • RAG / Retrieval components

  • Enterprise data sources

  • AI Agent tools

  • Downstream enterprise systems

• Analyse solution design trade-offs involving:

  • Quality

  • Complexity

  • Cost

  • Latency

  • Security

  • Data access

  • Maintainability

  • Operational risk

• Identify missing controls, integration assumptions, ownership gaps, failure scenarios and operational risks before development begins.

• Support the design of:

  • Human-in-the-loop approval

  • Fallback flows

  • Escalation

  • Exception handling

Solution Documentation

Produce practical technical and functional artefacts including:

• Process Flows

• Sequence Diagrams

• Context Diagrams

• Interface Specifications

• Decision Tables

• User Stories

• Acceptance Criteria

• Test Scenarios

• Traceability Documentation

Maintain traceability across:

Business Requirement → Solution Design → Implementation → Test / Evaluation Scenario → Release Evidence

Release Quality & Governance

Create and maintain:

• Test scenarios

• Test datasets

• Reusable regression scenarios

• Test evidence

• Defect reports

• Quality metrics

• Release quality reports

Provide evidence-based release recommendations identifying:

• Known defects

• Known limitations

• Residual risks

• Quality concerns

• Areas requiring production monitoring

Core Requirements

Experience

• 5–8 years of experience in Software Quality Engineering, Test Engineering, Test Automation, SDET or similar hands-on software testing roles.

• Strong experience testing complex enterprise applications.

• Strong experience testing:

  • Web applications

  • REST APIs

  • Backend services

  • Enterprise integrations

Test Automation / Programming

Hands-on experience with one or more of:

• Playwright

• Cypress

• Selenium

• pytest

• REST Assured

• Postman

• Equivalent automation frameworks

Working programming knowledge of:

• Python

• Java

• JavaScript

• TypeScript

Candidates should be capable of developing, reviewing and troubleshooting test automation.

Software Engineering / DevOps

Experience with:

• Git

• Pull Requests

• CI/CD

• Automated Testing

• Test Reporting

• Defect Management

Experience validating distributed systems including:

• Asynchronous Processing

• Queues

• Batch Processing

• APIs

• Downstream Dependencies

• Enterprise Integrations

GenAI / LLM Requirements

Practical understanding of:

• Generative AI / GenAI

• Large Language Models / LLM

• LLM Evaluation

• LLM Testing

• Retrieval-Augmented Generation / RAG

• RAG Evaluation

• Agentic AI

• AI Agents

• Multi-Agent Workflows

• Prompts / Prompt Engineering

• Context Windows

• Embeddings

• Tool Calling

• Agent Memory & State

• LLM Observability

Candidates should understand how GenAI applications differ from conventional deterministic software and how to validate probabilistic AI behaviour.

Security & Risk Testing

Understanding of software and GenAI security fundamentals including:

• Access Control

• Authentication / Authorisation

• Sensitive Data Handling

• Input Validation

• Auditability

• Prompt Injection

• AI Safety Testing

• Adversarial Testing

Nice to Have

Experience with:

• Banking / Financial Services

• Regulated enterprise environments

• Contract Testing

• Service Virtualisation

• Synthetic Monitoring

• Performance Testing

• AI Red Teaming

• Accessibility Testing / WCAG

• Kubernetes

• OpenShift

• AWS

• Containerised Application Deployment

Key Domain / Technical Skills

1. Software Quality Engineering, API Testing & Test Automation

2. GenAI / LLM Evaluation, RAG & Agentic AI Testing

3. Solution Analysis, Observability & Enterprise Integration

Key Search Keywords

GenAI Quality Engineer,AI Quality Engineer,LLM Quality Engineer,Generative AI Testing,GenAI Testing,LLM Testing,LLM Evaluation,AI Evaluation,Agentic AI Testing,AI Agent Testing,RAG Testing,RAG Evaluation,Retrieval-Augmented Generation,Software Quality Engineering,Quality Engineering,Software QA,Test Automation,SDET,Automation Testing,API Testing,REST API Testing,UI Testing,Integration Testing,Regression Testing,End-to-End Testing,Playwright,Cypress,Selenium,pytest,REST Assured,Postman,Python,Java,JavaScript,TypeScript,CI/CD,Git,Prompt Testing,Prompt Injection,Hallucination Testing,Grounding,Faithfulness,AI Safety Testing,Adversarial Testing,AI Red Teaming,Langfuse,LangSmith,OpenTelemetry,Elastic,Splunk,Observability,Distributed Systems,Kubernetes,OpenShift,Solution Analysis