AI Operations & QA Engineer - Hybrid

About the role

The Automation QA Engineer defines, implements, and continuously improves quality assurance practices for AI-enabled and agent-based solutions. The role ensures systems meet agreed standards for accuracy, reliability, performance, safety, and compliance, and supports operational readiness through automated evaluation and monitoring.

Key responsibilities

• Define quality criteria and testing strategies for agent workflows, covering accuracy, latency, safety, compliance, and operational risk

• Build automated evaluation harnesses to assess agent performance, including hallucination rates, tool misuse, policy violations, and task success

• Implement continuous production monitoring to detect anomalies, quality degradation, and emerging safety concerns

• Develop and maintain automated test suites using Playwright for UI testing and custom scripts for API and workflow validation

• Apply LLM evaluation frameworks to assess output quality, regression, and system drift over time

• Produce and maintain dashboards and reports that communicate quality metrics and trends to engineering and stakeholders

• Develop and maintain runbooks for common failure modes and contribute to incident response activities

• Collaborate closely with developers to improve prompts, tool definitions, and workflow designs based on test results

• Ensure testing, logging, and monitoring practices align with data privacy, audit, and regulatory requirements

About you

• Minimum 3 years' experience in QA, test automation, or DevOps roles (or 2 years with direct experience testing AI or ML-enabled systems)

• Strong Python skills for test automation, evaluation harnesses, and basic data analysis

• High attention to detail, with a focus on issues that materially impact reliability and user trust

• Comfort working with evolving tools, frameworks, and testing practices

• Collaborative mindset, using evidence-based insights to influence product and engineering decisions

• Programming: Python (test automation, evaluation harnesses, data analysis)

• UI Automation: Playwright (end-to-end workflow testing)

• AI Evaluation: Deepeval, RAGAS, Evidently.AI (LLM quality, drift, and regression analysis)

• Workflow Testing: API and agent workflow validation using custom scripts

• Monitoring: Production quality monitoring and anomaly detection

Preferred skills/experience

• Pytest or equivalent testing frameworks

• SQL for querying logs, metrics, or evaluation datasets

• Prometheus, Grafana, or similar monitoring tools

• Familiarity with hallucination detection and AI safety patterns

• CI/CD pipelines and Git-based workflows