Senior QA Engineer - Automation

Job Summary:

Convo is looking for QA Automation Engineers with 5+ years of experience in test automation to build robust automated testing frameworks for APIs, agent workflows, tool calls, and critical UI experiences. The ideal candidate should have strong expertise in Python, pytest, API testing, Playwright, CI/CD, contract testing, and asynchronous workflows, with experience creating reusable test fixtures, mocks, and simulators to ensure reliable and scalable product quality.

Technical mission

Automate regression testing for agent tool calls, planner outputs, APIs and the operator-facing agentic console.

Key responsibilities

• Create API and workflow regression suites for orchestrator, planner and tool-registry behavior.

• Build deterministic assertions around schemas, tool-call sequences, constraints and expected side effects.

• Implement pytest-based service tests and Playwright tests for relevant console workflows.

• Create mocks, simulators and fixtures for tools, model responses and external dependencies.

• Integrate suites into CI, publish actionable results and manage flaky-test remediation.

• Maintain traceability from technical requirements and defects to automated coverage.

Required technical capabilities

• Minimum experience: 4+ years in test automation, including 2+ years testing APIs, asynchronous workflows or distributed services.

• Strong Python, pytest and API-testing experience.

• Playwright or comparable browser automation for critical UI workflows.

• CI pipeline integration and test-result reporting.

• Contract/schema validation for REST and event-driven interfaces.

• Ability to test asynchronous, non-deterministic and retrying workflows without masking defects.

• Experience designing reusable test fixtures, mocks and service simulators.

Preferred experience

• Agent/LLM workflow evaluation and tool-call trace analysis.

• Performance, resilience and fault-injection testing.

• OpenTelemetry or distributed-trace-based test diagnostics.

Expected deliverables / acceptance evidence

• Automated regression suites for planner, tool calls, APIs and critical UI paths.

• CI integration with quality thresholds and readable failure evidence.

• Reusable mocks, fixtures and simulators.

• Coverage and defect-leakage reporting with maintained documentation.

Primary interfaces

Works with Agent Runtime, DevOps, AI Test & Dataset Engineering and product teams consuming the shared framework.