All jobs Remote Browse Skills

Search jobs collected from company career pages and public sources, in one place.

Built by Darshit Gajjar. Follow me:

Explore

All jobs Browse all categories Remote jobs Junior roles Senior roles

Popular

AI engineering Frontend Backend Data science

Legal

Disclaimer & Terms Job safety tips Report a job

Jobyssey is a job search engine. We are not the employer or a recruiter and do not verify listings. Jobs link to third-party sites and you apply there at your own risk. Never pay to apply for a job. Company names and trademarks belong to their owners. See the full disclaimer.

© 2026 Jobyssey

Home Jobs Remote Explore

Senior AI Quality Assurance (QA) Engineer at datamaxis – Bengaluru | Jobyssey

Home
Jobs
QA jobs
Senior AI Quality Assurance (QA) Engineer

Senior AI Quality Assurance (QA) Engineer

datamaxis · Bengaluru, India · 2 months ago

Apply on company site

Before you apply

Jobyssey is a search engine. We don't post, own or verify this job and are not the employer or a recruiter. You'll apply directly on datamaxis.breezy.hr, under that site's terms.

Never pay to apply, for training or equipment, and never share bank details, ID documents or passwords before you've confirmed the employer is real.

Job safety tips Report this job Disclaimer

Seniority: Senior
Category: QA
Experience: 7+ years

ai api automation conversational-ai llm mlflow observability pytest python rest root-cause-analysis sql

Explore more jobs

QA jobs Remote QA jobs Jobs in India QA jobs in India Senior jobs

test-automation

Senior QA engineer tests and benchmarks AI systems like LLM apps and NL2SQL pipelines, automating evaluations with Python and Pytest.

Job Position: Senior AI Quality Assurance (QA) Engineer
Experience: 7+ Years
Location: Bengaluru

Required Skills

7+ years of experience in Software Quality Assurance, Test Automation, or AI Quality Engineering.
Hands-on experience evaluating LLM-powered applications, conversational AI systems, NL2SQL solutions, or similar AI workflows.
Strong understanding of AI benchmarking methodologies and experience creating benchmark (golden) datasets for regression testing.
Experience with AI evaluation and observability platforms such as Lang Smith, MLflow, Arize Phoenix, or equivalent.
Strong understanding of AI evaluation metrics, including Precision, Recall, Precision@K, Recall@K, F1 Score, Exact Match (EM), Mean Reciprocal Rank (MRR), Execution Accuracy, latency (P50/P95/P99), token usage, and cost analysis.
Strong proficiency in Python with hands-on experience building automation using Pytest, including unit testing, integration testing, API testing, exception handling, mocking, fixtures, and parameterized testing.
Experience testing REST APIs and distributed backend services.
Strong SQL skills with the ability to validate generated queries, execution results, schema alignment, and business logic.
Strong analytical and debugging skills with the ability to perform root cause analysis across complex AI systems.