AES - Delivery - Digital Test Automation Engineer

You will own quality for data pipelines, microservices, APIs, GUIs, and LLM/ML integrations running on Databricks and Microsoft Azure. The role focuses on test strategy, test case design and execution, automation, ticket management, and agentic QA for LLM workflows. Minimum experience: 5 years

Key Responsibilities
Test Strategy and Planning
Define and maintain QA strategy for data, API, GUI, microservices, and LLM/ML features; prioritize test coverage based on risk and business impact.

Test Case Design and Execution
Create clear, reusable test cases for functional, integration, regression, performance, and pricing scenarios; execute manual and automated tests and maintain test artifacts.

Data and Database Testing
Validate ETL/ELT pipelines on Databricks and Delta Lake; write and run T‑SQL validation queries; verify data lineage, completeness, and correctness.

API Testing
Design and run contract, integration, and negative tests for RESTful APIs; use Postman, OpenAPI/Swagger, and automated API test suites.

GUI Testing
Create and execute UI test plans for web applications; perform exploratory testing and maintain automated UI tests using Selenium, Playwright, or equivalent.

Microservices Validation
Test service contracts, message flows, resilience, and backward compatibility across containerized microservices; validate observability and error handling.

LLM and ML QA
Design evaluation metrics and test harnesses for ML models and LLM integrations; validate inference correctness, hallucination rates, latency, and cost tradeoffs.

Prompt Engineering and Agentic QA
Create minimalistic, cost‑aware prompts and prompt templates; implement agentic QA processes for autonomous agents including safety checks, guardrails, and evaluation pipelines.

Pricing and Business Logic Testing
Validate pricing engines, discount rules, billing flows, and edge cases; create scenario matrices and automated checks for pricing correctness.

Automation and CI/CD
Integrate tests into CI/CD pipelines; author automated test suites, test runners, and validation scripts; collaborate with SRE/DevOps on test environments.

Ticketing and Incident Management
Raise, triage, and manage tickets in Azure DevOps or Jira; document reproduction steps, root cause analysis, and remediation.

Documentation and Collaboration
Produce test plans, runbooks, release checklists, and postmortems; mentor junior QA engineers and participate in cross‑functional reviews.

Required Skills and Experience
Experience: Minimum 5 years in QA, software testing, or related roles.

Databricks and Data Testing: Hands‑on experience validating Databricks pipelines, Delta Lake, and Spark outputs.

Microsoft SQL Server: Medium proficiency in T‑SQL for validation queries and database testing.

Azure: Practical experience with Azure Data Factory, ADLS/Blob Storage, Azure DevOps, and Azure Functions.

API Testing: Strong experience with REST APIs, Postman, OpenAPI/Swagger, and automated API testing.

GUI Testing: Experience with Selenium, Playwright, or equivalent UI automation tools and exploratory testing.

Microservices: Familiarity testing containerized services, message queues, and service orchestration.

LLM and ML Fundamentals: Understanding of ML lifecycle, model evaluation, prompt engineering basics, and LLM failure modes.

Prompt Engineering: Ability to craft minimalistic prompts and maintain prompt libraries for reproducible tests.

Agentic QA Awareness: Knowledge of agentic QA concepts, safety checks, and evaluation metrics for autonomous agents.

Testing Tools and Automation: Experience with test frameworks, CI/CD integration, and test data management.

Ticketing Tools: Experience using Azure DevOps, Jira, or similar.

Soft Skills: Strong analytical thinking, clear written communication, and cross‑team collaboration.

Nice to Have
Certifications: ISTQB, Azure Data Engineer, or Databricks certifications.

Programming: Python for test automation and data validation; familiarity with Scala or Java.

Observability: Experience with Application Insights, Datadog, Prometheus, or Grafana.

Performance Testing: Experience with JMeter, k6, or similar tools.

LLM Ops: Experience with prompt versioning, cost monitoring, and model evaluation frameworks.

Infrastructure as Code: Familiarity with Terraform or ARM templates for test environment provisioning.

Interview and Evaluation Guide
Technical screen: SQL validation problem and API testing scenarios.

Practical exercise: Create test cases and automated checks for a sample Databricks ETL and an API that returns pricing.

LLM prompt exercise: Provide minimal prompts for a given LLM task and design evaluation metrics for correctness and safety.

System design: Design an agentic QA pipeline for autonomous agents including test harness, safety gates, and monitoring.

Behavioral: Examples of ticket triage, incident postmortems, and cross‑team collaboration.

Success Criteria at 6 Months
Test Coverage: Comprehensive, automated test suites for data pipelines, APIs, and core GUI flows.

LLM Readiness: Reproducible prompt templates, evaluation metrics, and agentic QA checks integrated into CI.

Reliability: Reduced production incidents and faster mean time to resolution through clear runbooks and test automation.

Pricing Confidence: End‑to‑end validated pricing flows with scenario tests and regression guards.

Collaboration: Trusted QA partner to engineering and product teams with documented processes and mentoring outcomes.

Key Responsibilities
Test Strategy and Planning
Define and maintain QA strategy for data, API, GUI, microservices, and LLM/ML features; prioritize test coverage based on risk and business impact.

Test Case Design and Execution
Create clear, reusable test cases for functional, integration, regression, performance, and pricing scenarios; execute manual and automated tests and maintain test artifacts.

Data and Database Testing
Validate ETL/ELT pipelines on Databricks and Delta Lake; write and run T‑SQL validation queries; verify data lineage, completeness, and correctness.

API Testing
Design and run contract, integration, and negative tests for RESTful APIs; use Postman, OpenAPI/Swagger, and automated API test suites.

GUI Testing
Create and execute UI test plans for web applications; perform exploratory testing and maintain automated UI tests using Selenium, Playwright, or equivalent.

Microservices Validation
Test service contracts, message flows, resilience, and backward compatibility across containerized microservices; validate observability and error handling.

LLM and ML QA
Design evaluation metrics and test harnesses for ML models and LLM integrations; validate inference correctness, hallucination rates, latency, and cost tradeoffs.

Prompt Engineering and Agentic QA
Create minimalistic, cost‑aware prompts and prompt templates; implement agentic QA processes for autonomous agents including safety checks, guardrails, and evaluation pipelines.

Pricing and Business Logic Testing
Validate pricing engines, discount rules, billing flows, and edge cases; create scenario matrices and automated checks for pricing correctness.

Automation and CI/CD
Integrate tests into CI/CD pipelines; author automated test suites, test runners, and validation scripts; collaborate with SRE/DevOps on test environments.

Ticketing and Incident Management
Raise, triage, and manage tickets in Azure DevOps or Jira; document reproduction steps, root cause analysis, and remediation.

Documentation and Collaboration
Produce test plans, runbooks, release checklists, and postmortems; mentor junior QA engineers and participate in cross‑functional reviews.

Required Skills and Experience
Experience: Minimum 5 years in QA, software testing, or related roles.

Databricks and Data Testing: Hands‑on experience validating Databricks pipelines, Delta Lake, and Spark outputs.

Microsoft SQL Server: Medium proficiency in T‑SQL for validation queries and database testing.

Azure: Practical experience with Azure Data Factory, ADLS/Blob Storage, Azure DevOps, and Azure Functions.

API Testing: Strong experience with REST APIs, Postman, OpenAPI/Swagger, and automated API testing.

GUI Testing: Experience with Selenium, Playwright, or equivalent UI automation tools and exploratory testing.

Microservices: Familiarity testing containerized services, message queues, and service orchestration.

LLM and ML Fundamentals: Understanding of ML lifecycle, model evaluation, prompt engineering basics, and LLM failure modes.

Prompt Engineering: Ability to craft minimalistic prompts and maintain prompt libraries for reproducible tests.

Agentic QA Awareness: Knowledge of agentic QA concepts, safety checks, and evaluation metrics for autonomous agents.

Testing Tools and Automation: Experience with test frameworks, CI/CD integration, and test data management.

Ticketing Tools: Experience using Azure DevOps, Jira, or similar.

Soft Skills: Strong analytical thinking, clear written communication, and cross‑team collaboration.

Nice to Have
Certifications: ISTQB, Azure Data Engineer, or Databricks certifications.

Programming: Python for test automation and data validation; familiarity with Scala or Java.

Observability: Experience with Application Insights, Datadog, Prometheus, or Grafana.

Performance Testing: Experience with JMeter, k6, or similar tools.

LLM Ops: Experience with prompt versioning, cost monitoring, and model evaluation frameworks.

Infrastructure as Code: Familiarity with Terraform or ARM templates for test environment provisioning.

Interview and Evaluation Guide
Technical screen: SQL validation problem and API testing scenarios.

Practical exercise: Create test cases and automated checks for a sample Databricks ETL and an API that returns pricing.

LLM prompt exercise: Provide minimal prompts for a given LLM task and design evaluation metrics for correctness and safety.

System design: Design an agentic QA pipeline for autonomous agents including test harness, safety gates, and monitoring.

Behavioral: Examples of ticket triage, incident postmortems, and cross‑team collaboration.

Success Criteria at 6 Months
Test Coverage: Comprehensive, automated test suites for data pipelines, APIs, and core GUI flows.

LLM Readiness: Reproducible prompt templates, evaluation metrics, and agentic QA checks integrated into CI.

Reliability: Reduced production incidents and faster mean time to resolution through clear runbooks and test automation.

Pricing Confidence: End‑to‑end validated pricing flows with scenario tests and regression guards.

Collaboration: Trusted QA partner to engineering and product teams with documented processes and mentoring outcomes.