AI Software QA Tester at Cloudstaff – Australia | Jobyssey

Job Description

We are seeking an experienced Automation QA Tester to join our growing team and take ownership of testing our AI avatar and conversational workflows. The primary focus of this role is to ensure that the avatar consistently provides accurate and relevant responses to the correct enquiry, even when users ask the same question in different ways or use varied phrasing.

The ideal candidate will have strong experience in test automation, particularly with Playwright or similar tools, and will be comfortable validating both functional behaviour and conversational accuracy across web-based applications. Experience working with Azure DevOps pipelines and testing cloud-based applications is highly desirable.

This role is well-suited to someone who has a strong eye for detail, enjoys identifying gaps in user journeys, and can design robust test scenarios to validate how an AI-driven solution performs under different question variations, use cases, and edge cases.

Key Responsibilities

Design, develop, and maintain automated test scripts for web-based applications and AI avatar interactions.
Test the AI avatar to ensure it provides the correct answer to the intended enquiry, even when users phrase questions differently or use alternative wording.
Create and execute test cases covering multiple variations of the same question to validate the avatar’s accuracy, consistency, and contextual understanding.
Perform end-to-end testing using tools such as Playwright, ensuring strong coverage across functional, regression, and user interaction scenarios.
Validate conversational flows, response relevance, escalation handling, fallback responses, and overall user experience.
Work closely with development and product teams to identify defects, inconsistencies, and opportunities to improve the avatar’s response quality.
Integrate automated testing into CI/CD pipelines using Azure DevOps.
Test cloud-based applications deployed on Azure App Services and Azure Function Apps.
Analyse, document, and report test outcomes clearly, including response mismatches, failed scenarios, and conversational gaps.
Help define best practices for AI/conversational testing, including response validation, intent matching, and variation testing.
Ensure test environments remain current and aligned with production-like conditions.
Continuously identify and implement improvements to the automation and testing process.
Stay up to date with the latest tools, trends, and best practices in automation testing, AI testing, and conversational QA.

Key Focus of the Role

A major responsibility of this position is to test whether the avatar can:

Recognise and respond appropriately to the same enquiry asked in different ways
Maintain consistency and accuracy across varied user inputs
Distinguish between similar but different enquiries
Escalate, clarify, or respond appropriately when the question is unclear or outside scope
Deliver a reliable and user-friendly conversational experience

For example, the tester should be able to validate that the avatar responds correctly to variations such as:

“How many bedrooms does this house have?”
“Can you tell me the number of bedrooms in this property?”
“Is this home a 3-bedroom house?”

All of the above should lead to the correct or intended response or support pathway, where appropriate.

Qualifications and Requirements

Non-Negotiable Requirements

The QA resource will take full ownership of automation testing
Playwright is the preferred and required tool, as it integrates well with the existing CI/CD pipeline
Proficiency in Selenium is acceptable only if there is strong transferability to Playwright
The candidate must be capable of testing not only technical functionality, but also the accuracy and appropriateness of avatar responses across varied enquiries

Nice-to-Have Skills

Experience testing AI-driven, chatbot, avatar, or conversational platforms
Understanding of natural language variations, intent recognition, or conversational workflows
Experience with API testing tools such as Postman or Swagger
Familiarity with functional programming principles in test automation
Knowledge of Terraform or other infrastructure-as-code tools for setting up test environments
Experience in the real estate industry would be an advantage, but is not required
Experience in automation and workflows is a plus
Understanding of security testing best practices

Industry Background

Experience in the real estate industry would be a plus, but is not required.

Success in This Role

Success in this role will be measured by the tester’s ability to ensure the avatar delivers:

Accurate and relevant answers
Consistent responses across different question formats
Strong end-to-end functionality within the application
Reliable performance within an automated CI/CD testing environment
A smoother and more effective user experience overall

Perks and Benefits

Become part of the Employee Share Units program
Cloudstaff Dream Points - To be used for bidding on useful items like appliances, kitchenettes, etc.
Comprehensive health and life insurance on your 16th day of employment, covering 1 free dependent on the 16th day of employment
Endless opportunities for career advancement
Flexible leave credits which may be used for vacation, emergency and sick leaves
In-house psychiatrist available to support employees' well-being
International career growth and connections
Mental Wellness Employee Assistance program through Lifeworks
Participate and join our CS Social Clubs and Special Interest Groups to connect with colleagues
Quarterly perks boxes for WFH staff, offering groceries and snacks to keep you fueled
Superb and exciting Mid-Year Parties – with items to give away and cash prizes!
Unlimited cash incentives for hired referrals
We set you up for success with a company-provided PC/Laptop and fiber internet connection

Explore more jobs