Best practices for working with the AI Command

Using an AI Assistant to perform validations and user actions on the device under test opens up a new ease of use in automated testing. To learn what you can and cannot do with the AI Command, review the following best practices.

In AI Scriptless, the AI Command is available as part of Perfecto's AI commands. For Appium testing, AI-based validations are available through AI functions.

The AI Command command unifies and supersedes the AI User Action and AI Validation commands. The AI Command is the recommended choice for all new scripts. AI User Action and AI Validation continue to function for backward compatibility.

On this page:

What can the Assistant do?

For optimal performance, phrase each Test Step so it does not resolve to more than 15 actions and so that it takes less than 3 minutes.

If you need additional reasoning resources to get correct results, for example, for counting, sorting, or ordering, enable the Reasoning parameter.

We have tested the following AI User Actions with excellent results:

Identify and use common UI elements, such as the Hamburger menu, help icons, notification icons
Identify and click buttons, hyperlinks, icons, and floating action buttons (FAB)
Identify and type text into empty text fields, or replace their text
Select radio buttons and checkboxes, set values using sliders
Select tabs, entries from lists, items from navbars
Read and dismiss popups and notifications that interrupt the action
Browser navigation (go to, back, forward, refresh, clean)
Works in almost any language. Although not mandatory, for best results, consider using the same language as the application under test.

The following actions are still being tested, and may or may not work:

Identify a text field when the label is inline, above, left, or right of the field
select a cell from a table, select an action from a row in a table
select a color

We have not yet tested the following actions:

Append text to existing text in field

Be aware that a Large Language Model can make mistakes, the model is being constantly improved.

The AI Assistant will always try to perform the User Action, but the more complex the task is, the higher is the chance of failure. Correct the Assistant if it misunderstands, and evaluate its next suggestions until you find the perfect input. Experiment! We love challenges and are happy to get feedback at Perfecto Support whenever something unexpected happens!

What can I do that I couldn't do before?

During development, UI elements and procedures can change. Buttons can be moved to a different location or are relabeled. But you are already running tests. When you define test scripts traditionally as actions applied to locators, the test script breaks every time a UI element changes.

Using the AI Assistant, you can describe the logical task in the same way as when talking to a manual tester, for example, "Select VISA as payment method and confirm."

Use the same script even after the UI changes

Talking to a member of your tester team, you would not spell out details such as "move the mouse and click the button in the bottom left labeled Confirm". UI elements can change during development, for example, the "Confirm" button could be renamed to "Accept". Obviously, you would describe the intended goal of the action. The advantage of describing the logical task is that a manual tester —just as the AI Assistant— can follow the same instructions, even if the UI changes later. For example, the AI Assistant —just as a manual tester— can identify the button that confirms the selection and click it, no matter what it is called, or where on the screen it is.
Wait instead of fail

If your app is affected by irregular network latency, or a procedure is disrupted by a modal dialog, a hard-coded script can miss its cue. Instead, tell the AI Assistant to wait, to attempt to repeat an action until it is successful, to dismiss any modal dialog, or to wait for a progressbar to finish.
Write once, run everywhere

Once you have fine-tuned the instructions, you can even run the same Scriptless test across a variety of devices and validate the same test steps on them. This saves time otherwise spent with repetitive test development and duplicated maintenance. Since you do not refer to device-dependent UI elements, the AI Assistant interprets the instructions relative to what it sees on the device screen.
General knowledge questions
The AI Validation is not limited to visual validations on the device screen. You can also ask it questions about variable values and factual knowledge, such as general public domain knowledge. The condition is that you must phrase the input so that the answer can be either PASS or FAIL.
Examples:
- Verify that the recipient field contains a valid US address format.
- Confirm that the account balance shown is equal to ${balance}.
- Check whether the ${transaction_date} is today's date. (the environment date in UTC)

Validate test steps

A validation is a question or statement with a binary answer: yes or no, PASS or FAIL, true or false. When it comes to automated testing, a validation or assertion is the primary way to decide whether something is as expected.

Some examples:

√Is the account balance equal to $22?
√Verify the account balance is equal to $22.
√Is the color of the login button green?
√Check whether the color of the login button is green.
√Does the user name Joe appear in the app header?
√ The user name Joe appears in the app header.

Limitations

The following limitations apply to AI-powered validations:

AI validation models are not perfect, they can make mistakes. Before assigning a value to an AI-powered command in AI Scriptless, always test the validation using Perfecto's interactive AI assistant.
AI validation models are not connected to your backend, or to the world (this means, it doesn't know your local time or date, only the device environment time in UTC). If the information does not appear in the context of the device screen and is not factual public domain knowledge, it cannot be a validation.
Do not ask open-ended questions (what, who, where, when, why, how, ...) as validations.

Best Practices

When phrasing AI Commands, do the following:

Be specific
Use objective and clear definitions
Ask about contents that are visible on the device screen

Here are some good and bad examples for how to phrase the input.

Don'ts	Do's	Why?
⨯ Move the mouse to the button labeled Cart in the bottom right corner	√ Open the shopping cart	Describe the logical goal. Do not refer to locators or labels that can change during development
⨯ Fill in the form	√ Enter "Jill A. Smith" in the name fields. Enter Main street 1 in NYC, USA, as the address.	Don't be vague. Provide enough details for the AI Assistant to be able to repeat the task consistently.
⨯ Click the red shoes.	√ Select the red sneakers that cost $30.	Don't be ambiguous. Can it happen that the UI contains several similar elements? Use specific words.
⨯ Click the second menu item from the top right.	√ Select Home from the navigation menu.	When referring to UI elements, use platform agnostic terminology so the test runs on mobile and web.
⨯ Did it work?	√ Verify the red sneakers are in the shopping cart.	Describe the precise validation criterion that makes the test pass or fail.
⨯ Who is logged in?	√ Is Jill Smith logged in?	Validations are questions that have a yes or no answer.
⨯ How many search results were found?	√ Verify that search results have loaded.	Validations are statements that verify or confirm facts as true or false.
⨯ Does pizza taste good? ⨯ How does the pizza taste?	√ Is the shown rating of the selected pizza higher than 4?	Validations must be objective.

Need help?

If you need help with a complex user action, share it with us—we love a challenge! Reach out to Perfecto Support and include a screenshot with the exact prompt you used and information about how you wanted to interact with the UI.