Best practices for working with the AI Command

Using an AI Assistant to perform validations and user actions on the device under test opens up a new ease of use in automated testing. To learn what you can and cannot do with the AI Command, review the following best practices.

In AI Scriptless, the AI Command is available as part of Perfecto's AI commands. For Appium testing, AI-based validations are available through AI functions.

What can the Assistant do?

For optimal performance, phrase each Test Step so it does not resolve to more than 15 actions and so that it takes less than 3 minutes.

If you need additional reasoning resources to get correct results, for example, for counting, sorting, or ordering, enable the Reasoning parameter.

We have tested the following user actions:

• Identify and use common UI elements, such as the Hamburger menu, help icons, notification icons

• Identify and click buttons, hyperlinks, icons, and floating action buttons (FAB)

• Identify and type text into empty text fields, or replace their text

• Select radio buttons and checkboxes, set values using sliders

• Select tabs, entries from lists, items from navbars

• Read and dismiss popups and notifications that interrupt the action

• Browser navigation (go to, back, forward, refresh, clean)

• Works in almost any language. Although not mandatory, for best results, consider using the same language as the application under test.

The following actions are still being tested, and may or may not work:

• Identify a text field when the label is inline, above, left, or right of the field

• select a cell from a table, select an action from a row in a table

• select a color

We have not yet tested the following actions:

• Append text to existing text in field

Be aware that a Large Language Model can make mistakes, the model is being constantly improved.

What can I do that I couldn't do before?

During development, UI elements and procedures can change. Buttons can be moved to a different location or are relabeled. But you are already running tests. When you define test scripts traditionally as actions applied to locators, the test script breaks every time a UI element changes.

Using the AI Assistant, you can describe the logical task in the same way as when talking to a manual tester, for example, "Select VISA as payment method and confirm."

• Use the same script even after the UI changes

  Talking to a member of your tester team, you would not spell out details such as "move the mouse and click the button in the bottom left labeled Confirm". UI elements can change during development, for example, the "Confirm" button could be renamed to "Accept". Obviously, you would describe the intended goal of the action. The advantage of describing the logical task is that a manual tester —just as the AI Assistant— can follow the same instructions, even if the UI changes later. For example, the AI Assistant —just as a manual tester— can identify the button that confirms the selection and click it, no matter what it is called, or where on the screen it is.

• Wait instead of fail

  If your app is affected by irregular network latency, or a procedure is disrupted by a modal dialog, a hard-coded script can miss its cue. Instead, tell the AI Assistant to wait, to attempt to repeat an action until it is successful, to dismiss any modal dialog, or to wait for a progressbar to finish.

• Write once, run everywhere

  Once you have fine-tuned the instructions, you can even run the same Scriptless test across a variety of devices and validate the same test steps on them. This saves time otherwise spent with repetitive test development and duplicated maintenance. Since you do not refer to device-dependent UI elements, the AI Assistant interprets the instructions relative to what it sees on the device screen.

• Focus

  To improve interaction precision with small or unclear elements, instruct the assistant to focus on a certain area. For example, use this more robust approach to select densely packed checkboxes, small text fields, or complex filter inputs. In these cases, tell the assistant first to "focus on" the section, to "look closely at ...," to "zoom into...," to "look at the small the text in...," or similar.

How to phrase AI Commands

When phrasing AI Commands, do the following:

• Be specific

• Use objective and clear definitions

• Ask about contents that are visible on the device screen

An AI Command can perform an action or validate a condition. Describe the intended outcome in both cases.

• A user action changes the device state.

• A validation verifies a condition and produces a binary result: PASS or FAIL.
  Do not ask open-ended questions that ask what, who, where, when, why, or how as a test step.

Here are some good and bad examples for how to phrase the input.

Don'ts	Do's	Why?
⨯ Move the mouse to the button labeled Cart in the bottom right corner	√ Open the shopping cart	Describe the logical goal. Do not refer to locators or labels that can change during development
⨯ Fill in the form	√ Enter "Jill A. Smith" in the name fields. Enter Main street 1 in NYC, USA, as the address.	Don't be vague. Provide enough details for the AI Assistant to be able to repeat the task consistently.
⨯ Click the red shoes.	√ Select the red sneakers that cost $30.	Don't be ambiguous. Can it happen that the UI contains several similar elements? Use specific words.
⨯ Click the second menu item from the top right.	√ Select Home from the navigation menu.	When referring to UI elements, use platform agnostic terminology so the test runs on mobile and web.
⨯ Did it work?	√ Verify the red sneakers are in the shopping cart on the screen.	Don't be vague. Describe the precise validation criterion that makes the test pass or fail.
⨯ Who is logged in?	√ Is Jill Smith logged in on this screen?	Validations are statements that verify or confirm facts as true or false.
⨯ How many search results were found?	√ Verify that search results have loaded on the screen.	Validations are questions that have a yes or no answer.
⨯ Does pizza taste good? ⨯ How does the pizza taste?	√ Is the rating of the selected pizza shown on the screen higher than 4?	Validations must be objective and measurable.

General validation knowledge

The AI Command is not limited to visual validations on the device screen. You can also ask it questions about variable values and general public-domain knowledge. The condition remains that you must phrase the question so that there is a binary answer that can be either PASS or FAIL.

The following questions are examples of general knowledge validations:

• "Is '400 First Avenue North #400, Minneapolis, MN 55401' a valid US address?"

  • "Is '${addr}' a valid US address?"

• "Is 4 bigger than 5 and also divisible by 3?”

  • "Is ${x} bigger than 5 and also divisible by 3?"

• "Is ${value} between 1 and 10?"

• "Is the account balance equal to ${balance}?"

• "Does ${message} provide enough specific information as an error message?"

• “Is ${name} a French name?”

• "Is ${transaction_date} equal to today's date?" (the environment date in UTC)

The assistant cannot answer subjective, non-factual questions such as, "Is the pizza tasty?" or "Is the advertized car beautiful?"

Specialized knowledge

Perfecto AI's general knowledge can reach its limits when testing applications with specialized UI elements, such as airplane seat plans, custom dashboards with diagrams, or canvases with drag-and-droppable elements.

You don't need to explain this specialized context as part of every input. To include global user instructions, describe your user interface once, and the AI commands will automatically include this knowledge. This option exists for the Perfecto AI assistants as well as AI Scriptless and Selenium/Appium commands.

1. Write down your specialized knowledge in a plain text file.

   Examples:

   • "The dashboard pie chart has a legend on the right-hand side. Percentages are hidden by default. Clicking on the pie charts displays the percentage as a number over the chart."

   • "The top left number in blue indicates the level. The top right number indicates the current score."

2. Save the file under the name ai.txt.

3. Upload the ai.txt file to the root of the Perfecto Repository.

   The knowledge now appears in the AI Assistant as part of your request, as general user instructions.

   When a test execution uses AI commands with global knowledge, Perfecto adds ai.txt to the execution report as a text attachment that you can download. This applies to AI Scriptless and Selenium/Appium AI commands flows. If global knowledge is not used during the test, ai.txt is not attached.

To update the knowledge, reupload the file with new content. The knowledge reloads when you log in.

Wait on loading screens

Human testers easily recognize load screens as temporary obstacles and wait before validating the screen. Similarly, the AI Command can detect when a screen is still loading, and whether it is loaded partially or fully. If the page isn’t ready yet, the command automatically retries every few seconds until the screen is fully loaded.

To make the test wait explicitly in specific cases, add an AI Command such as, “Wait for the login button to appear, and then continue.” This ensures the validation is performed only after the expected screen is fully visible.

Time awareness

The AI Assistant uses UTC (Coordinated Universal Time) as its standard reference time, allowing you to ask questions such as:

• Does the time on the screen match the current time?

• Does the time on the screen displayed in a US timezone correspond to the current UTC time?

• Is the selected date yesterday's date?

If needed, the reference time can be used to convert it to a specific timezone.

Install mobile applications

When you use an AI Command to install a mobile app on the device, Perfecto detects that this is not an interaction with the content of the screen but with the device system. Perfecto then helps you configure the necessary instrumentation:

1. Copy the name of the app from the application repository. You can use the full repository path.

2. Input an AI Command. For example:

   Install app_from_repo.ipa with camera instrumentation

   Install ${myApp} on the device

   Install public:/path/myapp.ipa

   Install ${myApp} on the device, open it and log in

3. In the configuration widget that appears, enable instrumentation features as needed: Sensor, WebView, Secured Screen, Resign Application. To learn more, see Instrumentation.

4. Under Launch after installing, select whether you want to start the app next or whether you want to continue with the Assistant dialog.

5. Click Apply to save the command.

Consider the following when installing iOS apps:

• To install apps on virtual iOS devices, provide the file name in the format appname.zip.

• To install apps on real iOS devices, the Resign Application option is enabled by default and mandatory. Only if you enable WebView, Resign Application is optional.

Fill in forms

When using AI Commands to fill in forms, instead of filling each field as a separate action, write one prompt combining several value inputs. Then tell the Assistant to click the submit button as a separate action.

Example:

Without Fill Form functionality	With Fill Form functionality
Fill in username John Enter last name Doe Select year 2001 Select UK from the drop down Submit the form	Fill in username John, last name Doe, year 2001. In the drop down, select UK. Submit the form

The following table lists supported field types.

Form field	Supported types
Text	Standard inputs, textareas, MUI TextField
Select	Native HTML <select>
Date	<input type="date"> in the format YYYY-MM-DD
Checkbox / Radio button	Standard checkboxes, radio buttons, MUI Checkbox/Switch
Combobox	MUI Autocomplete/Select, Ant Select, and similar libraries

The following field types are not supported:

• Masked inputs (for example, Stripe, react-input-mask)
• Custom date pickers (wheel, calendar dialogs)
• Sliders
• File upload inputs
• Rich-text editors (Lexical, ProseMirror, Quill, Slate, TinyMCE)
• Chip / tag inputs
• Cross-origin iframes
• Closed Shadow DOM
• Canvas-based inputs (e.g., signature pads)

Scrolling and swiping

The following tables show sample instructions and clarify how the assistant interprets the direction.

Touch interface (mobile screen)

Instruction	Finger swipes	What comes into view
Scroll up	down	content toward the top
Scroll down	up	content toward the bottom
Scroll left	right	content toward the left
Scroll right	left	content toward the right
Swipe up	up	content toward the bottom
Swipe down	down	content toward the top
Swipe left	left	content toward the right
Swipe right	right	content toward the left

Web interface (mouse scroll wheel)

Instruction	Mouse wheel motion
Scroll up	wheel up
Scroll down	wheel down
Scroll left	Shift + wheel up
Scroll right	Shift + wheel down

Limitations

The following limitations apply to AI-powered validations:

• AI validation models are not perfect, they can make mistakes. Before assigning a value to an AI-powered command in AI Scriptless, always test the validation using Perfecto's interactive AI assistant.

• AI validation models are not connected to your backend, or to the world. This means, it doesn't know your local time or date, only the device environment time in UTC. If the information does not appear in the context of the device screen and is not factual public domain knowledge, it cannot be a validation.

• Do not ask open-ended questions (what, who, where, when, why, how, ...) as validations.

Need help?

If you need help with a complex user action or validation, share it with us—we love a challenge! Reach out to Perfecto Support and include a screenshot with the exact prompt you used and information about how you wanted to interact with the UI.