Best practices for working with AI User Actions

Using an AI Assistant to perform user actions on the device under test opens up new ease of use in automated testing. To learn what you can and cannot do with AI User Actions, review the following best practices.

In Scriptless Mobile, AI User Actions are available through AI commands. For Appium testing, AI-based validations are available through AI functions.

What can the Assistant do?

For optimal performance, phrase each User Action so it does not resolve to more than 15 steps and so that it takes less than 3 minutes.

We have tested the following AI User Actions with excellent results:

  • Identify and use common UI elements, such as the Hamburger menu, help icons, notification icons

  • Identify and click buttons, hyperlinks, icons, and floating action buttons (FAB)

  • Identify and type text into empty text fields, or replace their text

  • Select radio buttons and checkboxes, set values using sliders

  • Select tabs, entries from lists, items from navbars

  • Read and dismiss popups and notifications that interrupt the action

  • Browser navigation (goto, back, forward, refresh, clean)

The following actions are still being tested, and may or may not work:

  • Identify a text field when the label is inline, above, left, or right of the field

  • select a cell from a table, select an action from a row in a table

  • select a color

We have not yet tested the following actions:

  • Append text to existing text in field

Be aware that a Large Language Model can make mistakes, the model is being constantly improved.

Tip: The AI Assistant will always try to perform the User Action, but the more complex the task is, the higher is the chance of failure. Correct the Assistant if it misunderstands, and evaluate its next suggestions until you find the perfect input. Experiment! We love challenges and are happy to get feedback at Perfecto Support whenever something unexpected happens!

What can I do that I couldn't do before?

During development, UI elements and procedures can change. Buttons can be moved to a different location or are relabeled. But you are already running tests. When you define test scripts traditionally as actions applied to locators, the test script breaks every time a UI element changes.

Using the AI Assistant, you can describe the logical task in the same way as when talking to a manual tester, for example, "Select VISA as payment method and confirm."

  • Use the same script even after the UI changes

    Talking to a member of your tester team, you would not spell out details such as "move the mouse and click the button in the bottom left labeled Confirm". UI elements can change during development, for example, the "Confirm" button could be renamed to "Accept". Obviously, you would describe the intended goal of the action. The advantage of describing the logical task is that a manual tester —just as the AI Assistant— can follow the same instructions, even if the UI changes later. For example, the AI Assistant —just as a manual tester— can identify the button that confirms the selection and click it, no matter what it is called, or where on the screen it is.

  • Wait instead of fail

    If your app is affected by irregular network latency, or a procedure is disrupted by a modal dialog, a hard-coded script can miss its cue. Instead, tell the AI Assistant to wait, to attempt to repeat an action until it is successful, to dismiss any modal dialog, or to wait for a progressbar to finish.

  • Write once, run everywhere

    Once you have fine-tuned the instructions, you can even run the same Scriptless test across a variety of devices and validate the same test steps on them. This saves time otherwise spent with repetitive test development and duplicated maintenance. Since you do not refer to device-dependent UI elements, the AI Assistant interprets the instructions relative to what it sees on the device screen.

Best Practices

Here are some good and bad examples for how to phrase the input.

Don'ts Do's Why?
⨯ Move the mouse to the button labeled Cart in the bottom right corner √ Open the Shopping Cart Describe the logical goal. Do not refer to locators or labels that can change during development
⨯ Fill in the form √ Enter "Jill A. Smith" in the name fields. Enter "Main st 1 in NYC, USA" as the address. Don't be vague. Provide enough details for the AI Assistant to be able to repeat the task consistently.
⨯ Click the red shoes.

√ Select the red sneakers that cost $30.

 Don't be ambiguous. Can it happen that the UI contains several similar elements? Use specific words.

Need help?

If you need help with a complex user action, share it with us—we love a challenge! Reach out to Perfecto Support and include a screenshot with the exact prompt you used and information about how you wanted to interact with the UI.