Perform AI-driven user actions

The AI User Action Assistant can navigate the device screen like a user would. Phrase your prompts like you would when describing a test step to a manual tester.

To have the test interact with the application, you no longer need to identify object locators or learn a scripting language. The AI Assistant understands natural language like "select London as travel destination." The assistant interprets your instructions accordingly as button clicks, menu selections, typing values in fields, and so on. The prompt is then included as the value for the Action parameter in the AI User Action command.

For examples and limitation, see Best practices for working with AI User Actions

Important: To have this functionality set up in your cloud, contact your Perfecto account representative. A separate license is required to access this feature.

To learn how to add user actions with the AI User Action command, watch this video or review the instructions following the video.

To add a user action:

To be able to insert the AI User Action command, open a device first.

  1. In the test editor, double-click the AI User Action command. This adds the command to the test.

  2. In the parameters pane at the bottom, navigate to the Action parameter and click the AI Assistant button .

    The AI Assistant button is not visible if this parameter is used with a variable instead of a constant.

    Important: When the AI Input assistant is open in the left pane, the device screen is locked. To interact with the device screen, click the Unfreeze device screen button .

  3. In the AI Input assistant pane, at the bottom, describe the logical goal of a user action to perform, such as "Log in as Joe with password 123."

  4. (Optional) Include one or more variables in your user action:

    • To add a variable, type ${. This brings up the list of existing variables that have been defined.

    • To create a new variable, enter it in the following format: ${variable-name}

  5. The Assistant confirms by repeating the instructions as it understood them. Verify that the Assistant interpreted the command as intended.

    • If it needs time to process, the Assistant will say "Working on it..."
    • If needed, cancel the processing by starting to type another prompt or by clicking the Stop button.

  6. (Optional) During your conversation with the assistant, click thumbs-up on good answers, or click thumbs-down if the assistant's answer was not helpful. A screenshot and the reported response will be sent to the Perfecto AI feedback list. To get more information on the answers provided by the assistant, click the Explain button .

  7. Click Apply to save the user action command.

  8. (Optional) Click Copy to paste this code snippet into an existing Appium or Quantum script.