AI user action (FR)

Use the Perfecto AI user action command to perform AI-driven actions on a device and retrieve results from the command response.

Perfecto command

perfecto:ai:user-action

Purpose

Use Perfecto's perfecto:ai:user-action command in Appium, Selenium, or Quantum tests while running tests on real and virtual Perfecto devices.

To learn more about working with AI-powered user actions, see the following resources:

• AI commands FAQ

• Best practices for working with AI User Actions

Parameters

Return value

The return type depends on two factors:

• The value of the outputVariable parameter

• The semantics of the prompt, as detailed in the following table.

Return type matrix

Boolean return value

When the command returns a Boolean, the value indicates whether the action was successfully performed.

Object return value

When the command returns an Object, it includes the following fields:

• result: Boolean

• outputVariables: Map<String, Object>

• validationResponse: Object (present when the prompt includes validation semantics)

For a successful user action, you get an Object where result is true. For example, for a variable named ${country} with the value "France":

{
  "result": true,
  "outputVariables": {
    "country": {
      "name": null,
      "value": "France"
    }
  }
}

For an unsuccessful user action or a policy violation, you get the following Object:

{
  "result": false,
  "outputVariables": {}
}

If no output variables were created, you get the following Object:

{
  "result": true,
  "outputVariables": {}
}

Best practices

To avoid runtime casting errors, handle the return value:

• If the prompt can include validation languagetreat the return value as an Object.

• If you expect a Boolean return value, use an instanceof check before casting.

Test failure

The test fails and the action aborts if the action could not be performed.

Exceptions

The following message appears when your cloud does not have the consent to use AI:

Failed to execute command ai-user-action: Unexpected error occurred while processing command 'ai-user-action'. Error message: AI commands are disabled. To use these commands, contact your administrator. Status: Forbidden

JavaScript examples

Without using an output variable in the user action, the constant result will be true or false.

const result = await driver.execute('perfecto:ai:user-action', { 
    action: "Open new order screen!", 
    reasoning: false,
    outputVariable: false
});
console.log(result); // true or false

A user action that is using an output variable:

const response = await driver.execute('perfecto:ai:user-action', { 
    action: "Take the temperature from the screen and put it in ${temp}", 
    reasoning: false,
    outputVariable: true
});

This call returns the following response:

{
  "result": true,
  "outputVariables": {
    "temp": {
      "name": null,
      "value": "21"
    }
  }
}

Java examples

The following examples show how to handle the return value of the AI user action command in Java, depending on whether the prompt returns a Boolean or an Object.

Without using an output variable

Without using an output variable in the user action, the return value can be true or false. If the prompt includes validation language, the command can return an Object instead of a Boolean.

Boolean result = (Boolean) driver.executeScript("perfecto:ai:user-action",
    Map.of(
                "action", "Open new order screen!",
                "reasoning", false,
                "outputVariable", false
    )
);

A user action that is using an output variable

When a user action creates an output variable, the command returns an Object that contains the result and the outputVariables map.

// Execute Perfecto AI User Action
Object raw = driver.executeScript(
    "perfecto:ai:user-action",
    Map.of(
        "action", "Take the temperature from the screen and put it in ${temp}",
        "reasoning", false,
        "outputVariable", true
    )
);

// Convert raw response to Map
Map<String, Object> response = (Map<String, Object>) raw;
boolean success = (Boolean) response.get("result");
System.out.println("Success = " + success);

// Extract output variables
Map<String, Object> outputVars = (Map<String, Object>) response.get("outputVariables");

if (outputVars != null) {
    for (Map.Entry<String, Object> entry : outputVars.entrySet()) {
        String variableName = entry.getKey();
        Map<String, Object> variableObject = (Map<String, Object>) entry.getValue();
        String value = (String) variableObject.get("value");
        System.out.println(variableName + " = " + value);
    }
}

In this example, the value is "21". The raw Object contains the following response:

{
    "result": true,
    "outputVariables": {
        "temp": {
            "name": null,
            "value": "21"
        }
    }
}

Handle mixed return types safely

If the prompt can include validation language (for example, check, verify, confirm, or validate), the return value can be either a Boolean or an Object. Use the following defensive pattern to avoid casting errors.

// Safe defensive pattern
Object result = driver.executeScript("mobile:ai:userAction", params); 
if (result instanceof Boolean) { 
  boolean actionResult = (Boolean) result; 
} else if (result instanceof Map) { 
  Map<String, Object> validationResult = (Map<String, Object>) result; 
  // handle outputVariables and validationResponse 
}