AI user action (FR)

Perfecto command

perfecto:ai:user-action

Purpose

Use Perfecto's perfecto:ai:user-action command in Appium, Selenium, or Quantum tests while running tests on real and virtual Perfecto devices.

To learn more about working with AI-powered user actions, see the following resources:

• AI commands FAQ

• Best practices for working with AI User Actions

Parameters

Return value

If outputVariable= "false" or not provided, the return value is true or false.

If outputVariable="true", the return value is an Object:

• result: boolean

• outputVariables: Map of <String, Object>

For a successful user action, you get an object where result="true". For example, for a variable named ${country} with the value "France":

{
    "result": true,
    "outputVariables": {
        "country": {
            "name": null,
            "value": "France"
        }
    }
}

For an unsuccessful user action or a policy violation, you get the following object:

{
    "result": false,
    "outputVariables": {}
}

If no output variables were created, you get the following object:

{
    "result": true,
    "outputVariables": {}
}

Test failure

The test fails and the action aborts if the action could not be performed.

Exceptions

The following message appears when your cloud does not have the consent to use AI:

Failed to execute command ai-user-action: Unexpected error occurred while processing command 'ai-user-action'. Error message: AI commands are disabled. To use these commands, contact your administrator. Status: Forbidden

JavaScript examples

Without using an output variable in the user action, the constant result will be true or false.

const result = await driver.execute('perfecto:ai:user-action', { 
    action: "Open new order screen!", 
    reasoning: false,
    outputVariable: false
});
console.log(result); // true or false

A user action that is using an output variable:

const response = await driver.execute('perfecto:ai:user-action', { 
    action: "Take the temperature from the screen and put it in ${temp}", 
    reasoning: false,
    outputVariable: true
});

This call returns the following response:

{
  "result": true,
  "outputVariables": {
    "temp": {
      "name": null,
      "value": "21"
    }
  }
}

Java examples

Without using an output variable in the user action, the Boolean result will be true or false.

Boolean result = (Boolean) driver.executeScript("perfecto:ai:user-action",
    Map.of(
                "action", "Open new order screen!",
                "reasoning", false,
                "outputVariable", false
    )
);

A user action that is using an output variable:

// Execute Perfecto AI User Action
Object raw = driver.executeScript(
    "perfecto:ai:user-action",
    Map.of(
        "action", "Take the temperature from the screen and put it in ${temp}",
        "reasoning", false,
        "outputVariable", true
    )
);

// Convert raw response to Map
Map<String, Object> response = (Map<String, Object>) raw;
boolean success = (Boolean) response.get("result");
System.out.println("Success = " + success);

// Extract output variables
Map<String, Object> outputVars = (Map<String, Object>) response.get("outputVariables");

if (outputVars != null) {
    for (Map.Entry<String, Object> entry : outputVars.entrySet()) {
        String variableName = entry.getKey();
        Map<String, Object> variableObject = (Map<String, Object>) entry.getValue();
        String value = (String) variableObject.get("value");
        System.out.println(variableName + " = " + value);
    }
}

In this example, the value is "21". The raw Object contains the following response:

{
	"result": true,
	"outputVariables": {
		"temp": {
			"name": null,
			"value": "21"
		}
	}
}