Compare screens with AI vision

The AI Visual Comparison command compares two device screens like a human tester would. The assistant detects and describes visual differences in test result screenshots, and fails the test if the differences are in certain categories. This AI command is currently available only for mobile, not for tablets nor web testing.

• You define a screen shown in a test as the baseline. Later test runs are called checkpoints. Perfecto compares the checkpoints to the baseline and highlights differences. Setting a baseline is required for AI Visual Comparison.
  Tip: Choose a unique name. Name the baseline after the screen you are comparing, for example, Login Screen or Shopping Cart.

• To customize the command, select the specific Failure Criteria categories that shall fail the test. Failure criteria are required.

Initialize the visual comparison

To perform a visual comparison and receive results, first initialize a baseline.

1. Open a device.

2. In the left sidebar, select the AI tab.

3. Drag and drop, or double-click the AI Visual Comparison command to add it to the script editor.

4. Navigate to the Fail Criteria parameter and select any.

5. Navigate to the Baseline ID parameter and click Edit. Give the baseline a name, for example, Login Screen or Shopping Cart.

6. Save the test.

7. To initialize the baseline, run the test.
   

After its first test run, the AI Visual Comparison pane in the results window is disabled, because there is nothing to compare yet. The log contains the line "No visual comparison yet. Baseline was captured, execute again."

Define failure criteria

To be able to get comparison results, define failure criteria.

1. Select the AI Visual Comparison command in the script editor.

2. Navigate to the Fail Criteria parameter and click to open the drop down.

3. Select one or more visual differences that will fail the test:

   • Device - These are common, expected differences that typically do not fail tests. For example, the device shows a different signal strength, different battery level, different time of the day, or an additional open browser tab.
     You typically do not select this as a failure criterion, but you could if needed.

   • Style - Text style, font, text size, color, alignment, borders, shading, or similar, have changed in the checkpoint. The visual context and values stayed the same as in the baseline.

   • Value - Text content or numeric values have changed in the checkpoint. The style and context stayed the same as in the baseline.

   • Missing - An element (text, icons, buttons, tooltips, headers, footers) that appears in the baseline is absent in the checkpoint.

   • Moved - An element (text, icons, buttons, tooltips, headers, footers) that appears in the baseline is in a different position in the checkpoint.

   • Addition - An element (text, icons, buttons, tooltips, headers, footers) that is absent in the baseline has appeared in the checkpoint.

   • Error - An error message has appeared, GUI elements are cut off, GUI elements overlay each other, and similar.

   • Uncategorized - Other significant detected differences that don't fit any previous category.

   • Pixel difference - An algorithmic pixel-by-pixel comparison has detected differences, but the AI Assistant does not consider them significant as defined by the above categories. These are usually artifacts caused by normal anti-aliasing, image compression, or image scaling. You typically do not select this as a failure criterion, because it triggers almost every time, but you could in special cases.

4. Save the script.

Get comparison results

After initializing the baseline and configuring failure criteria, you get comparison results when running the test.

1. Open the test and select the AI Visual Comparison command in the script editor.

2. Navigate to the Fail Criteria parameter and select one or more criteria from the list.

3. Select an existing baseline or create a new one.

   • Script editor: Double-click the AI Visual Comparison command or select Edit from its Actions menu.

   • Parameters pane: Click the Baseline ID field to open the baseline picker.

   Type a name to filter and select a baseline from the list. If no baseline matches, click Add as a new name.

   Each baseline entry shows the baseline name and device information (a single device name or number of devices). An image of the baseline is displayed on the right side. Click it to enlarge it.

4. Run the test, then click View Results.

5. On the STR screen, click AI Visual Comparison.

6. In the AI Visual Comparison pane, review the differences highlighted in the baseline and the checkpoint screenshots.

   The Assistant highlights all detected categories.

7. View results:

   1. Hover the pointer over a highlight to read the detailed report for the detected difference.

   2. Hover the pointer over the baseline header to view the baseline name.

   3. Use the buttons to zoom in and out or view the screenshot in fullscreen size.

8. If any detected categories are failure criteria, the test fails.

Set a new baseline

Choose one of two ways to create a new baseline.

1. Open the test and select the AI Visual Comparison line.

   • Script editor: Double-click the AI Visual Comparison line or select Edit from its Actions menu. Type a new unique name and select Add as a new name to create a new baseline.

   • Parameters pane: Enter a new unique name in the Baseline ID field.

2. Run the test to initialize the new baseline.

3. Run the test to get the report.

Or from the STR screen:

1. View the results showing the baseline and the current checkpoint.

2. Click the button next to the checkpoint header to make the current result the new baseline.

3. Confirm the dialog.

   The baseline is applied from the next test run on.