Insights

Insights is designed to help you jump-start your root cause analysis (RCA) efforts, improve your testing success over time, and accelerate identification of real bugs in the tested application. Insights can help you with constantly improving your success rate and efficiently find the real bug in the tested application. The latter goal may be hard to achieve when many tests are failing for various reasons. Which error should you focus on first? Which test is the one that deserves immediate attention?

On this page:

Perfecto's AI-powered RCA approach

Perfecto's root cause analysis (RCA) feature now uses artificial intelligence (AI) to identify the underlying error behind test failures. This enhanced approach to RCA empowers you to identify the true root cause of failures quicker instead of having to painstakingly sift through possibly thousands of non-informative error messages daily.

How does this new approach work? Perfecto performs AI analysis in any of the following cases:

An "Element not found" or "Class not found" error occurs
A test execution fails without any assigned failure reason
The status of a test execution is unknown

Based on the information displayed on the screen at the moment of failure, Perfecto's AI process tries to define the root cause of the error, which in turn will allow you to efficiently address and resolve the underlying problem.

It is possible to change the failure reasons assigned by Perfecto's RCA feature to custom failure reasons. If this is something you need, contact Perfecto Support.

The Insights dashboard

The Insights dashboard includes different widgets that slice and dice the information in different ways so you can quickly decide which issues to investigate first, which tests to re-run, when to replace a device, and so on. They provide actionable data, advice, and solutions along with trending information and predictive analytics.

You can configure the data in this view through the configuration pane. For more information, see Group or filter data.

To access Insights:

Go to Test Analysis > Insights. When you open the dashboard this way, it shows by default data for the last 7 days.
In the CI Dashboard, click the icon, available in the right-most column and in the tooltips available for the Result History and Duration History columns.

Watch this short video for a demonstration of how to navigate and work with Insights.

Tip: Perfecto also provides information on the failure history of a particular test. This widget is available at the level of the single test report (STR). For details, see Test failure history.

Available widgets

Following is a short description of all widgets, including their logic:

Failures by category

Displays a pie chart that shows failed or blocked tests broken down by category. The categories are designed to help you work more efficiently. Perfecto expertise automatically classifies test failures by similar root causes that can be resolved using a similar approach. This kind of information jump starts your root cause analysis.

The following table describes the available categories. The mapping of failure reasons for failed tests is also available in the Failure reasons view. For the mapping of blocked failure reasons, see Blocked failure reasons.

Category	Description
Application issue	Failures related to issues with the application. These are typically caused by bugs in the tested application. We suggest that you check these first.
Lab	Failures related to physical issues on mobile or web devices, such as Wi-Fi problems or devices that are in error or disconnected.
Orchestration	Failures related to issues with organizing tests, such as failing to reserve a device or devices being already in use when a test is run.
Scripting	Failures that are most likely related to errors in the script, such as incorrect commands, parameters, or values. Custom failure reasons that existed in the system when Insights was first added are also mapped to this category.
Test data	Failures related to incorrect test data used to drive the tests, typically caused by outdated test data, insufficient volume of test data, or test data not meeting requirements, such as uniqueness.
Test environment	Failures related to the setup of the test environment, typically caused by incorrectly configured back-end services or unavailable third-party dependencies.
Test execution	Failures that cause a test to fail but that are not related to the application, scripting issues, or the environment. Possible examples are disabled notifications, disabled location settings, or login failures.

Each failure reason is mapped to a category. Uncategorized tests, which are tests that are not mapped to a failure reason, appear in 2 distinct groups: Uncategorized - Failed and Uncategorized - Blocked.

Interact: Click a group to open the Report Library in context, filtered by the selected category.

History and trends

Displays the following:

A statistical overview that shows general statistics of all tests run during the specified time frame, split into the number of passed, failed, blocked, and other tests. In addition, if Perfecto has detected performance issues, an incident indicator is displayed next to the total number of tests. Moving the pointer over the indicator displays a tooltip with the overall number of incidents detected. You can click the tooltip to view all reports with performance incidents in the Report Library.
A line graph that indicates how many tests run each day and how they perform over time. Also includes the option to show trendlines for tests with a status of Failed, Blocked, Passed, or Unknown.

Trendlines can forecast the general direction of your tests. Is the number of failing and blocked tests decreasing? Is the number of passing tests increasing? This kind of data provides clarity and helps you assess the situation and take action. The Failed trendline is displayed by default.

The following images show the History and trends widget with incident indicator. The right image also illustrates the tooltip with the number of performance incidents detected.

Interact:

Move the pointer anywhere along the graph's horizontal timeline to view the number of tests run on a particular day, by status.
In the Trendlines area at the bottom, select a checkbox to display the respective trendline (Failed, Blocked, Passed, or Unknown).

Most common issues

Lists the top issues that are the root cause of many failures, along with relevant details (such as device info or test name) and ordered by fail rate (highest to lowest). You can use these issues as starting points for fixing your tests.

The recommendations in this widget are based on the grouping options available in the Report Library (see also Group or filter data). The widget shows the top issue for some of those grouping options. Instead of filtering the reports in the library successively by all of the available grouping options, you can go to this widget first for an at-a-glance view of the highest-priority tests.

Interact: Click the link to open the Report Library in context of the selected grouping, filtered by the relevant status.

The following table provides examples of issues and recommended next steps.

Issue type	Description
Most common blocked reason	Lists the most prevalent blocked failure reason. Possible next steps: Open the Report Library and check the Exception summary column and/or Exception details column. Tests are grouped by failure reason, sorted by the number of blocked tests, and filtered by Blocked status. Focus on fixing one blocked reason to resolve the problem for all tests that are blocked by the same issue.
Most common exception summary	Lists the most prevalent cleaned-up exception, which is the same for several issues. Possible next steps: Open the Report Library and check the Exception summary column along with the Exception details column. Tests are grouped by Exception summary, sorted by the number of failed tests, and filtered by Failed status. Focus on fixing the exception.
Most common failure reason	Lists the most prevalent failure reason (of all categorized tests). Possible next steps: Open the Report Library and check the Exception summary column and/or Exception details column. Tests are grouped by failure reason, sorted by the number of failed tests, and filtered by Failed status. Focus on fixing one failure reason to resolve the problem for all tests that failed with the same issue.
Test with the most failures	Shows the test that failed most often (considering all failed tests). Possible next steps: Open the Report Library and check the Exception summary column and/or Exception details column. Tests are grouped by test name, sorted by the number of failed tests, and filtered by Failed status. Open the STR and look at the Failed test history widget to find out if the test is flaky or the issue is related to something else.
Mobile device with most failed tests	Shows the most problematic device (only considers real devices). Possible next steps: Make sure the device is connected and there is no physical issue with the device. Open the Report Library and check the Exception summary column and/or Exception details column. Tests are grouped by device ID, filtered by platform type Mobile, sorted by the number of blocked tests, and filtered by Blocked status. Look at the failure reason. Is the device in error? Is it in use?