Basic test result concepts

The Perfecto Smart Reporting feature optimizes the visibility of test results for enhanced analysis. To gain the maximum benefits, this section provides an overview of the basic concepts.

Smart Reporting requires an implementation of the Reporting SDK. 

Custom fields

Custom fields are the recommended way of refining your filter and group results beyond the built-in filter categories (devices, jobs, operating system, browser, and platform type) and user-defined tags. They add another level of granularity in the form of <name, value> pairs that allow you to drill-down to specific tests. For example, to identify a particular tester, you may want to define a "tester" tag that should be "John" for some tests and "Judy" for other tests. In this case, you could define a custom field called "tester" with the values "John" and "Judy". This way, you can use custom fields to add information to a specific test that requires separate values.

Other use case examples:

  • Filter or group tests based on error messages by providing the failure reason as the custom field value
  • Filter or groups tests based on author metadata for future test maintenance
  • Work with automated test approvers for future test maintenance
  • Map tests to test management tools by using test artifact IDs from different sources, such as JIRA, Raily, and so on, for better traceability of requirements

When you add a custom field to your filter, you can specify whether to include or exclude a given value. 

You can set custom fields at the execution context instance (when the reporting client is initialized), the test context instance (test start/stop method calls), or (if you are using a CI tool) as part of the JVM command line parameters. The order of precedence of conflicting custom field values is:

  1. JVM values (highest)
  2. TestContext values (medium)
  3. ExecutionContext values (lowest)

The following code samples illustrate how to implement custom fields using the Perfecto Execution Context.

Copy

Perfecto execution context | Java sample code

    // Define other fields like tags, project, and job details.
    Project project = new Project(reportConstants.getProjectName(), reportConstants.getProjectVersion());

    Job job = new Job(reportConstants.getJobName(), Integer.parseInt(reportConstants.getJobVersion()))
    .withBranch("awesome_vcs_branch");


    // create a list of custom fields.
    List<CustomField> customFields = new ArrayList<CustomField>();

    // Define a custom field with Key value pair.
    CustomField author = new CustomField("author", "sdet1@awesomecompany.com");

    // Add the custom field to custom field collection.
    customFields.add(author);
    
    // Perfecto Execution Context definition.
    PerfectoExecutionContext execContext = new PerfectoExecutionContext.PerfectoExecutionContextBuilder()
    .withContextTags(tags)            // Add tags to Perfecto Execution Context    
    .withProject(project)            // Add Project details to Perfecto Execution Context
    .withJob(job)                    // Add Job details to Perfecto Execution Context    
    .withCustomFields(customFields)    // Add Custom fields to Perfecto Execution Context
    .withWebDriver((WebDriver) driver).build();
Copy

Perfecto execution context | Javascript sample code

    // Define CustomFields Object
    var customFieldsObject = {
        'author':'sdet1@awesomecompany.com'
    };

    // Initializing Perfecto Execution Context
    var executionContext = await new Reporting.Perfecto.PerfectoExecutionContext({
        webdriver: browser.driver,
        job: new Reporting.Model.Job({
            jobName:  this.pJobName,
            buildNumber: this.pJobNumber,
            branch: this.pBranchName
        }),
        customFields: customFieldsObject,
        tags: ["awesome"]
    })
   
    // Initializing Reportium client
    browser.reportingClient = await new Reporting.Perfecto.PerfectoReportingClient(executionContext);

You can also add custom fields via the testStart or testEnd methods. With this implementation, custom fields are created based on data created by the test itself.

Copy

Test context sample

@Test
public void myTest() {
reportiumClient.testStart("myTest", new TestContext.Builder()
                                        .withTestExecutionTags("Sanity", "Nightly")
                                        .withCustomFields(new CustomField("version", "OS11"))
                                        .build());
...
}

In the Report Library and Heatmap views, the list of custom fields is available in the right pane from the ADD FILTER GROUP option. You can also group by custom fields and add a custom field column to the report table. For information on how to filter by custom fields, see Report Library.

Tags

Smart Reporting allows you to attach a set of tags to a test or a specific subset of the test. You may later use these tags to create different cross-sections of the test results and to focus on a specific subset of test reports.

Best Practice: For greater flexibility and more refined filter and group results, consider the use of custom fields.

It is important to use tags that are meaningful and valuable to your teams. Tags should be classified into different buckets:

  • Execution level tags: Configured across all tests in a particular execution.
  • Single test tags: Identify specific tests that may exercise a particular subset of the application.
  • Logical step names: Identify the different logical execution steps of the test.

To learn more about how to optimize your tags for easier report analysis, see Tag-driven reports (RTDD workflow).

Context classes

The Reporting SDK includes the following Context classes used to declare parameters associated with different sets of tests:

The PerfectoExecutionContext class is used to associate different settings with general text execution. You create instances of this class using a builder method - PerfectoExecutionContextBuilder. Using the builder method, you can also declare the following settings with this class:

  • Project identifier - use the "withProject()" method to set a label for the execution to associate with a particular testing project.
  • Job identifier - use the "withJob()" method to associate the job identifier (especially when running tests within a CI tool) of a particular run with an execution.
  • Tags - use the "withContextTags()" method to declare a set of tags to be associated with all tests run in this execution.
  • Custom Fields - use the "withCustomFields()" method to define a collection of <"name", "value"> pairs of information associated with all tests run in this execution.

All of these settings are associated with all tests executed within the entire Execution unit. This association could be used to filter all of the tests relative to a particular execution when examining the actual reports (see below for more information).

In addition, the class supports the following methods:

  • withWebDriver() - defines the automation driver that the Reporting system will use to collect the execution data and artifacts from.
  • build() - activates the builder to configure the Reporting execution.

The TestContext class supports defining a separate set of data associated with a particular test execution. Instances of the TestContext class are configured using the Builder() method. You may provide:

  • A list of specific tags, using the withExecutionTags() method, as a comma separated list of tags, to the instance builder.
  • Custom field <name, value> pairs, using the withCustomFields() method, as a collection of CustomField objects.
Copy

Using the TestContext Constructor

reportiumClient.testStart("myTest", new TestContext.Builder()
                   .withTestExecutionTags("Sanity", "NewNote", "Android")
                   .withCustomFields(new CustomField("team", "dbApps"))
                   .build());

Job identification

When using Continuous Integration tools to automatically generate and run your tests, for example when performing regression testing, the CI tool associates a Job Name and Job Number with each run of the process. In addition, you may have different versions of the tests defined in different branches of the repository. Import these identifiers into your tests and tag the tests to associate the Single Test Reports with these CI Jobs. Learn how to import the job identifiers for different CI Tools here. Then view the Test summary reports in the CI Dashboard.

Test steps

Within each test, it is possible to group the different executed commands into logical steps. You can identify each step by a step name, and the step name displays in the test report together with any data and artifact (for example, a screenshot or video) associated with the commands executed by the step. Identifying the steps is part of the ReportiumClient stepStart() method, and attaching a result for the step is part of the stepEnd() method.