Basic test result concepts

The Perfecto Smart Reporting feature optimizes the visibility of test results for enhanced analysis. To gain the maximum benefits, this section provides an overview of the basic concepts.

Smart Reporting requires an implementation of the Reporting SDK

Custom fields

Custom fields are the recommended way of refining your filter and group results beyond the built-in filter categories (devices, jobs, operating system, browser, and platform type) and user-defined tags. They add another level of granularity in the form of <name, value> pairs that allow you to drill-down to specific tests. For example, to identify a particular tester, you may want to define a "tester" tag that should be "John" for some tests and "Judy" for other tests. In this case, you could define a custom field called "tester" with the values "John" and "Judy". This way, you can use custom fields to add information to a specific test that requires separate values.

Other use case examples:

  • Filter or group tests based on error messages by providing the failure reason as the custom field value
  • Filter or groups tests based on author metadata for future test maintenance
  • Work with automated test approvers for future test maintenance
  • Map tests to test management tools by using test artifact IDs from different sources, such as JIRA, Raily, and so on, for better traceability of requirements

When you add a custom field to your filter, you can specify whether to include or exclude a given value. 

You can set custom fields at the execution context instance (when the reporting client is initialized), the test context instance (test start/stop method calls), or (if you are using a CI tool) as part of the JVM command line parameters. The order of precedence of conflicting custom field values is:

  1. JVM values (highest)
  2. TestContext values (medium)
  3. ExecutionContext values (lowest)

The following code samples illustrate how to implement custom fields using the Perfecto Execution Context.

Copy

Perfecto execution context | Java sample code

    // Define other fields like tags, project, and job details.
    Project project = new Project(reportConstants.getProjectName(), reportConstants.getProjectVersion());

    Job job = new Job(reportConstants.getJobName(), Integer.parseInt(reportConstants.getJobVersion()))
    .withBranch("awesome_vcs_branch");


    // create a list of custom fields.
    List<CustomField> customFields = new ArrayList<CustomField>();

    // Define a custom field with Key value pair.
    CustomField author = new CustomField("author", "sdet1@awesomecompany.com");

    // Add the custom field to custom field collection.
    customFields.add(author);
    
    // Perfecto Execution Context definition.
    PerfectoExecutionContext execContext = new PerfectoExecutionContext.PerfectoExecutionContextBuilder()
    .withContextTags(tags)            // Add tags to Perfecto Execution Context    
    .withProject(project)            // Add Project details to Perfecto Execution Context
    .withJob(job)                    // Add Job details to Perfecto Execution Context    
    .withCustomFields(customFields)    // Add Custom fields to Perfecto Execution Context
    .withWebDriver((WebDriver) driver).build();
Copy

Perfecto execution context | Javascript sample code

    // Define CustomFields Object
    var customFieldsObject = {
        'author':'sdet1@awesomecompany.com'
    };

    // Initializing Perfecto Execution Context
    var executionContext = await new Reporting.Perfecto.PerfectoExecutionContext({
        webdriver: browser.driver,
        job: new Reporting.Model.Job({
            jobName:  this.pJobName,
            buildNumber: this.pJobNumber,
            branch: this.pBranchName
        }),
        customFields: customFieldsObject,
        tags: ["awesome"]
    })
   
    // Initializing Reportium client
    browser.reportingClient = await new Reporting.Perfecto.PerfectoReportingClient(executionContext);

You can also add custom fields via the testStart or testEnd methods. With this implementation, custom fields are created based on data created by the test itself.

Copy

Test context sample

@Test
public void myTest() {
reportiumClient.testStart("myTest", new TestContext.Builder()
                                        .withTestExecutionTags("Sanity", "Nightly")
                                        .withCustomFields(new CustomField("version", "OS11"))
                                        .build());
...
}

In the Report Library and Heatmap views, the list of custom fields is available in the right pane from the ADD FILTER GROUP option. You can also group by custom fields and add a custom field column to the report table. For information on how to filter by custom fields, see Report Library.

Tags

Smart Reporting allows you to attach a set of tags to a test or a specific subset of the test. You may later use these tags to create different cross-sections of the test results and to focus on a specific subset of test reports.

Best Practice: For greater flexibility and more refined filter and group results, we recommend the use of custom fields.

It is important to use tags that are meaningful and valuable to your teams. Tags should be classified into different buckets:

  • Execution level tags: Configured across all tests in a particular execution.
  • Single test tags: Identify specific tests that may exercise a particular subset of the application.
  • Logical step names: Identify the different logical execution steps of the test.

To learn more about how to optimize your tags for easier report analysis, see Tag-driven reports (RTDD workflow).

Context classes

The Reporting SDK includes the following Context classes used to declare parameters associated with different sets of tests:

Job identification

When using Continuous Integration tools to automatically generate and run your tests, for example when performing regression testing, the CI tool associates a Job Name and Job Number with each run of the process. In addition, you may have different versions of the tests defined in different branches of the repository. Import these identifiers into your tests and tag the tests to associate the Single Test Reports with these CI Jobs. Learn how to import the job identifiers for different CI Tools here. Then view the Test summary reports in the CI Dashboard.

Test steps

Within each test, it is possible to group the different executed commands into logical steps. You can identify each step by a step name, and the step name displays in the test report together with any data and artifact (for example, a screenshot or video) associated with the commands executed by the step. Identifying the steps is part of the ReportiumClient stepStart() method, and attaching a result for the step is part of the stepEnd() method.