Smart Reporting provides information regarding the logical steps of the test execution. The step information is based on the information provided to the reporting server via the Reporting SDK. This information includes:
- Logical step identification
- Commands sent to the Perfecto Lab interface
Often, accessing the original source code used to run the test is helpful to gain a better understanding of how to improve the test to perform the automation of the UI and application better. This section explains how.
Source code information
When displaying the STR for the test execution, it is helpful to correlate between the test run as shown in the STR and the source code of the test script executed. Finding the source code for the test is only possible if the tester provides certain path identification information to associate the source code file to the specific test. The STR will then be able to connect to the source files and open the proper view of the source in a browser window.
With the Perfecto API, you can use custom fields to provide the following source-code information units:
- Repository URL - The URL of the Version Control System (VCS) used to store the source code.
- File path - This is the relative file path within the repository for the specific source code file.
- Commit - Provides the identifier of the commit to the repository performed when running the test.
In addition, the branch information is taken into consideration when accessing the source files. To learn more on using the branch information see the relevant Integrate Smart Reporting with CI tools page.
Repository URL format
Smart Reporting supports accessing source file repositories in GitHub, GitLab, and Bitbucket. The URL can be copied from the repository's web site (see below for an example) and it will, usually, include:
- An indication of the VCS type. This may not be true, if the VCS is hosted on an internal machine, or addressed with an explicit IP address.
- The address of the VCS host.
- The repository's owner and repository name.
For example, in the repository URL: https://github.com/PerfectoCode/Samples -
- "https://github.com" is the address of the VCS host and includes the indicator that the VCS type is GitHub.,
- "PerfectoCode" is the owner, and "Samples" is the repository name.
- If the URL does not include an indication of the VCS type - then provide it as a prefix. For example, github:https://10.10.35.17/PerfectoCode/Samples
Smart Reporting supports URLs in either HTTP or SSH format - for example the previous example URL would be git@github.com:PerfectoCode/Samples in SSH format.
The general format of the Repository URL for Smart Reporting would be:
Most tools provide easy options to get this URL. For example, if you are using GitHub:
-
Browse to the Git repository and click Clone or download. The Clone with HTTPS dialog box opens.
- In the dialog box, click the clipboard icon to the right of the address to copy the address.
- Paste the URL into your code or CI configuration.
Alternatively, you can copy the HTTP URL from the browser address field and paste it into your code or CI configuration. If the URL does not include an indication of the VCS type, remember to add the type prefix.
Commit identifier
The format of the commit string is dependent on the VCS you use. The easiest way to retrieve the relevant commit identifier is to use the CI integration detailed below.
Smart Reporting will use this identifier as is to retrieve the information relevant for the state of the source code at the time the test was run.
Custom fields for source code
To supply the information units to identify the source code file, use the following Custom Fields:
- perfecto.vcs.repositoryUrl
- perfecto.vcs.commit
- perfecto.vcs.filePath
To access the source code directly from the STR, the script must supply a Repository URL in the perfecto.vcs.repositoryUrl custom field, and a value for one or both of the two other custom fields - perfecto.vcs.commit& perfecto.vcs.filePath.
Setting the values of these Custom Fields can be performed at the following points in the Test Script:
- As part of the PerfectoExecutionContext (or ImportExecutionContext if using the import SDK) instance (using the withCustomFields method) used to create the ReportiumClient instance.
- During the TestStart (using the withCustomFields method).
- In the command line (usually using the CI configuration) using the -DReportiumCustomFields parameter.
CI integration - best practice
If you build and run the test script using a Continuous Integration tool, for example Jenkins or CircleCI, some of the information - in particular the Repository URL and Commit values - can be obtained directly from the CI parameters and supplied to the Custom Fields using the command-line parameters. The information is used by the CI tool to retrieve the source files and build the test scripts before running them and are therefore readily available.
For example in Jenkins, if you work with the Git plugin you will have access to the following environment variables for your build jobs:
- GIT_URL - contains the Repository URL value.
- GIT_COMMIT - contains the commit ID value.
Using these values you can supply the custom field values in the build step of your CI tool by adding the following run-time parameters:
-DReportiumCustomFields=perfecto.vcs.repositoryUrl=${GIT_URL},perfecto.vcs.commit=${GIT_COMMIT}See how to integrate these custom field settings with other CI tools in the CI Integration section.
Supply the file path
The perfecto.vcs.filePath custom field should be set to the file location of the actual test class' source file - relative to the Repository URL.
In many cases, the CI tool cannot supply the source code file path custom field value, because the CI processes the full test job which may include more than a single test script. Therefore, it is best that each test supply the file path during the Test Start (see below) of the specific test.
If the testStart() method is called directly from the class that implements the test script, the code can simply hard-code the file path. For example, in this sample code, the file path for the com.perfectomobile.appTest.MyTest class is “Java/main-sample/src/main/java/com/perfectomobile/appTest/MyTest.java”. For example:
public void myTest() {
reportiumClient.testStart("myTest",
new TestContext.Builder()
.withTestExecutionTags("Sanity", "Nightly")
.withCustomFields(new CustomField("perfecto.vcs.filePath",
"Java/main-sample/src/main/java/com/perfectomobile/appTest/MyTest.java"))
.build());
...
}If, however, if your test scripts utilize a Utility class that actually activates the testStart() method, you may need to either pass in the file path as a parameter, or utilize a utility method that extracts the information from the Stack trace. Similar to the following example taken from our VcsUtils sample:
private static String getCallingClassName() {
StackTraceElement[] stackTrace = Thread.currentThread().getStackTrace();
int index = 1;
String thisClassName = VcsUtils.class.getName();
StackTraceElement stackTraceElement;
do {
index++;
stackTraceElement = stackTrace[index];
} while (stackTraceElement.getClassName().equals(thisClassName));
String callingClassName = stackTraceElement.getClassName();
return callingClassName;
}The file path, dependent on the tree structure of the different VCS tools, may be composed of the following parts:
- A constant string that identifies the structure of the tree until the level of the source files. “Java/main-sample/src/main/java" in the example given above.
- The specific path to the test class: "com/perfectomobile/appTest/MyTest.java” which is tied to the package name of the test class.
Sample code
There are three sample test jobs, that utilize the Version Control System Integration:
- Main sample - shows basic use of the VCS integration
- JUnit sample - shows VCS integration for test run with JUnit
- TestNG sample - shows VCS integration for test run withTestNG
Access source code from STR
Learn how to access the source code, based on the information supplied in the custom fields, directly from the Smart Reporting Single Test Report (STR) here.