SUT performance testing

This article provides comprehensive information on Single User Performance Testing with Perfecto.

Prerequisites

Before implementing SUT performance testing, make sure that:

  • MITM is enabled in the cloud and the certificate installed on the device.
  • You use the unpinned version of binaries (IPA and\or APK ) to capture the network traffic. Mobile applications will not allow an MITIM proxy to intercept network traffic due to HTTPS requests. Ask your developer to provide what you need.

Implementation

Client-side performance testing is also called Single User Performance Testing (SUT) because in this case, no matter what the state of the servers is, it evaluates client-side KPIs like response time and device vitals.

SUT needs to take into account all of the factors that could impact the end-user experience across all tiers, including last-mile networks and client devices: network changes and signal degradation, changes in location, usage of onboard resources and sensors (that consume CPU, memory, and so on), background applications competing for device resources, and more. The following image shows external factors influencing application performance. So, simple functional testing is not enough.

All of these variables should be considered in test scenarios, and they should be automated because the time to test is limited. Service level agreements (SLA) for acceptable user experience should be set (for example, time to log in, time to accomplish a task) and measured in all scenarios.

The following example evaluates Etihad web application performance metrics.

The following transactions are under test: 

  • Launch App:The time it takes to load the app
  • Search Flights: The time it takes to list the available flights on the given itinerary

The Perfecto predefined function helps to simulate end-user conditions and offers the following abilities to extend coverage to real user conditions when testing user journeys or flows. The test includes the following sections:

  1. End-user conditions that simulate network, location, background applications, and other properties

    Copy 
               //Set End User conditions
                //Throttle Network and Capture Traffic - In this example, HAR captured only for Android since Etihad IOS app expects trusted certificate
                if(platformName.equalsIgnoreCase("Android")) {
                    Map<String, Object> pars = new HashMap<>();
                    pars.put("profile", "4g_lte_good");
                    pars.put("generateHarFile", "true");
                    driver.executeScript("mobile:vnetwork:start", pars); 
                }
                else {
                    Map<String, Object> pars = new HashMap<>();
                    pars.put("profile", "4g_lte_good");
                    driver.executeScript("mobile:vnetwork:start", pars); 
                }
                
                //Set location
                Map<String, Object> params = new HashMap<>();
                params.put("address", "Bangalore, India");
                driver.executeScript("mobile:location:set", params); 
                
                //Run background application
                String backGroundApps = "YouTube,Messages,Maps,Calculator,Calendar";
                 String[] bApps = backGroundApps.split(",");
                 for(String i: bApps) {
                    try {
                        Map<String, Object> params1 = new HashMap<>();
                        params1.put("name", i);
                        driver.executeScript("mobile:application:open", params1);
                    } catch (Exception e) {}        
                 }

  2. Points of interest that identify crucial application flows, such as AppLaunch and Search Flight

    Copy 
    reportTimer(driver, AppLaunchTime, 10000, "Checkpoint load time of App launch.", "AppLaunchTime");

  3. Timer reporting ability to track timings and performance of the application
    An example might be how long it took for a page to load or a process to happen, and what that actually looks like for a user (so the UX or user experience on the device glass). In this test, we have chosen app launch time and flight search transactions.

    Perfecto provides page load time as User Experience Time (UX) with the help of visual functions (OCR and image analysis). In other words, the timer will wait for a specific text or image to actually appear (be rendered) on the screen. In our example, upon launching, the Etihad web timer will wait for the text  One Way to appear on the screen.

    The Perfecto page load time includes:

    Page Load Time = UX rendering time + (API process time + network time)

    For details, see Network traffic capture and analysis (HAR) | Performance testing.

    The reference text or image should be the last component on the page that was loaded.

    Copy 
    //Launch Web application
    driver.get("https://www.etihad.com/en-us/book");
       ocrTextValidation(driver, "one Way");
       // Wind Tunnel: Measure UX timer 1 - Able to retrieve UX Timer value
       long AppLaunchTime = timerGet(driver, "ux");
       System.out.println("'Measured UX time for App launch time is: " + AppLaunchTime);

    private static void ocrTextValidation (RemoteWebDriver driver, String content) {    
       // verify that the correct page is displayed as result of signing in.
       Map<String, Object> params1 = new HashMap<>();
       // Check for the text that indicates that the sign in was successful
       params1.put("content", content);
       // allow up-to 30 seconds for the page to display
       params1.put("timeout", "30");
       // Wind Tunnel: Adding accurate timers to text checkpoint command
       params1.put("measurement", "accurate");
       params1.put("source", "camera");
       params1.put("analysis", "automatic");
       params1.put("threshold", "90");
       String resultString = (String) driver.executeScript("mobile:checkpoint:text", params1);
    }
  4. A detailed report about the result of the script that is easy to understand and provides the following details:

    • Functional Validation: The feature is functional with detailed steps and a recorded video

    • Performance validation, which includes:

      • Page Load Time: Measurement of responsiveness of the application’s rendered page load time retrieved by visual analysis

      • Network Traffic: Report including a detailed (HAR) network traffic log. For details, see

      • Device Vitals: Recorded device vitals to provide visibility into network, CPU, and memory consumption during the execution of the test script

A sample Java project is available here.