Image injection

Different mobile applications employ the built-in device camera to "look" at the outside world and perform different operations based on the image that the camera records. Examples of this would be a barcode-reader application or a bank check scanning application.

If you want to mimic camera behavior when testing these applications using a Perfecto device, you need to provide the application with an image that mocks the use of the camera. Perfecto enables this with the image injection feature. Image injection takes an image (in either .jpeg, .jpg, .png, or .bmp format) that is stored in the Perfecto repository and presents it to the application as if it was read by the device camera. This allows you to test the application actions for different types of focus, image placement, and so on. Image injection requires that you install the application with sensor instrumentation.

The steps in this article guide you through using image injection with Appium.

Limitations

The image injection feature is subject to the following limitations:

API support limitations:

  • AVCaptureVideoPreviewLayer: Used by the application to display the preview view of the camera. The injected image is displayed only after the camera takes the picture, not during the preview display.
  • AVCapturePhotoCaptureDelegate: Image injection is not currently supported for applications based on this class.
  • AVCaptureMetadataOutput: Used by different bar-code reader applications. When an application uses this API, use Start image inject prior to activating the camera.
  • UIImagePicker: Applications that use this API in iOS 10 only see the injected image after clicking the Use Photo button in the user interface.

Functional limitations:

  • When the application offers different view options for image display, Start image inject should be invoked separately when switching between different screens that use the camera, even if the same image needs to be injected.
  • Image injection is supported only for a single application that is instrumented for the image injection on the device. If an application on the device, other than the currently tested application, is instrumented for injection, uninstall it before testing the current application.
  • Image injection is supported for the back camera of iOS devices. It is not supported for front-camera of devices running iOS prior to iOS 11.

API support limitations:

  • Perfecto supports image injection for Android 4.4 or later.
  • Apps that store the image to a private location or to FileProvider locations are not supported. Perfecto only supports image injection for apps that activate the AndroidMediaStore.ACTION_IMAGE_CAPTURE Intent and store the image only to a public URL.
  • Camera2 support is available for Android 5 and later. It does not include QR code and barcode scanning.
  • CameraX support is available for Android 8 and later. Earlier Android versions may work as well but are not officially supported.

Functional limitations:

  • When testing multiple applications, close the application that started image injection prior to starting image injection for a second application.
  • Image injection is not supported for applications running in multi-processing mode.
  • In Android 5, some applications using image injection may invoke a Perfecto popup that will cause the image injection to fail.

1 | Prepare the image files

When preparing the image files, make sure to limit the image size to 1,600,000 pixels (height x width). For images to be used with check scanning apps, make sure that:

  • The background color of the image is different from the color of the check.
  • The width of the check takes up about 90% of the image width. If necessary, crop the image to fit accordingly.
  • The height of the image is the same height as the check.

Following is a sample check image taken based on these guidelines above.

In addition, the following guidelines may be relevant on some Android devices when creating images to be used with check scanning apps:

  • Keep the width of the check at 1650 pixels.
  • Leave 90 pixels of black background on each side.
  • Leave 150 pixels of black background from top and bottom.
  • Make sure the image is sharp and focused.
  • Make sure to take the picture under good light conditions.

2 | Upload the image files to the repository

When you have good quality image files, upload them to the repository via the Upload Item to Repository API.

3 | Install and configure the application

  1. Install the application with sensor instrumentation. This is required to enable support for image injection. For details, see the Install Application API.
    In particular:
    • When installing the application as part of the driver creation, set the sensorInstrument capability to true.
    • When installing the app after the driver has been created, set the sensorInstrumentation parameter of the Perfecto command mobile:application:install to true.
  2. Configure the app to allow camera access (Settings > <application name>).

4 | Start the app

Typically, the app is started as part of the driver creation. If the tested app needs to be relaunched later, use the driver.launchApp() method. When launching a different app than the one specified in the driver capabilities, use Perfecto's Start Application extension.

5 | Start and stop image injection

To start and stop image injection with Appium testing, use the following Perfecto commands to supply the image's repository key using executeScript:

Consider the following:

  • Start image injection just before entering the app page where the image needs to be used. For example, in a check scanning app, injection should take place just before pressing the button that takes the user to the check scanning screen.
  • Always stop injection when leaving the page where image injection is required.

On many devices, the camera includes two main operations:

  • Preview: The camera screen initializes a preview display and a preview callback (setPreviewDisplay, setPreviewCallback). The preview display is responsible for showing the camera on the screen and the preview callback sends camera frames to the application.
  • Take picture: When you click the designated Take Picture button in the app, the app can take a picture in one of the following ways:
    • By using the takePicture API that is designed specifically for taking pictures
    • By taking the last frame from the preview callback

As the tester, you should be familiar with the flow of the camera operation of the tested application and be prepared to inject the image at the proper junction of the test to cover not only the preview but also the take picture operation and any refresh operation that may be used in the flow.

Code samples

For an Appium code sample that installs a simple camera application, after applying sensor instrumentation, and then injects a .jpg image, see the Community-Samples folder of the PerfectoCode Git repository: https://github.com/PerfectoCode/Community-Samples/tree/master/ImageInjection/JavaSample/MobileSimpleTest

Supported APIs

Image injection is supported for apps using specific Apple/Google APIs. The feature may not work as expected if the app uses SDK functions that are not listed in this section. The same applies for apps that rely on 3rd party frameworks rather than directly on the SDK. We do not officially support any 3rd party SDK because the SDK implementation may change from time to time and utilize unsupported APIs. Furthermore, in some cases, 3rd party SDKs contain logic that relies on the states of phone sensors other than the camera. 

iOS APIs

Function Description

captureStillImageAsynchronouslyFromConnection()

Initiates a still image capture and returns immediately

jpegStillImageNSDataRepresentation()

Returns an NSData representation of a still image and metadata in a JPEG sample buffer

captureOutput()

Notifies the delegate that a new video frame was written

AVMetadataMachineReadableCodeObject

Concrete subclass of AVMetadataObject defining the features of a detected barcode.

Android APIs

For Android.hardware.Camera (API level 1-20):

Function

Description

takePicture()

Triggers an asynchronous image capture

setOneShotPreviewCallback()

setPreviewCallback()

setPreviewCallbackWithBuffer()

setPreviewDisplay(SurfaceHolder)

setPreviewTexture(SurfaceTexture surfaceTexture)

Camera.PreviewCallback()

onPreviewFrame()

APIs for handling the camera preview layer

For Android.hardware.Camera2 (API level >= 21), available for Android 5 and later:

Function

 Description

createCaptureSession()

Create a new camera capture session

addTarget(Surface)

Add a surface to the list of targets for this request

Image.PlanesgetBuffer()

Get a direct ByteBuffer containing the frame data.

Image.getPlanes()

Get the array of pixel planes for an image.

Image.getHeight()

Get the height of the image in pixels.

Image.getWidth()

Get the width of the image in pixels.

close()

Close the capture session

For androidx.camera (API level >= 30), available for Android 8 and later:

Function

Description

bindToLifecycle()

Binds a collection of UseCase to a LifecycleOwner

ImageCapture.OnImageSavedCallback

Listener containing callbacks for image file I/O events

ImageProxy.getHeight()

Returns the image height

ImageProxy.getWidth()

Returns the image width

ImageProxy.getPlanes()

Returns an array of planes

ImageProxy.PlaneProxy.getBuffer()

Returns a pixels buffer

BarcodeScanner.process(InputImage)

(ML Kit method) Detects barcodes from the supplied image