Device testing for iOS

With Bitrise’s iOS device testing solution, you can run iOS tests on physical devices without having to set up and register your own devices: you just need to use our dedicated Steps and set the device type(s) on which you want to test your app. There are no limits to using the Step, other than your overall build time limit.

Our device testing solution is based on Firebase Test Lab: it uses real, production devices running in a Google data center to test your app. The devices are flashed with updated APIs and have customizable locale settings. You can find the resulting logs, videos and screenshots on Bitrise.

For iOS apps, Firebase Test Lab runs XCTest tests. Find the list of the available devices here.

Limitations

You might be limited by your overall build time. Also note that a single build can contain only one iOS Device Testing Step, performing one type of test. This is because Bitrise sends the build slug to Firebase Test Lab. Sending the same build slug results in a Build already existserror.

Running device tests for iOS apps

To run device tests with the Firebase Test Lab solution, you will need to add two Steps to your Workflow:

  • Xcode Build for testing for iOS.

  • iOS Device Testing.

Beta status

Please note that currently, both Steps are in beta phase. They are already stable and work well but we’re working on improving them!

The Xcode Build for testing for iOS Step performs the xcodebuild command’s build-for-testing action: it builds the tests defined in your iOS app’s Xcode scheme. The Step exports a .zip file that contains your test directory (by default, it’s Debug-iphoneos) and the xctestrun file.

To use this Step, you will need code signing files for the test app: we recommend using codesigndoc to export the necessary files!

The iOS Device Testing Step takes the path to this .zip file - exported as an Environment Variable - as input to run your tests and export the test results to Bitrise.

  1. Add the Xcode Build for testing for iOS Step to your Workflow.

    Note that it should come AFTER the Certificate and profile installer Step.

    Device testing for iOS
  2. In the Scheme name input, set the Xcode Scheme you want to use. By default, the value of the input is the $BITRISE_SCHEME Environment Variable which is stored when your app is created.

    By default, the scheme defines which configuration - for example, Debug or Release - will be used. In the Configuration name input, you can define the configuration type manually.

  3. Set a destination option for xcodebuild in the Device destination input of the Step.The default value is generic/platform=ios. For the available values, call man xcodebuild on your local machine and check the Destinations section of the printout.

  4. Add the iOS Device Testing Step to the Workflow.The Step has to come after the Xcode Build for testing for iOS Step.

    Device testing for iOS
  5. In the Test devices input field, specify the devices on which you want to test the app.Find the list of the available devices here.

  6. Make sure you have the Deploy to Bitrise.io Step in your Workflow, with version 1.4.1 or newer. With the older versions of the Step, you won’t be able to check your results on the Test Reports page!

  7. Start a build.

If all goes well, you should be able to view your results on the Builds page.

Checking test results

Test Reports display the most important information about all the tests you ran on the Test Summary tab. You will see all the tests you ran, their duration, and their results. You can also click on the tabs of the individual tests to see their details.

To access the result of a particular test:

  1. Open your app’s page on Bitrise.

  2. Click the Builds tab and select the build that ran the tests.

  3. Under the build status, click See more next to the Test reports label.

    Test Reports
  4. On the Test Summary tab, find the test you want to check. Alternatively, on the top bar, click the tab of the test set you want to check out.To filter tests based on their results, open the menu on the top right, which is set to All by default.

    Test Reports
  5. Click Test cases to see the details.

    UI tests are grouped according to device while unit tests are grouped according to test cases.

You will find:

  • The total number of tests you ran, as well as the ratio of successful and failed tests.

  • The duration of each individual test (both unit tests and UI tests).

  • In the case of UI tests, the orientation and the locale of the device.

    Test Reports

You can check, depending on the test type:

  • Individual test cases

  • Performance data

  • Videos

  • Screenshots

  • Test artifacts

  • Logs

As noted above, this depends on test type, too: for unit tests, you won’t see screenshots or videos, of course.