Running Xcode tests

Last updated at 2021-07-07

To run Xcode tests on Bitrise, you need to have test targets defined in your projects. You need two Steps to run Xcode tests and view their results:

Code signing files

Running Xcode tests and deploying their results to Bitrise do not require any code signing files. So don’t worry about them just yet!

The Xcode Test for iOS step runs the pre-defined Xcode tests. It has a default configuration that will work if the tests are written correctly. You can find the same configuration options in Xcode, too.

You can also generate code coverage files, and export the test results as a compressed zip file.

Test reports

Your Xcode test results will be exported to Test Reports: check out our Test Reports guide to find out how to make it work for you.

Configuring the Xcode tests on Bitrise

The default input values of the Xcode Test for iOS Step could work if your tests are written correctly. However, we recommend making sure that all the inputs have the value you want so the Step can do what you need!

Exporting the generated result

The Xcode Test for iOS Step generates an .xcresult bundle and exports it to a location set in the Env Var $BITRISE_XCRESULT_PATH.

Be aware that some tools, like xcov, will look for the results bundle in its default location, not at $BITRISE_XCRESULT_PATH. Because of this your build will fail when using those tools. You need to configure them to look for the bundle in the location set by the Env Var. Here’s an example xcov setup:
  scheme: 'ExampleScheme',
  workspace: 'example.xcworkspace',
  xccov_file_direct_path: ENV['BITRISE_XCRESULT_PATH'],
  include_targets: ',ExampleShared.framework,ExampleWidget.appex'

Check the following required inputs of the Xcode Test for iOS Step before running a build. These inputs determine the sort of tests the Step will run.

Headless mode

From Xcode 9 onwards, tests are run in headless mode by default: this means that the simulator will run in the background only. To change it, go to the Step’s Debug input group and set the Run the test in headless mode? input’s value to no. However, with this option, tests will take more time.

Using xcpretty

The xcpretty output tool does not support parallel tests. If parallel tests are enabled in your project, go to the Step’s Debug input group and set the Output tool input’s value to xcodebuild.

Rerunning failed tests (Not available in Xcode 13+)

The Should retry test on failure? (Not available in Xcode 13+) input in the Debug section of the Xcode Test for iOS Step allows you to automatically rerun ALL your tests, not just the failed ones. If you set this input to yes, the Step will run xcodebuild one more time in the case of test failure.

From Xcode 13 and above the Should retry test on failure? (Not available in Xcode 13+) feature is not available anymore. We recommend you use Test Repetitions Mode (Available in Xcode 13+) input with the retry_on_failure option selected. This allows you to rerun only the failed test/s instead of running all your tests. You can find this test repetition feature with our Xcode Test for iOS Step from version 3.0.0 and above.

Test Repetitions

Xcode’s test repetition modes are available with the Xcode Test for iOS Step on stacks running Xcode 13 and above on Bitrise. With test repetitions, you can run any type of tests multiple times in various ways such as retry on failure, until failure and until max repetitions.

The Xcode Test for iOS Step’s Test Repetitions Mode (Available in Xcode 13+) input offers the following options:

Please note that these options are only available from Xcode Test for iOS version 3.0.0 and above.

With the Maximum Test Repetitions (Available in Xcode 13+) input you can specify the maximum number of test repetitions. Please note that you have to add a greater number than one if the Test Repetition Mode (Available in Xcode 13+) input is set to other than none.

Enable the Relaunch Tests for Each Repetition (Available in Xcode 13+) input to launch tests in a completely new process for each repetition.