Test stats in Build Insights

Insights Pro only

Test stats in Build Insights are only available with Insights Pro. Request Pro access now!

Build Insights integrates with Bitrise Test reporting to provide historic performance tracking across all your tests. Test stats enables you to locate poorly performing tests and stability problems. As well as this, track the number of test you have in order to measure improvement and ensure too many skipped or failing test don't slip through.

Setting up access to test stats

To be able to view your test stats in Build Insights, you need at least two Steps in your Workflow:

  • A testing Step that runs your test cases.

  • The Deploy to Bitrise.io Step.

We have five testing Steps that automatically export their results for the Deploy to Bitrise.io Step: if you use any of these Steps and have access to Build Insights, you don't need to configure anything to access your test stats. The Steps are:

  • Xcode Test for iOS

  • Android Unit Test

  • iOS Device Testing

  • Virtual Device Testing for Android

  • Flutter Test

If you use any other Step, you just need to make sure that your test results end up in the $BITRISE_TEST_RESULT_DIR, which is an automatically exposed Environment Variable pointing to the correct path.

Accessing the test stats dashboard

To get to the test stats dashboard:

  1. Log in to your Bitrise account.

  2. In the top right corner, open the account selector dropdown menu and select a Workspace.

  3. On the header, click Insights. This takes you to the Insights dashboard of the selected Workspace.

  4. Click the TEST STATS tab.

Filtering test stats

In the top right of the dashboard, you can filter for time range and test case properties.

test-stats-filter.png

Filtering for test case properties means being able to filter for:

  • Status: filter for Passed, Failed, Skipped, Inconclusive test stats.

  • Branch: filter test stats for a specific set of branches.

  • Module: filter test stats for a particular module or package.

  • Test Suite: filter test stats for a particular set test suite.

Test counts and test summary

On the test stats dashboard, you can see the number of tests over the previously selected time range, broken down by the status of the tests. The possible statuses are:

  • Passed

  • Failed

  • Skipped

  • Error

test-counts.png

You can use the graph to track the performance of your test cases over time, making it easy to spot any changes in test coverage.

Below the test count graph, you can find the test summary widget which shows you each individual test case during the previously selected time range. You can sort the test cases to locate troublesome tests.

test-summary.png

We support sorting via:

  • Total run time: how much time and therefore credits spent executing each test case.

  • Runs: the total number of executions for each test case.

  • Re-runs: the number of times each test case has required a re-run to pass. It is an indicator of test flakiness.

  • Success rate: the percentage of completed test runs that passed. It doesn't include aborted test runs.

  • Typical duration: the 50th percentile (median) run time for each test case.

  • Slowest duration: the 95th percentile run time for each test case.

For each test case, we also show a recent history chart to indicate the recent performance. Clicking a test case takes you to the Test case history page.

Test case history

Build Insights tracks the metrics of each test case over time, allowing you to analyze the historical performance of each test case.

On the top of the test case history page, you can see the aggregated statistics for that particular test case. These are the same stats that are displayed for each test case on the test summary widget on the test stats dashboard.

test_case_stats.png

To get into test case performance in more detail, you can look at:

  • Duration trend: shows the historical performance for the test case. The chart shows the 50th percentile (median) and 95th percentile performance, allowing you to see if the performance of the test is getting better or worse over time and how the consistency varies.

    duration_trend.png
  • Success rate trend: success rate trends show the percentage of successful runs of the test case over time.

    success_rate_trend.png
  • Latest tests: this timeline shows the history of the recent runs of this particular test case, and detailed information about each run.

    run_timeline.png

On the Latest tests chart, hover over any run to see its details.

  • Status: the status of the test run, indicated by the color.

  • Repetition: the number of re-runs the test case required.

  • Duration: how long the test run took.

  • Test Suite: the parent suite of the test case.

  • Branch: the git branch containing the code that the test ran against.

  • Build number: the Bitrise build number this test ran on.

Click any of the test runs to get to the Bitrise build log of the build that ran the test.