Key-based caching

Abstract

Key-based caching on Bitrise works by associating cache archives with a key. A Workflow can restore a cache archive by referring to the key; the build files can be saved into the cache archive that the key indicates.

Feature in beta

Key-based caching is in the beta phase, and as such, it is subject to changes in terms of both function and availability.

Key-based caching works by associating cache archives with a key. A Workflow can restore a cache archive by referring to the key; at the end of the Workflow, the build files can be saved into the cache archive that the key indicates. This overwrites the cache archive.

Cache keys can contain variables and other dynamic elements that are replaced at runtime. Users can set up caching that periodically creates new cache archives and automatically discard the old archives.

A new cache archive is valid for seven days after creation. If the archive is updated, the timer restarts. If an archive is not updated for seven days, it is automatically discarded.

Using key-based caching

Key-based caching requires two Steps that must be used together:

Save Cache: saves the build files into a cache archive, identified by a key.
Restore Cache: restores build files into a cache archive, identified by a key.

Both Steps access cache archives via key strings. These keys need to be specified as the values of the Cache keys Step input; each key identifies a separate cache archive.

Creating a new cache archive

Key limitations

The maximum length of a cache key is 512 characters (longer keys get truncated). Commas (,) are not allowed in keys.

Add the Save Cache Step at the end of your Workflow.
In the Cache key input, define a cache key. This key will be used to identify the cache archive.

You can use templates and functions to create dynamic keys that change depending on the build environment or other factors.

You can specify multiple keys; the Step will evaluate them in order and match the first one. Read more: Key matching for cache archives.

Conditional caching with dynamic keys

You can configure the Step in a way that allows it to automatically skip archiving and uploading the cache if its content has not changed during the build: Using conditional caching with dynamic keys.
Example 1. Creating a new cache archive with a key referring to the OS
In this example, we're creating a cache archive with a key that refers to the type of the operating system of the build machine.
```
- [email protected]:
    inputs:
    - key: |-
        npm-cache-{{ .OS }}
```
In the Paths to cache input, define the files and folders that you want to cache.

The input allows for wildcards: * and **. The input value is evaluated at runtime.
Example 2. Caching all files and folders recursively in the node_modules folder
```
- [email protected]:
    inputs:
    - key: |-
        npm-cache-{{ .OS }}
    - paths: node_modules/
```
Dedicated key-based caching Steps

You can use dedicated key-based caching Steps, such as the Save NPM Cache, to cache npm or yarn dependencies.

For a full list of dedicated key-based caching Steps, check out Dedicated caching Steps for dependency managers.

Once a cache archive has been created, you can access it on the App Settings page: Accessing key-based cache archives.

Restoring an existing cache archive

Add the Restore Cache Step at the start of your Workflow.
In the Cache key input, type the key of the cache that you want to restore.

You can use templates and functions to create dynamic keys that change depending on the build environment or other factors.

You can specify multiple keys; the Step will evaluate them in order and select the first matching one. Read more: Key matching for cache archives.
Example 3. Restoring a cache archive from one of two keys
In this example, we're restoring one of two keys:
- First, we'll look for a cache archive with a key that includes the name of the current Workflow. For example, if the current Workflow's name is primary, the Step will look for an archive with the key npm-cache-primary.
- Second, we'll look for an archive with a key that includes the name of the current branch. For example, if the current branch's name is main, the Step will look for an archive with the key npm-cache-main.
```
- [email protected]:
    inputs:
    - key: |-
        npm-cache-{{ .Workflow }}
        npm-cache-{{ .Branch }}
```

Key-based caching templates and functions

Both key-based caching Steps support using template elements in their Step inputs. The Steps evaluate the key template at runtime and the final cache key to be used can change depending on the build environment or on certain files in the repository.

Available caching templates

Table 1. Key-based caching templates

Template expression	Definition	Possible values
`cache-key-{{ .Branch }}`	Current git branch the build runs on.	The exact name of any existing branch of the app.
`cache-key-{{ .CommitHash }}`	SHA-256 hash of the git commit the build runs on.	Any existing commit hash.
`cache-key-{{ .Workflow }}`	Current Bitrise workflow name (for example, `primary`).	The exact name of any existing Workflow of the app.
`{{ .Arch }}-cache-key`	Current CPU architecture of the build stack (`amd64` or `arm64`).	`amd64`: For Intel-based stacks. `arm64`: For M1-based stacks.
`{{ .OS }}-cache-key`	Current operating system of the build stack (`linux` or `darwin`).	`linux`: For Linux-based stacks. `darwin`: For macOS-based stacks.

Using functions in caching templates

The key-based caching templates support the use of two different functions:

checksum: This function computes the SHA-256 checksum of the contents of one or more files. This is useful for creating unique cache keys based on files that describe content to cache. Example 1, “Using the checksum function”.
getenv: This function returns the value of an Environment Variable (Env Var) or an empty string if the variable is not defined. Example 2, “Using the getenv function”

Example 1. Using the checksum function

Use the checksum function to create a key that computes the checksum of the package-lock.json file:

- [email protected]:
    inputs:
    - key: npm-cache-{{ checksum "package-lock.json" }}
    - paths: node_modules

Use the checksum function to create a key that computes a checksum for any .gradle file and the gradle.properties file:

- [email protected]:
    inputs:
    - key: gradle-cache-{{ checksum "**/*.gradle*" "gradle.properties" }}
    - paths: AndroidApp

Example 2. Using the getenv function

Use the getenv function to create a key that contains the value of the BITRISE_BUILD_NUMBER Env Var.

- [email protected]:
    inputs:
    - key: npm-cache-{{ getenv "BITRISE_BUILD_NUMBER" }}
    - paths: node_modules

Key matching for cache archives

It's possible to define more than one key in the Cache keys input of the key-based caching Steps. You can specify additional keys by listing one key per line. The list is in priority order, so the Step will first try to find a match for the first key you provided, and if there is no cache stored for the key, it will move on to find a match for the second key (and so on).

inputs:
  key: |
    key-1
    key-2
    key-3

In addition to listing multiple keys, each key can be a prefix of a saved cache key and still get a matching cache archive. For example, the key my-cache- can match an existing archive saved with the key my-cache-a6a102ff. We recommend configuring the keys in a way that the first key is an exact match to a checksum key, and to use a more generic prefix key as a fallback:

inputs:
  key: |
    npm-cache-{{ checksum "package-lock.json" }}
    npm-cache-

Using conditional caching with dynamic keys

You might not want to update a cache archive every time you run a build. By skipping caching, you can:

Avoid saving potentially incorrect build data in the cache.
Make the PR validation workflow faster.
Limit the amount of data stored and transferred.

If the cached content did not change during your build at all, the Save cache Step can automatically skip compressing and uploading the cache. To make this work, you need to make sure your cache archive has a unique cache key that changes whenever the contents of the cache change: in practice, this requires a dynamic key with a checksum and a file that describes the cached content. The checksum will be calculated based on the contents of the file; if the file remains unchanged, so does the checksum.

We strongly recommend configuring the Step in a way to take advantage of this feature. Follow the steps below, or skip ahead to our YAML example.

Skipping the Save cache Step entirely

If you don't wish to update the cache archive even if the content that would be cached has changed, you can either remove the Save cache Step from your Workflow or you can set conditions for the Step: Enabling or disabling a Step conditionally.

Open your app on Bitrise.
Click the Workflows button on the main page.
Open your Workflow and select the Save cache Step.
Set the Unique cache key input to true.

If the input is set to false, the Step can still skip uploading the cache but it will need to create the cache archive first to calculate its checksum. This takes time.
In the Cache key input, create a key with a checksum for a file that describes the cached content.

For example, npm-cache-{{ checksum "package-lock.json" }} is a dynamic key that remains unchanged as long as the package-lock.json file does not change.
Add the same dynamic key to the Cache keys input of the Restore cache Step.

If everything is configured correctly, if the keys in the two caching Steps match, the cache archive will not be updated.

Example 1. YAML example for using a dynamic key with checksum

In this example, the cache key calculates the sum for any .gradle file and the gradle.properties file. Both the Restore cache and the Save cache Step looks for the same cache key:

workflows:    
  caching:      
    steps:      
    - [email protected]:          
        inputs:          
        - key: gradle-cache-{{ checksum "**/*.gradle*" "gradle.properties" }}      
    - [email protected]:          
        inputs:          
        - key: gradle-cache-{{ checksum "**/*.gradle*" "gradle.properties" }}          
        - is_key_unique: 'true'

Accessing key-based cache archives

You can access your cache archives separately, without running a build. This allows you to keep track of several things:

What cache keys are associated with different cache archives.
How much storage space is taken up by your cache archives.
When each cache archive is due to expire.

In addition, you can download any cache archive - if it hasn't expired yet -, you can copy the cache keys, and you can delete archives.

To access your archives:

Admin access only

Only users with the role of admin on the app's team can access all functions of the App Settings page of an app.

Open your app on Bitrise.
Click on App Settings.
On the left side, select Builds.
Scroll down to the Manage build caches section.
Select the Key-based caching tab.
You can:
- Copy the cache key by clicking the Copy key icon.
- Downloading the cache archive by clicking the Download icon.
- Deleting the cache archive by clicking Delete icon.

Dedicated caching Steps for dependency managers

Key-based caching is powerful and flexible: you can use the Save cache and Restore cache Steps with any app, regardless of platform or the type of data you want to cache. However, these Steps require careful configuration and it is possible to make mistakes. If you are looking for a simple solution to cache dependencies of the most frequently used dependency managers, we offer dedicated Steps that require no configuration whatsoever.

The dedicated caching Steps store their archives in the same storage space as the Save cache and Restore cache Steps but unlike those two, they automatically set up the appropriate cache paths and cache keys.

The following dedicated Steps are available:

Save Cocoapods Cache and Restore Cocoapods Cache
Save SPM Cache and Restore SPM Cache
Save NPM Cache and Restore NPM Cache
Save Gradle Cache and Restore Gradle Cache

To use these Steps, simply add the Restore version to the start of your Workflow and the Save version to the end of your Workflow. No other configuration is needed.

In this section: