Using key-based caching in your builds

Abstract

Key-based caching requires defining a key that identifies a cache archive. A key can be static or dynamically generated, and you can use templates in your keys. There are dedicated key-based caching Steps for dependency managers that require no configuration.

Key-based caching requires two Steps that must be used together:

Save Cache: saves the build files into a cache archive, identified by a key.
Restore Cache: restores build files into a cache archive, identified by a key.

Dedicated key-based caching Steps

You can use dedicated key-based caching Steps, such as the Save NPM Cache, to cache npm or yarn dependencies.

For a full list of dedicated key-based caching Steps, check out Dedicated caching Steps for dependency managers.

Both Steps access cache archives via key strings. These keys need to be specified as the values of the Cache keys Step input; each key identifies a separate cache archive.

Creating a new cache archive

Retention and eviction policy

Your cache entries are retained for seven days since the last use. If they're not used in your workflows for a continuous period of seven days, they will be automatically removed.

If you run out of your cache storage allowance, the Least Recently Used (LRU) cache entries on your app will be replaced with any new ones produced. LRU is a cache replacement algorithm that removes the least recently used data in order to make room for new data. This might mean that your cache hit rate could reduce if the replaced cache entries are needed by any Workflows. We recommend upgrading to a higher plan to get more storage so that you don't lose any cache entries that might be needed by your Workflows.

Add the Save Cache Step at the end of your Workflow.
In the Cache key input, define a cache key. This key will be used to identify the cache archive.

Key limitations

The maximum length of a cache key is 512 characters (longer keys get truncated). Commas (,) are not allowed in keys.

You can use templates and functions to create dynamic keys that change depending on the build environment or other factors.

You can specify multiple keys; the Step will evaluate them in order and match the first one. Read more: Key matching for cache archives.

Conditional caching with dynamic keys

You can configure the Step in a way that allows it to automatically skip archiving and uploading the cache if its content has not changed during the build: Using conditional caching with dynamic keys.
Example 1. Creating a new cache archive with a key referring to the OS
In this example, we're creating a cache archive with a key that refers to the type of the operating system of the build machine.
```
- save-cache@1:
    inputs:
    - key: |-
        npm-cache-{{ .OS }}
```
In the Paths to cache input, define the files and folders that you want to cache.

The input allows for wildcards: * and **. The input value is evaluated at runtime.

Archive size limit

The size of a single cache archive cannot exceed 15 GB.
Example 2. Caching all files and folders recursively in the node_modules folder
```
- save-cache@1:
    inputs:
    - key: |-
        npm-cache-{{ .OS }}
    - paths: node_modules/
```

Once a cache archive has been created, you can access it on the App Settings page: Accessing key-based cache archives on the website.

Restoring an existing cache archive

Cache archives cannot be shared between builds running on Linux and Mac stacks

A cache archive saved on a Linux-based stack can only be restored in builds also running on Linux machines. Similarly, Mac builds can only restore caches previously saved in a Mac build. The Restore Cache Step fails with a "The specified key does not exist." error if the build tries to restore a cache archive saved on a different OS.

Cache archives can be shared among stacks with different versions of the same OS (for example, between different Xcode stacks).

Add the Restore Cache Step at the start of your Workflow.
In the Cache key input, type the key of the cache that you want to restore.

You can use templates and functions to create dynamic keys that change depending on the build environment or other factors.

You can specify multiple keys; the Step will evaluate them in order and select the first matching one. Read more: Key matching for cache archives.
Example 3. Restoring a cache archive from one of two keys
In this example, we're restoring one of two keys:
- First, we'll look for a cache archive with a key that includes the name of the current Workflow. For example, if the current Workflow's name is primary, the Step will look for an archive with the key npm-cache-primary.
- Second, we'll look for an archive with a key that includes the name of the current branch. For example, if the current branch's name is main, the Step will look for an archive with the key npm-cache-main.
```
- restore-cache@1:
    inputs:
    - key: |-
        npm-cache-{{ .Workflow }}
        npm-cache-{{ .Branch }}
```

Key-based caching templates and functions

Both key-based caching Steps support using template elements in their Step inputs. The Steps evaluate the key template at runtime and the final cache key to be used can change depending on the build environment or on certain files in the repository.

Available caching templates

Table 1. Key-based caching templates

Template expression	Definition	Possible values
`cache-key-{{ .Branch }}`	Current git branch the build runs on.	The exact name of any existing branch of the app.
`cache-key-{{ .CommitHash }}`	SHA-256 hash of the git commit the build runs on.	Any existing commit hash.
`cache-key-{{ .Workflow }}`	Current Bitrise workflow name (for example, `primary`).	The exact name of any existing Workflow of the app.
`{{ .Arch }}-cache-key`	Current CPU architecture of the build stack (`amd64` or `arm64`).	`amd64`: For Intel-based stacks. `arm64`: For M1-based stacks.
`{{ .OS }}-cache-key`	Current operating system of the build stack (`linux` or `darwin`).	`linux`: For Linux-based stacks. `darwin`: For macOS-based stacks.

Using functions in caching templates

The key-based caching templates support the use of two different functions:

checksum: This function computes the SHA-256 checksum of the contents of one or more files. This is useful for creating unique cache keys based on files that describe content to cache. Example 1, “Using the checksum function”.
getenv: This function returns the value of an Environment Variable (Env Var) or an empty string if the variable is not defined. Example 2, “Using the getenv function”

Example 1. Using the checksum function

Use the checksum function to create a key that computes the checksum of the package-lock.json file:

- save-cache@1:
    inputs:
    - key: npm-cache-{{ checksum "package-lock.json" }}
    - paths: node_modules

Use the checksum function to create a key that computes a checksum for any .gradle file and the gradle.properties file:

- save-cache@1:
    inputs:
    - key: gradle-cache-{{ checksum "**/*.gradle*" "gradle.properties" }}
    - paths: AndroidApp

Example 2. Using the getenv function

Use the getenv function to create a key that contains the value of the BITRISE_BUILD_NUMBER Env Var.

- save-cache@1:
    inputs:
    - key: npm-cache-{{ getenv "BITRISE_BUILD_NUMBER" }}
    - paths: node_modules

Key matching for cache archives

It's possible to define more than one key in the Cache keys input of the key-based caching Steps. You can specify additional keys by listing one key per line. The list is in priority order, so the Step will first try to find a match for the first key you provided, and if there is no cache stored for the key, it will move on to find a match for the second key (and so on).

inputs:
  key: |
    key-1
    key-2
    key-3

In addition to listing multiple keys, each key can be a prefix of a saved cache key and still get a matching cache archive. For example, the key my-cache- can match an existing archive saved with the key my-cache-a6a102ff. We recommend configuring the keys in a way that the first key is an exact match to a checksum key, and to use a more generic prefix key as a fallback:

inputs:
  key: |
    npm-cache-{{ checksum "package-lock.json" }}
    npm-cache-

Using conditional caching with dynamic keys

You might not want to update a cache archive every time you run a build. By skipping caching, you can:

Avoid saving potentially incorrect build data in the cache.
Make the PR validation workflow faster.
Limit the amount of data stored and transferred.

If the cached content did not change during your build at all, the Save cache Step can automatically skip compressing and uploading the cache. To make this work, you need to make sure your cache archive has a unique cache key that changes whenever the contents of the cache change: in practice, this requires a dynamic key with a checksum and a file that describes the cached content. The checksum will be calculated based on the contents of the file; if the file remains unchanged, so does the checksum.

We strongly recommend configuring the Step in a way to take advantage of this feature. Follow the steps below, or skip ahead to our YAML example.

Skipping the Save cache Step entirely

If you don't wish to update the cache archive even if the content that would be cached has changed, you can either remove the Save cache Step from your Workflow or you can set conditions for the Step: Enabling or disabling a Step conditionally.

Open your app on Bitrise.
Click the Workflows button on the main page.
On the Workflows & Pipelines pages, you can:
- Click the Edit bitrise.yml button to get to the bitrise.yml tab of the Workflow Editor.
- Select a Workflow from the list of the app's Workflows.
Open your Workflow and select the Save cache Step.
Set the Unique cache key input to true.

If the input is set to false, the Step can still skip uploading the cache but it will need to create the cache archive first to calculate its checksum. This takes time.
In the Cache key input, create a key with a checksum for a file that describes the cached content.

For example, npm-cache-{{ checksum "package-lock.json" }} is a dynamic key that remains unchanged as long as the package-lock.json file does not change.
Add the same dynamic key to the Cache keys input of the Restore cache Step.

If everything is configured correctly, if the keys in the two caching Steps match, the cache archive will not be updated.

Example 1. YAML example for using a dynamic key with checksum

In this example, the cache key calculates the sum for any .gradle file and the gradle.properties file. Both the Restore cache and the Save cache Step looks for the same cache key:

workflows:    
  caching:      
    steps:      
    - restore-cache@1:          
        inputs:          
        - key: gradle-cache-{{ checksum "**/*.gradle*" "gradle.properties" }}      
    - save-cache@1:          
        inputs:          
        - key: gradle-cache-{{ checksum "**/*.gradle*" "gradle.properties" }}          
        - is_key_unique: 'true'

In this section: