Skip to main content

Running Bitrise builds on your AWS EC2 instance

Bitrise is available as an Amazon Machine Image (AMI) in the AWS Marketplace. With the Bitrise AMI, you can run Bitrise builds using your own AWS resources instead of our own build infrastructure.

To do so, you need to:

  1. Configure agent pools on Bitrise: Set up an agent pool for your builds and get the token required to connect your Bitrise Workspace to your AWS EC2 instance.

  2. Subscribe to the Bitrise AMI and launch an AWS EC2 instance. You can launch either a Mac or a Linux instance. For a Mac instance, you'll need to allocate a dedicated host before launching the instance.

  3. Use the token to connect the instance to your Bitrise Workspace. This will allow the Bitrise agent to run your builds on your EC2 instances.

Configuring agent pools

Important

Please note that for now, you won't find the Bitrise agents option among your Workspace settings if our team hasn't specifically enabled it for you. If you are interested in running Bitrise builds on your own infrastructure, contact us!

Before you can run the Bitrise agent on your own hardware - either on your own premises or on an AWS instance -, you need to reserve an agent pool and authenticate with a token.

To do so:

  1. Log in to Bitrise and open the account selector dropdown menu in the top right.

  2. Find your Workspace and click the little gear icon next to its name to get to your Workspace settings page.

    workspace-settings.png

  3. On the left, select Bitrise agents.

  4. On the Bitrise agent pools page, click Add new agent pool.

    agent-pools-overview.png

  5. Type a name into the Name field then click Next.

  6. You will receive a token. Copy this token: you will need it for authentication.

    agent-token.png

  7. Open the app you need on Bitrise and click Workflows.

  8. Click Edit bitrise.yml to go the Workflow Editor, and go to the Stacks & Machines tab.

  9. Find your own stack in the stack selection dropdown menus: it will be called Self-hosted agents: <agent pool name>.

    For example, if you named your pool my-pool, the stack will be called Self-hosted agents: my-pool.

  10. Click Save in the top right corner.

Now your builds will run using the reserved agent pools. You can run builds either on your own hardware or on AWS EC2 instances.

Launching an EC2 instance for the Bitrise AMI

We offer a dedicated Bitrise Amazon Machine Image (AMI) to run the Bitrise agent on your own Amazon EC2 instance.

You can set up your AWS EC2 instance either on the AWS UI or on the CLI.

Launching an instance on the AWS UI

Launch an instance via the AWS UI:

Mac instance

Linux instance

  1. Subscribe to the Bitrise AMI on AWS: AMI subscriptions.

  2. Create a dedicated host on AWS: on the EC2 Dashboard, find the Instances menu, select Dedicated Hosts, and click Allocate dedicated host.

  3. Fill out the required fields:

    • Name tag: This will be the name of your instance.

    • Instance family: mac2

    • Instance type: mac2.metal

    • Availability Zone: It depends on the subnets you plant to use. AWS will tell you if there is no node capacity in your chosen zone.

    • Quantity: The number of hosts you want to create.

  4. Launch an instance of the dedicated host: open the Actions menu and select Launch instance(s) onto host.

  5. Fill out the required fields on the Launch an instance page:

    Name and tags section:

    • Name: The name of your instance.

    Application and OS Images section:

    • Amazon Machine Image (AMI): Select the Bitrise AMI you subscribed to.

    Instance type section:

    • Instance type: Select mac2.metal.

    Key pair (login) section:

    • Key pair name - required:

    Network settings section - click Edit to modify the settings:

    • VPC - required: Select a virtual private cloud from the dropdown menu.

    • Subnet: Select your preferred subnet from the dropdown menu, or leave it on No preference.

    • Firewall (security groups): Create a new security group or select an existing one.

    Configure storage section:

    • Set the size of the storage volume in gigabytes.

    • Select a root volume type from the dropdown menu.

    Advanced details section:

    • Tenancy: Select the Dedicated host - launch this instance on a dedicated Host option from the dropdown menu.

    • Target host: Select the Host ID option from the dropdown menu.

    • Tenancy host ID: Select the host ID of the previously created dedicated host

  6. Click Launch instance.

    The launch can take up to 10-15 minutes. You will see system status checks and instance status checks: once the instance is ready, both checks will be green.

    aws-status-check.png

  1. Subscribe to the Bitrise AMI on AWS: AMI subscriptions.

  2. Launch an instance via the instance launch wizard.

  3. Fill out the required fields on the Launch an instance page:

    Name and tags section:

    • Name: The name of your instance.

    Application and OS Images section:

    • Amazon Machine Image (AMI): Select the Bitrise AMI you subscribed to.

    Instance type section:

    • Instance type: Select an x86_64 Linux metal instance.

    Key pair (login) section:

    • Key pair name - required:

    Network settings section - click Edit to modify the settings:

    • VPC - required: Select a virtual private cloud from the dropdown menu.

    • Subnet: Select your preferred subnet from the dropdown menu, or leave it on No preference.

    • Firewall (security groups): Create a new security group or select an existing one.

    Configure storage section:

    • Set the size of the storage volume in gigabytes.

    • Select a root volume type from the dropdown menu.

  4. Click Launch instance.

    The launch can take up to 10-15 minutes. You will see system status checks and instance status checks: once the instance is ready, both checks will be green.

    aws-status-check.png

Launching an instance on the AWS CLI

Mac instance

Linux instance

  1. Configure the AWS CLI.

  2. Subscribe to the Bitrise AMI in the Marketplace and get its Marketplace ID: Find AMI IDs.

  3. Reserve a dedicated host on AWS EC2:

    aws ec2 allocate-hosts \
  --instance-type mac2.metal \
  --quantity 1 \
  --availability-zone <YOUR_AVAILABILITY_ZONE> \
  --tag-specifications 'ResourceType=dedicated-host,Tags={Key=Name,Value=Example}'

    The instance type must be mac2.metal and make sure you choose the best availability zone for you.

    The output will look something like this:

    {
    "HostIds": [
        "h-07879acf49EXAMPLE"
    ]
}

    Save the host ID as you will need it later.

  4. Launch an EC2 mac2 instance from a Bitrise macOS AMI:

    aws ec2 run-instances \
    --image-id <BITRISE_MARKETPLACE_AMI_ID> \
    --instance-type mac2.metal \
    --subnet-id <YOUR_SUBNET_ID> \
    --security-group-ids <YOUR_SECURITY_GROUP_ID> \
    --key-name <YOUR_KEYPAIR_NAME> \
    --placement 'AvailabilityZone=<YOUR_PREFERRED_AZ>,HostId=<DEDICATED_HOST_ID>,Tenancy=host'

  1. Configure the AWS CLI.

  2. Subscribe to the Bitrise AMI in the Marketplace and get its Marketplace ID: Find AMI IDs.

  3. Launch an EC2 Linux instance from a Bitrise Linux AMI:

    aws ec2 run-instances \
    --image-id <BITRISE_MARKETPLACE_AMI_ID> \
    --instance-type <YOUR_X86_64_METAL_INSTANCE_TYPE> \
    --subnet-id <YOUR_SUBNET_ID> \
    --security-group-ids <YOUR_SECURITY_GROUP_ID> \
    --key-name <YOUR_KEYPAIR_NAME> \

Connectivity and security of your EC2 instance

Once your EC2 instance is ready, make sure you can connect to the instance and that it can connect to the relevant Bitrise services. You can connect to the instance using SSH or in the case of Mac instances, VNC.

You can also set a password for your instance.

Instance passwords

By default, the Bitrise AMI doesn't have user passwords. It is the subscriber's responsibility to set the desired password.

For a Mac instance, you need to set a user password and the default login.keychain password:

sudo /usr/bin/dscl . -passwd /Users/ec2-user
security set-keychain-password  -o "" -p "<new password>" ~/Library/Keychains/login.keychain-db

For a Linux instance:

sudo passwd

Connecting to the instance

To connect to your instance using SSH, we recommend using TCP port 22. To connect to the instance via SSH:

 ssh -i ~/.ssh/key   ubuntu@<your-aws-instance>

If you have a Mac instance, you can also connect with VNC. We recommend using TCP port 5900. To connect:

open vnc://ec2-user@<aws-mac2-instance>

Security group configuration

Make sure to enable the use of both ports in your security group configuration of the instance!

Connecting to Bitrise services

The instance must be able to access the following endpoints to be able to run Bitrise builds:

  • https://den.services.bitrise.io

  • https://build-log.services.bitrise.io

Pre-warming the disk after booting

You can improve EBS performance by pre-warming the disk right after booting it up. You only need to do it once.

Mac instance

Linux instance

  • export cnt=$(($(df -h | grep "/$" | awk '{print $4}' | grep -oE "[0-9]+")-2))
dd if=/dev/random of=bigfile bs=1g count=$cnt
  • sudo dd if=/dev/xvdf of=/dev/null bs=1M

    dev/xvfd

    Be aware that xvdf might be different on your machine

Increasing disk size on a Mac instance

If you have configured a bigger EBS storage for your EC2 Mac instance than the default (400 GB), resize the partition so macOS can use all available disk space:

PDISK=$(diskutil list physical external | head -n1 | cut -d" " -f1)
APFSCONT=$(diskutil list physical external | grep "Apple_APFS" | tr -s " " | cut -d" " -f8)
yes | sudo diskutil repairDisk $PDISK

sudo diskutil apfs resizeContainer $APFSCONT 0

For more information, please refer to the AWS macOS EC2 documentation.

Connecting your AWS instance to your Bitrise Workspace

After you successfully launch your instance with the Bitrise AMI on AWS, you need to connect the Bitrise agent pool you added on bitrise.io to the instance, using the token you received during the process. This allows you to run builds on Bitrise using your AWS EC2 instance as your build stack.

You have two ways of configuring the connection:

  • Using the AWS Secret Manager. We recommend this method.

  • Connecting to the instance and using the token directly in the command.

Using the AWS Secret Manager

  1. Make sure that your instance can access the following endpoints:

    • https://den.services.bitrise.io

    • https://build-log.services.bitrise.io

    Without accessing these endpoints, you won't be able to run builds even after connecting the instance.

  2. Get the token from the process of adding the agent pool on Bitrise.

  3. Create an AWS Manager Secret and store the token in the secret.

  4. Create an IAM role with permission to read the secret and attach it to your EC2 instance.

  5. Modify the User data of the instance: add the command to launch the Bitrise agent, using the Secret you created in the AWS Secret Manager:

    mac2 instance

    Linux instance

    • TOKEN=$(aws secretsmanager get-secret-value --secret-id MY_SECRET | jq  -r '.SecretString')

sudo sed -i '' “s/BITRISE_AGENT_TOKEN/$TOKEN/” ~/Library/LaunchDaemons/io.bitrise.self-hosted-agent.plist
    • TOKEN=$(aws secretsmanager get-secret-value --secret-id MY_SECRET | jq  -r '.SecretString')

sudo sed -i “s/BITRISE_AGENT_TOKEN/$TOKEN/” /etc/systemd/system/bitrise-den-agent.service
sudo systemctl start bitrise-den-agent.service

Connecting to the instance directly

mac2 instance

Linux instance

  1. Make sure that your instance can access the following endpoints:

    • https://den.services.bitrise.io

    • https://build-log.services.bitrise.io

    Without accessing these endpoints, you won't be able to run builds even after connecting the instance.

  2. Get the token from the process of adding the agent pool on Bitrise.

  3. Connect to your instance on AWS and run the following commands on it:

    sudo sed -i '' 's/BITRISE_AGENT_TOKEN/<YOUR_AGENT_POOL_TOKEN>/' ~/Library/LaunchDaemons/io.bitrise.self-hosted-agent.plist
sudo launchctl load -w /Users/ec2-user/Library/LaunchDaemons/io.bitrise.self-hosted-agent.plist

  4. Validate that the agent pool is running on the instance:

    ps aux | grep bitrise-den-agent

  1. Connect to your instance on AWS and run the following commands on it:

    sudo sed -i 's/BITRISE_AGENT_TOKEN/<YOUR_AGENT_POOL_TOKEN>/' /etc/systemd/system/bitrise-den-agent.service
sudo systemctl start bitrise-den-agent.service

  2. Validate that the agent pool is running on the instance:

    ps aux | grep bitrise-den-agent

or

sudo systemctl status bitrise-den-agent.service

Preinstalled tooling for the Bitrise AMI

The Bitrise AMI comes with a set of pre-installed tools, just like the Bitrise stacks on our Cloud offerings. We aim to provide identical tooling for both but this isn't always possible. For the full list of available tools on any given stack, check out the system reports: Bitrise stack reports.

In this section: