Run a remote Docker builder with a warm BuildKit cache

How it works#

• buildkitd runs on my-builder as a systemd service, listening only on a Unix socket, /run/buildkit/buildkitd.sock, that members of the buildkit group can use.
• Your docker CLI has a builder that uses Buildx's remote driver with an ssh://my-builder.exe.xyz endpoint. For each connection it runs ssh my-builder.exe.xyz buildctl dial-stdio, which joins SSH to the socket. BuildKit's gRPC travels inside your SSH session.
• The cache lives in /var/lib/buildkit and belongs to the VM, not to any one client. BuildKit sends only the files your Dockerfile needs from your checkout, deduplicates identical steps across concurrent builds, and prunes old cache by the limits in /etc/buildkit/buildkitd.toml.

Nobody gets Docker on the VM through this: BuildKit builds, and that's all it does.

What it buys you#

We built Gitea v28.3.0 with its own Dockerfile: a pnpm frontend, a Go backend with RUN --mount=type=cache for the Go build cache, and an Alpine runtime image of 72 MB. The client was a fresh 2 vCPU / 4 GiB exe.dev sandbox, about the size of a hosted CI runner, with Ubuntu's Docker 29 and Buildx 0.30. The builder was an 8 vCPU / 16 GiB VM with a 100 GB disk, running BuildKit v0.34.2. Times are wall-clock, as the client saw them:

The first row is the builder's raw speed: more cores, same work. The fourth row is what changes CI: a fresh runner has nothing cached, so every job is a first build, while on the builder it is a rebuild of what changed. The 14 to 28 seconds are uploading the 44 MB build context (3 seconds), rebuilding the frontend, and compiling the changed Go packages from the warm cache mount. Each fresh checkout's .git differs, which is why even an already-built commit recompiles the backend in Gitea's Dockerfile.

Create the VM#

Size it#

Builds are bursty: a builder uses all its cores for a minute and then nothing for an hour. On exe.dev a VM's CPU and memory come from your plan's capacity, which your VMs share, so a big builder costs nothing extra while idle. Give it most of your plan, and disk for the cache. On your computer:

ssh exe.dev new --name=my-builder --cpu=8 --memory=16GB --disk=100GB

VM names are global, so pick any free name and use it wherever this page says my-builder. To change the size later, even with a cache in it:

ssh exe.dev resize my-builder --cpu=16 --memory=32GB --disk=200GB

Disks only grow, and are billed by what's used, not by their size. The GC limits below are percentages of the disk, so they grow with it. See your plan's capacity, or use a standalone VM, billed by the second, for a builder bigger than your plan.

curl -fsSL https://exe-staging.dev/deploy/docker-builder/builder-setup.sh | bash

sudo buildctl debug workers -v

Connect docker buildx#

On your computer, which needs the docker CLI and the Buildx plugin (see Only want the docker CLI?), check that SSH reaches BuildKit:

ssh my-builder.exe.xyz buildctl debug workers

Accept the host key if asked; its fingerprint is exe.dev's. You should see one worker and linux/amd64. Then create a builder named exe and start it:

docker buildx create --name exe --driver remote ssh://my-builder.exe.xyz
docker buildx inspect exe --bootstrap

inspect shows Status: running, BuildKit version: v0.34.2, and the GC rules. Build with it:

docker buildx build --builder exe -t ghcr.io/you/app:dev --push .

To make it the default for docker build and docker buildx build, docker buildx use exe; docker buildx use default switches back. BUILDX_BUILDER=exe does the same for one shell.

If ssh my-builder.exe.xyz doesn't connect from your network, use the other form of the address: ssh://vm+my-builder@vm.exe.xyz.

Tip

Each build opens a new SSH connection. Let them share one with ControlMaster in ~/.ssh/config, as in Make it fast on the Docker host page, with my-builder.exe.xyz on the Host line.

Get the image out#

The build runs on the VM, so its result is there too, in the cache. docker buildx build without an output says No output specified with remote driver and leaves it there. Say where it goes:

• --push sends it straight from the VM to the registry in -t. Run docker login on your computer as usual; Buildx passes your credentials to the build when the registry asks, and they aren't stored on the VM. Pushes count toward your plan's outbound bandwidth.
• --load sends the image back into your local Docker (Gitea's 72 MB took 7.5 s above). To load every build, create the builder with --driver-opt default-load=true (Buildx 0.14 or newer).
• --output type=local,dest=./out copies the final stage's files to ./out, for a binary or a static site.

.dockerignore matters more with a remote builder: whatever your Dockerfile copies is uploaded, and leaving out node_modules, build output, and large data keeps the first upload of each checkout small. Later builds from the same directory send only what changed.

Use it from CI#

A CI job needs three things: an SSH key that exe.dev accepts for my-builder, exe.dev's host key, and a Buildx builder with the same endpoint. Give CI its own key, scoped to this one VM and logging in as an unprivileged user that can build and do nothing else as root.

On the VM, create the user:

sudo useradd --create-home --groups buildkit ci

On your computer, make a key for CI and register it for this VM only, as that user:

ssh-keygen -t ed25519 -N '' -C builder-ci -f builder-ci
cat builder-ci.pub | ssh exe.dev ssh-key add --vm=my-builder --ssh-user=ci
ssh -i builder-ci -o IdentitiesOnly=yes my-builder.exe.xyz 'id; buildctl debug workers'

The last line shows uid=…(ci) with the buildkit group, and the worker. The key can't reach any other VM or run ssh exe.dev commands; see VM-scoped SSH keys. ssh exe.dev ssh-key remove builder-ci revokes it. Then the host key, the same for every exe.dev VM:

ssh-keyscan -t rsa my-builder.exe.xyz | tee known_hosts | ssh-keygen -lf -

The fingerprint must be SHA256:JJOP/lwiBGOMilfONPWZCXUrfK154cnJFXcqlsi6lPo.

GitHub Actions#

In the repository's Settings → Secrets and variables → Actions, add the contents of builder-ci as the secret BUILDER_SSH_KEY, and the line in known_hosts as the variable BUILDER_KNOWN_HOSTS. Then delete both files from your computer. A workflow that builds and pushes to the GitHub Container Registry:

name: image
on: push
permissions:
  contents: read
  packages: write
jobs:
  image:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - name: SSH to the builder
        env:
          KEY: ${{ secrets.BUILDER_SSH_KEY }}
          KNOWN_HOSTS: ${{ vars.BUILDER_KNOWN_HOSTS }}
        run: |
          install -d -m 700 ~/.ssh
          printf '%s\n' "$KEY" > ~/.ssh/id_ed25519
          chmod 600 ~/.ssh/id_ed25519
          printf '%s\n' "$KNOWN_HOSTS" >> ~/.ssh/known_hosts
      - uses: docker/setup-buildx-action@v4
        with:
          driver: remote
          endpoint: ssh://my-builder.exe.xyz
      - uses: docker/login-action@v4
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/build-push-action@v7
        with:
          context: .
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}

The job needs no Docker of its own and no cache-from/cache-to: the cache is on the builder. When the job ends, setup-buildx-action removes the builder from the runner; that touches nothing on the VM.

The same steps work on jobs you run on exe.dev with GitHub Actions (runs-on: [self-hosted, exe]). Keep the key in GitHub's secrets there too, not on the base VM, which every job's VM is copied from.

Warning

Don't give this key to workflows that run on pull requests from forks. Their code would build on your builder, next to your release builds; see Who to share it with.

Other CI#

Any CI that can run a shell step works the same way: write the key to ~/.ssh/id_ed25519 and the host key line to ~/.ssh/known_hosts, then docker buildx create --use --driver remote ssh://my-builder.exe.xyz and docker buildx build --push …. GitLab CI, Buildkite, and Forgejo or Gitea Actions only need the docker CLI with Buildx in the job's image.

Share it with your team#

Anyone who can SSH in as a member of the buildkit group can build. There are two ways to let people in.

• Give them their own access. On your computer:

  ssh exe.dev share add my-builder alice@example.com --root
  ssh exe.dev share add my-builder team --root     # everyone on your exe.dev team
  

  They use their own exe.dev keys and run the same docker buildx create as you. Root access means a shell with sudo on the builder, so they can change it too.

• Give them build-only access. Do as for CI above: a user on the VM in the buildkit group but not sudo, and their public key, added with ssh-key add --vm=my-builder --ssh-user=<user>. They don't need an exe.dev account. The key is on your account, and ssh exe.dev ssh-key remove takes it away.

BuildKit handles many builds at once. Two people building the same commit at the same time share the work, and a step one person has built is cached for everyone. Cores are shared, so a team that builds all day wants a bigger builder or a second one.

Who to share it with#

A shared cache is shared trust. Everyone who can build can run any RUN step on the builder, as root inside BuildKit's sandbox, with the VM's network (BuildKit uses host networking here), and they can write to cache mounts that other builds then use: a malicious build can put a poisoned ~/.cache/go-build or npm cache in front of your release build. So:

• Share a builder with people and CI jobs you would trust with your release builds. Give untrusted code, such as pull requests from forks, a separate builder or GitHub's own runners.
• Pass secrets with --secret and RUN --mount=type=secret; BuildKit doesn't keep them in the cache. Values in ARG and ENV do end up there, and in the image.
• Keep the builder for building. Don't run other services on it that build steps could reach on localhost.

SSH or tcp+mTLS#

BuildKit can also listen on TCP with mutual TLS, which is how most remote builder guides set it up. On exe.dev, SSH is the better default:

Choose TCP when your clients are already on a Tailscale network with the builder and you'd rather hand out certificates than SSH keys, or when a client can't run ssh.

SAN="my-builder 100.101.102.103" docker buildx bake --builder exe \
  "https://github.com/moby/buildkit.git#v0.34.2:examples/create-certs"
scp -r .certs/daemon my-builder.exe.xyz:buildkit-certs

sudo install -d -m 700 /etc/buildkit/certs
sudo install -m 600 ~/buildkit-certs/* /etc/buildkit/certs/ && rm -r ~/buildkit-certs
sudo mkdir -p /etc/systemd/system/buildkit.service.d
sudo tee /etc/systemd/system/buildkit.service.d/tcp.conf >/dev/null <<'EOF'
[Service]
ExecStart=
ExecStart=/usr/local/bin/buildkitd --addr unix:///run/buildkit/buildkitd.sock --group buildkit \
  --addr tcp://0.0.0.0:1234 --tlscacert /etc/buildkit/certs/ca.pem \
  --tlscert /etc/buildkit/certs/cert.pem --tlskey /etc/buildkit/certs/key.pem
EOF
sudo systemctl daemon-reload && sudo systemctl restart buildkit

docker buildx create --name exe-tcp --driver remote \
  --driver-opt cacert=$PWD/.certs/client/ca.pem,cert=$PWD/.certs/client/cert.pem,key=$PWD/.certs/client/key.pem,servername=my-builder \
  tcp://100.101.102.103:1234
docker buildx inspect exe-tcp --bootstrap

Multi-arch images#

exe.dev VMs are x86-64, so linux/amd64 builds natively. BuildKit's release ships QEMU emulators (buildkit-qemu-aarch64 and others), and BuildKit uses them by itself for any RUN step of another platform:

docker buildx build --builder exe --platform linux/amd64,linux/arm64 \
  -t ghcr.io/you/app:dev --push .

docker buildx ls lists only the amd64 platforms; arm64 works anyway. Emulation costs a lot for compiling and little for copying files and installing packages:

So cross-compile what you can: build on --platform=$BUILDPLATFORM and set GOOS/GOARCH (or use xx) from TARGETOS and TARGETARCH, as Docker's multi-platform guide shows. For native arm64, Buildx can add a second node to the exe builder on an arm64 machine running BuildKit (docker buildx create --append), and builds each platform on its own node.

If a step fails with exec format error, the image mixes binaries of several architectures and needs the emulators registered with the kernel. On the VM, sudo apt-get install -y qemu-user-static does that at every boot.

Keep the disk in check#

BuildKit prunes its own cache: when it starts, and a minute after builds. builder-setup.sh wrote these rules, applied in order until the cache fits; percentages are of the VM's disk:

1. Build contexts and Git checkouts unused for 3 days, once they take over 5%. They're cheap to upload again.
2. Cache mounts (RUN --mount=type=cache) unused for 2 weeks, once over 20%. BuildKit's default drops them after 48 hours, which empties a team's Go or npm cache over a long weekend.
3. Anything unused for 30 days, once the cache is over 50%.
4. The least recently used cache, until it's under 60% of the disk and at least 20% of the disk is free, never going below 10%.

See what the cache holds, from your computer:

docker buildx du --builder exe | tail -3

Prune by hand if you need room now:

docker buildx prune --builder exe --filter until=72h --force   # unused for 3 days
docker buildx prune --builder exe --all --force                # everything

To change the limits, edit /etc/buildkit/buildkitd.toml on the VM (Docker's garbage collection guide explains each field), then sudo systemctl restart buildkit and check with sudo buildctl debug workers -v. A restart stops builds that are running, so do it when nobody is building. If the cache is always near its cap, grow the disk instead (ssh exe.dev resize my-builder --disk=200GB); the percentages follow.

Update#

Run the setup script again on the VM when a new BuildKit release is out, when nobody is building:

curl -fsSL https://exe-staging.dev/deploy/docker-builder/builder-setup.sh | bash

It replaces the binaries and restarts the service, keeps /etc/buildkit/buildkitd.toml and the cache, and prints the new version. Clients don't need to change; Buildx and BuildKit agree on what they both support.

ssh my-builder.exe.xyz buildctl debug workers