[auxesis]

Summary

This PR moves every job that runs on Blacksmith to RunsOn.
RunsOn starts a machine in our own AWS account for each job.
After this PR, no job in this repository runs on Blacksmith.

Blacksmith costs too much for this repository's CI.
From 22 to 28 October 2026, these jobs used 8,841 Blacksmith minutes at 16 vCPUs.
That cost $283 for the week.
CipherStash keeps CI spending for all its repositories under $1,000 a month.
The cost model puts this repository at $335 to $567 a month on RunsOn, at October's pace plus 20%.

Changes

Each job below now uses this label in place of its Blacksmith label:

runs-on=${{ github.run_id }}/family=c8a+m8a/cpu=8/image=<image>/volume=80gb/extras=s3-cache

Workflow	Job	Before	Image now
test.yml	Test (PostgreSQL 14 to 17)	blacksmith-16vcpu-ubuntu-2204	ubuntu22-full-x64
test.yml	Build (Rust MSRV)	blacksmith-16vcpu-ubuntu-2204	ubuntu22-full-x64
test.yml	Lint (Rust stable)	blacksmith-16vcpu-ubuntu-2204	ubuntu22-full-x64
test.yml	Encrypted burn-in (PostgreSQL 17)	blacksmith-16vcpu-ubuntu-2204	ubuntu22-full-x64
test.yml	Supported pooler compatibility (PostgreSQL 17)	blacksmith-16vcpu-ubuntu-2204	ubuntu22-full-x64
benchmark.yml	Performance regression check	blacksmith-16vcpu-ubuntu-2204	ubuntu22-full-x64
release.yml	Build binaries + Docker images, linux/amd64	blacksmith-16vcpu-ubuntu-2404	ubuntu24-full-x64
release.yml	Publish multi-platform image	blacksmith-16vcpu-ubuntu-2404-arm	ubuntu24-full-x64

The labels mean:

• family=c8a+m8a: an AMD c8a machine. RunsOn can use an m8a machine when AWS has no spare c8a spot capacity.
• cpu=8: 8 vCPUs. The Blacksmith machines had 16.
• image: the same Ubuntu version that the Blacksmith label named.
• volume=80gb: a larger disk. The default disk can fill during a Rust build.
• extras=s3-cache: the Actions cache is kept in an S3 bucket in our AWS account. The runner reaches it faster than GitHub's cache.

"Publish multi-platform image" moves from arm64 to x64.
It only joins the two per-platform image digests into one manifest list.
That does not depend on the machine's architecture.
The arm64 image itself is still built on linux-arm64-public, a GitHub-hosted runner.

Docker Hub login

Docker Hub limits image pulls without a login for each IP address.
On RunsOn machines it refused some of those pulls.
So every job that pulls from Docker Hub now logs in first.
The login uses the same credentials and action as release.yml: DOCKER_HUB_USERNAME and DOCKER_HUB_PASSWORD from .github/secrets.env.encrypted, and docker/login-action.

• A "Login to Docker Hub" step now follows "Decrypt secrets" in these jobs: the four PostgreSQL tests, the burn-in, pooler compatibility, and the benchmark.
• In release.yml, the build job's existing login step moves up to just after "Decrypt secrets". Before, it ran after mise run build, and that task pulls ubuntu:25.10 to build the image.
• "Build (Rust MSRV)" and "Lint (Rust stable)" pull no images, so they do not log in.

Apart from the runner labels and these login steps, nothing changes.
Job names and timeouts stay the same.
The required check "Build (Rust MSRV)" keeps its name.

Verification

• I ran actionlint 1.7.12 with shellcheck 0.11.0 on all four workflows, on main and on this branch.
  • main: 11 findings. Seven of them are "unknown label" findings for the Blacksmith labels this PR replaces.
  • This branch: 5 findings, and none is new. They are three shellcheck findings in release.yml, and a missing description in the local Slack action, reported at two call sites.
• CI on the commit that adds the Docker Hub login: every job passed on the first attempt, with no re-runs. All 10 login steps printed "Login Succeeded". No log from the Test, Benchmark Proxy, or Release runs contains "toomanyrequests" or "429 Too Many".
• Earlier commits: without the login, Docker Hub refused pulls in the four "Test (PostgreSQL …)" jobs. The details are in the comments on this PR.

Related

• The RunsOn trial in ci: trial RunsOn runner sizes for Mutants and Test JS (do not merge) stack#1297. It used the same label form.
• The decision record "Move CI from Blacksmith to GitHub-hosted runners and RunsOn" (ADR 0002) on the docs/ci-cost-decisions branch of cipherstash/stack.

Review notes

• Docker Hub plan. With a login, the pull limit belongs to the Docker Hub account, not to the machine's IP address. How many pulls the account gets depends on its Docker Hub plan.
• Forks. Workflow runs from every outside contributor's pull request now need approval before they start.
• Half the vCPUs. The new machines have 8 vCPUs, not 16. On the latest run, each "Test (PostgreSQL …)" job took about 13.5 minutes. Its limit is 30 minutes.
• Benchmark baseline. The stored benchmark history comes from Blacksmith machines. Results on RunsOn may differ enough to raise an alert, and the job sets fail-on-alert: true. The first run raised none. Results can also vary between c8a and m8a machines.
• Release build name. GitHub adds the matrix values to the name of the release build job. So the linux/amd64 build's name now shows the RunsOn label. No ruleset requires that check.
• release.yml runs on this PR, because the PR changes that file. As on any such PR, it pushes both images by digest to Docker Hub, and the publish step runs with --dry-run.
• A RunsOn job starts 40 to 50 seconds later than a Blacksmith job did.

[auxesis]

First CI run on RunsOn

Every moved job ran on a RunsOn machine and passed. Each one waited 19 to 22 seconds for its machine. The cache was empty, because this was the first run on RunsOn.

Job	RunsOn, this PR	Blacksmith, last main run
Build (Rust MSRV)	5.2 min	7.7 min
Lint (Rust stable)	1.1 min	0.5 min
Test (PostgreSQL 14)	16.3 min	12.5 min
Test (PostgreSQL 15)	16.0 min	17.6 min
Test (PostgreSQL 16)	15.9 min	18.2 min
Test (PostgreSQL 17)	15.9 min	11.5 min
Encrypted burn-in (PostgreSQL 17)	6.7 min	7.6 min
Supported pooler compatibility (PostgreSQL 17)	10.2 min	6.5 min
Performance regression check	7.1 min	5.0 min
Build binaries + Docker images, linux/amd64	6.6 min	7.0 min

• The Blacksmith column comes from the runs on main on 2026-10-26: Test run 40275357801, Benchmark Proxy run 40275357818, and Release run 40275357773. Those runs had a warm cache.
• Every job finished well inside its timeout-minutes.
• The performance regression check raised no alert.
• "Publish multi-platform image" still ran on blacksmith-16vcpu-ubuntu-2404-arm, as planned.

[auxesis]

Second commit: Docker Hub refused image pulls

"Publish multi-platform image" ran on a RunsOn x64 machine and passed. So did the required check "Build (Rust MSRV)".

The four "Test (PostgreSQL …)" jobs failed. None of the failures came from the tests themselves. Docker Hub refused anonymous image pulls from the RunsOn machines:

• First attempt: all four jobs passed their tests up to the multitenant phase. Then building the proxy image failed. Docker Hub answered the pull of ubuntu:25.10 with "429 Too Many Requests".
• I re-ran the four failed jobs once. PostgreSQL 14, 15, and 16 passed, in about 13 minutes each. PostgreSQL 17 failed after 48 seconds. Docker Hub answered the pull of postgres:17 with "toomanyrequests: You have reached your unauthenticated pull rate limit".

Job	Result on this commit
Build (Rust MSRV)	passed, 4.6 min
Lint (Rust stable)	passed, 0.4 min
Encrypted burn-in (PostgreSQL 17)	passed, 5.0 min
Supported pooler compatibility (PostgreSQL 17)	passed, 7.9 min
Test (PostgreSQL 14, 15, 16)	failed, then passed on the re-run
Test (PostgreSQL 17)	failed twice, on Docker Hub's pull limit
Performance regression check	passed
Build binaries + Docker images, linux/amd64	passed
Publish multi-platform image	passed

Docker Hub limits anonymous pulls for each IP address. The run on the first commit did not hit the limit. Nor did the Blacksmith run on main that I compared with. The same limit has refused pulls on RunsOn machines in cipherstash/stack#1419.

This needs a decision before this PR merges. The options I see:

1. Log in to Docker Hub before the jobs pull images. release.yml already logs in with credentials from .github/secrets.env.encrypted. The limit then depends on the Docker Hub plan.
2. Pull the same official images from public.ecr.aws/docker/library/. This changes the image names in tests/docker-compose.yml, proxy.Dockerfile, and the test Dockerfiles.
3. Add a Docker Hub pull-through cache to the RunsOn stack in AWS. This changes no workflow, but RunsOn documents it for the Terraform install only.