CI perf: coverage-docker (sbt) — 5.6-min image rebuild warms 3 unused toolchains; merge-queue critical path (~2.5 min/merge-group run) #1225

Measurement

Window: CI runs from 2026-10-08 18:00 to 2026-10-09 04:16 UTC. Job detail for 23 successful merge_group runs, 7 pull_request runs and 4 push runs.

• coverage-docker (sbt) takes 13.0 min p50 / 14.5 min p90 (n=23 merge_group). The other nine coverage-docker legs take 6.2–7.2 min p50.
• It is now the merge-queue critical path. In 7 of the last 10 successful merge_group runs (since 22:50), the last job before ci-ok was coverage-merge, which waits for coverage-docker (sbt). Examples:
  • run 37876500787: sbt leg 7.9 → 21.4 min, ci-ok at 23.3. The next-latest job, test (windows-latest, 2), ended at 20.2.
  • run 37871671066: sbt leg 8.6 → 21.0, ci-ok at 22.0. Next-latest: Gradle e2e shard at 18.9.
  • run 37856468611: sbt leg 8.3 → 21.5, ci-ok at 22.4. Next-latest: Gradle e2e shard at 19.1.
• The leg runs on every non-draft PR push, merge_group run and push to main. That is about 300 runs/day after cancellations.

Where the time goes

Per-step p50 for coverage-docker (sbt) (n=37, all events):

The maven leg runs the same steps, and its image build takes 0.2 min.

The job log breaks down the 354 s image build. Layer #18, the warm-cache RUN, takes 335 s:

The three JDK COPY steps, the sbt launcher, Mill and scala-cli downloads take about 20 s together.

Root cause

1. coverage-docker builds tests/docker/Dockerfile.sbt with its defaults. The Dockerfile is written for the nightly/compat matrix, so it warms SBT_WARM_VERSIONS="0.13.18 1.2.8 1.13.0 2.0.9", Mill and scala-cli. The blocking slice in ci.yml only runs SOCKET_PATCH_SBT_DOCKER_VERSIONS="1.2.8 1.13.0" with FILTER="agent_sbt_". About 140 s of each build warms toolchains this job never runs.
2. driver: docker has no layer cache (see the comment in ci.yml), so the whole build repeats in every run.
3. "Build sbt image" (5.6 min) and "Build instrumented socket-patch binary" (2.7 min) run one after the other. They don't depend on each other: the image build does not read target/.

Proposed fix

All changes are in .github/workflows/ci.yml (coverage-docker) and tests/docker/Dockerfile.sbt.

1. Trim the warm set for the blocking slice.
   • In Dockerfile.sbt, add ARG SBT_WARM_TOOLS=1 and guard the Mill/scala-cli warm block with it.
   • In coverage-docker's "Build image" step, pass build-args only for matrix.ecosystem == 'sbt': SBT_WARM_VERSIONS=1.2.8 1.13.0 and SBT_WARM_TOOLS=0.
   • Defaults stay unchanged, so e2e-docker (nightly) and sbt-compatibility.yml still get the full image.
   • Saves about 140 s.
2. Overlap the image build with the instrumented cargo build.
   • Option A: run the sbt image build as docker build ... & in a step before "Build instrumented socket-patch binary", and wait on it before the test step.
   • Option B: move the cargo build ahead of the image build and start the image build in the background.
   • Either way, the 2.7 min cargo build hides behind the image build. This works for every leg, but only matters for sbt, whose image build is the long one.
   • Saves about 2.7 min on the sbt leg.

Expected leg time: 13.0 → about 7.5–8 min, in line with the other nine legs.

Expected saving

• Merge-queue critical path: the sbt leg would end at about 16 min instead of about 21.4. The next blockers are test (windows-latest, 2) (ends at about 20), Gradle e2e shards (about 18–19, CI perf: Gradle e2e shards unbalanced — b-p leg is the merge-queue critical path (~3 min/merge-group run) #1171) and host coverage (about 19). ci-ok moves from about 23 to about 20.5 min, so ~2–3 min per merge_group run in the ~70% of runs where this leg is last.
• Job-minutes: fix 1 saves about 2.3 min per run on Linux. Fix 2 doesn't reduce billed minutes, because the work overlaps instead of disappearing. That is ~2.3 min × ~300 runs/day ≈ ~700 Linux job-min/day.
• PR feedback latency: no change. test-release is still the PR critical path (CI perf: ci.yml test-release — single 29.5-min job is now the whole PR critical path (~12 min off PR push→ci-ok) #1181).

Coverage and risk

• The blocking slice runs exactly the same tests: the agent_sbt_ cells on 1.2.8 and 1.13.0. Their warm caches stay baked in.
• Cells on sbt 0.13.18 and 2.0.9, Mill and scala-cli run only in nightly e2e-docker and sbt-compatibility.yml. Both keep the default full warm set, so nothing loses coverage.
• ci-ok and clippy are unchanged.
• Risk: if a future change adds a 0.13.18 or 2.0.9 cell to the blocking slice, it would resolve cold. That slows it down but does not break it. A comment next to the FILTER and build-args lines should say the two lists must match.
• The background docker build must surface its exit status (wait $pid) so a failed image build still fails the job.

Effort

S. Two small edits, about 15 lines.

ROI

Dashboard scale: (0.7k Linux job-min/day + 2.5 critical-path min) × 0.8 confidence ÷ 1 (S) ≈ 2.6

Generated by Claude Code