Skip to content

Build coverage-docker's binary once per leg - #1261

Merged
Mikola Lysenko (mikolalysenko) merged 1 commit into
mainfrom
ci-perf/1199-coverage-docker-one-build
Oct 9, 2026
Merged

Mikola Lysenko (mikolalysenko) merged 1 commit into
mainfrom
ci-perf/1199-coverage-docker-one-build

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1199

Problem

Each of the 10 coverage-docker legs (npm, pypi, gem, cargo, golang, maven, composer, nuget, deno, sbt) compiled the dependency graph twice (#1199, profiler sample over 55 merge_group, 15 PR and 6 push runs):

  • Build instrumented socket-patch binary: cargo build --bin socket-patch under cargo llvm-cov show-env into target/. It took 2.4–2.9 min per leg, 2.7 min p50.
  • Run <eco> Docker e2e test with coverage: cargo llvm-cov --no-report rebuilt everything into target/llvm-cov-target/ (~2m50s), and then the tests ran in seconds.

That is ~22.5 / 18.3 / 25 job-min per merge_group / PR / push run, about 8–10k Linux job-min/day.

The first build was also wasted. In cargo-llvm-cov 0.8.7, report outside show-env sets its target dir to target/llvm-cov-target (src/cargo.rs). It merges only <that dir>/*.profraw (src/report.rs merge_profraw) and walks only that dir for objects (object_files). So neither target/debug/socket-patch nor the in-container profraws, which landed in target/, reached the per-ecosystem lcov.

Change

.github/workflows/ci.yml, coverage-docker job only:

  • Removed the Build instrumented socket-patch binary step.
  • SOCKET_PATCH_COV_BIN now points at target/llvm-cov-target/debug/socket-patch. This is the instrumented binary that the test step's own cargo llvm-cov builds; cargo builds a package's bins before it runs that package's integration tests. It uses the same RUSTFLAGS and features (docker-e2e) as the test binaries.
  • SOCKET_PATCH_COV_PROFRAW_DIR now points at target/llvm-cov-target, so cargo llvm-cov report merges the in-container profraws.

The binary is still built on the ubuntu-22.04 host, so the glibc constraint described in the job comment still holds.

Expected saving

  • About 2.7 min per leg × 10 legs, so ~22–27 job-min per CI run and ~8–10k Linux job-min/day.
  • Each leg gets ~2.7 min shorter. That includes coverage-docker (sbt), which is usually what coverage-merge waits on in the merge queue. Expect ~1 min off merge-queue p50; Windows test and Gradle e2e also bound that path.
  • Bonus: in-container code paths now count toward the docker lcov.

Measured result

This PR's run is 37918223729. The baseline is PR run 37914251835, from just before.

leg before (min) after (min)
npm 6.5 4.1
pypi 6.8 4.1
gem 6.7 3.9
cargo 6.3 3.4
golang 6.0 2.4
maven 7.0 3.9
composer 6.6 3.7
nuget 6.7 4.1
deno 6.6 4.0
sbt 10.1 8.0
total 69.3 job-min 41.6 job-min

That is −27.7 Linux job-min per CI run (−40%). The sbt leg, which coverage-merge usually waits on, got 2.1 min shorter. All 10 legs passed. Merge-queue effects show only after merge; the profiler will verify them.

The coverage now reaches the lcov. I compared per-ecosystem lcov artifact sizes against main's push run 37910917261 at this PR's base commit:

  • cargo, deno, golang and sbt were byte-identical-sized there (527,4xx B), meaning nothing from inside their containers was counted. They are now 538,587–542,167 B.
  • npm went from 528,397 to 550,815 B, and pypi from 534,589 to 550,859 B.
  • Every leg grew by 7–22 KB.

Where each test runs

Nothing moves. Every docker e2e suite and filter still runs in coverage-docker on every PR, merge_group and push, and e2e-docker still runs nightly. ci-ok and clippy are unchanged, and ci-ok still needs coverage-docker and coverage-merge.

Risk

  • If cargo did not build the bin before the docker tests, docker run -v <missing path> would fail loudly, not silently. The integration tests already rely on CARGO_BIN_EXE_socket-patch, which guarantees the build.
  • The merged lcov may now show more covered lines, from the in-container runs. That is intended.

Validation: YAML parses; actionlint and zizmor --offline output on ci.yml is identical before and after (all findings are pre-existing).

Note: open PRs #1251, #1247 and #1009 also edit ci.yml, but only in hunks far from coverage-docker (lines ~1127 and ~1375).

🤖 Generated with Claude Code

https://claude.ai/code/session_01NwbTSymWm3Yuq2kZW9Eagi


Generated by Claude Code


Note

Low Risk
CI workflow-only change; failure modes are loud (missing mount path or unchanged test behavior via CARGO_BIN_EXE_socket-patch).

Overview
Removes the redundant ~2.7 min per-leg cargo build --bin socket-patch under cargo llvm-cov show-env in the coverage-docker job. That build wrote to target/ but cargo llvm-cov report only merges profraws and objects from target/llvm-cov-target/, so the extra compile was wasted CI time and did not improve lcov.

Coverage hooks now point SOCKET_PATCH_COV_BIN and SOCKET_PATCH_COV_PROFRAW_DIR at target/llvm-cov-target, matching the instrumented binary and in-container profraws produced when the existing cargo llvm-cov --no-report step runs docker e2e tests (cargo builds package bins before integration tests). Expected savings are on the order of ~22–27 job-minutes per CI run across the ten ecosystem matrix legs, with docker paths more likely to appear in merged coverage.

Reviewed by Cursor Bugbot for commit 4e07334. Configure here.


Generated by Claude Code

Each coverage-docker leg compiled the whole dependency graph twice: a
`cargo build --bin socket-patch` under `cargo llvm-cov show-env` into
target/, then `cargo llvm-cov --no-report` into
target/llvm-cov-target/. The first build (~2.7 min per leg, 10 legs
per run) was also wasted: `cargo llvm-cov report` outside show-env
reads only target/llvm-cov-target/*.profraw and objects under that
directory, so neither the target/debug binary nor the in-container
profraws written to target/ reached the lcov.

Mount the socket-patch that the test step's own build produces for
the integration tests and write the container's profraws into
target/llvm-cov-target, so the in-container code paths merge into
the per-ecosystem lcov. Fixes #1199.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NwbTSymWm3Yuq2kZW9Eagi
@mikolalysenko Mikola Lysenko (mikolalysenko) added the ci-perf CI / merge-queue performance finding (profiler routine) label Oct 9, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 4e07334. Configure here.

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Oct 9, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Labeled Ready for review by the burn-down agent.

  • Head: 4e073342b0e5a82595f3d39aa75b35e7fe78146b
  • CI: all 285 check runs on this head completed success/skipped/neutral; no merge conflict with main.
  • Bugbot: reviewed this head, no findings; no unresolved review threads.
  • CHANGELOG.md untouched.

Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Final review brief (4e073342b)

What it does: Removes the separate instrumented cargo build --bin socket-patch step from each of the 10 coverage-docker legs. Docker e2e now mounts the binary that the test step's own cargo llvm-cov already builds in target/llvm-cov-target, and in-container profraws are written to that directory too. CI-only, one file.

Risk: low. It's a workflow change in a single job. A wrong path fails loudly (missing docker -v mount), not silently. The PR's run passed all 10 legs at about 40% fewer job-min, and per-ecosystem lcov grew 7-22 KB, so in-container coverage that used to be dropped now reaches the report.

Look here:

  • ci.yml:713-724: the new hook paths. Removing the build step relies on cargo building the package's bins before its integration tests (CARGO_BIN_EXE_socket-patch).
  • ci.yml:757: cargo llvm-cov report only merges target/llvm-cov-target/*.profraw, which is why the old target/ paths never counted.

Verified: Read the diff and the consumers of SOCKET_PATCH_COV_BIN/_PROFRAW_DIR (docker_e2e_*.rs, docker_vendor_common). The test step has no --target/profile override that would move the binary. No other job used the removed step. CHANGELOG.md untouched. CI 285/285 success/skipped (ci-ok, clippy green), Bugbot success, no review threads, mergeable.

Changes I made: none.

Auto-merge is armed, so approving sends it straight to the merge queue.


Generated by Claude Code

@mikolalysenko
Mikola Lysenko (mikolalysenko) added this pull request to the merge queue Oct 9, 2026
Merged via the queue into main with commit fa2933e Oct 9, 2026
285 checks passed
@mikolalysenko
Mikola Lysenko (mikolalysenko) deleted the ci-perf/1199-coverage-docker-one-build branch October 9, 2026 13:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-perf CI / merge-queue performance finding (profiler routine) Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI perf: coverage-docker — each of 10 legs compiles the dependency graph twice (target/ and llvm-cov-target/) (~10,000 Linux job-min/day)

3 participants