CI speed: make the compiler cache store entries and shorten the macOS path #310

Part of #194. A CI review on 9 October (develop at 8941d478) found new evidence about the compiler cache and seven speed paths that #194 does not yet record. #194 keeps ownership of everything it already lists. This issue adds only the new items, so the lane stays in one place.

Owner: not assigned (Anders triages). Task base and PR target: develop.

Current state: #311 owns the private compiler-cache startup repair and its review. Trusted remote write, warm read and cache-budget proof remain open here. Public PR #326 passed all 11 jobs on its first attempt in run 37973992785 (71m01s overall; macOS 65m23s). This remains a long-run baseline; the private cache repair has not been integrated and hosted time savings are still unproved. PR #326 owns local warning/scope consistency and the installed journey preparation window. The remaining speed items are measured follow-ups after the cache baseline; hosted documentation skips also depend on #317 input coverage.

Summary

A CI run takes 50–76 minutes. The macOS job sets the end time: 46–70 minutes, while every other job finishes by about 45 minutes. Two causes dominate:

1. The shared Rust compiler cache stores nothing. Every job compiles about 1,800 Rust crates cold, on every event.
2. The macOS job runs four long phases one after another: source-home build, model journey, two Runtime builds, and the update journey.

Fix the cache first; the other estimates depend on it.

Evidence

Runs measured: develop push 37902075974 (49m50s) and PR 37906858619 (76m08s), both on 9 October.

macOS job steps (PR / develop): source-home 41.1 / 26.4 min, two Runtime versions 15.9 / 9.4 min, model journey 5.1 / 3.6 min, update journey 5.8 / 5.0 min. The same steps vary by 24 minutes between runs, which points to macos-14 runner speed (3 cores).

Compiler cache.

• Rust hit rates, the same in both runs:
  • source-home-macos: 2 hits and 1,816 misses (0.11 %);
  • source-home-linux x86-64: 4 hits and 1,720 misses (0.23 %).
• All cache error counters are 0, and "Average cache read hit" is 0.000 s. C/C++ hits are about 50 %, which matches repeated compiles in the job's private disk tier.
• The repository cache list (GET /repos/Elacity/elastos-runtime/actions/caches) holds 34 entries using 8.9 GB of the 10 GB cap. None of them comes from sccache. lint, test-elastos and test-capsules use the same action in single-level GHA mode and print no statistics; they add no sccache entry either.
• The develop push is a writer: CI_SAVE_CACHE is true (ci.yml:35) and the action sets READ_WRITE (.github/actions/rust-compile-cache/action.yml:11). A writer runs, but no entry persists.
• CI and tests: warm caches, fail fast, keep only what catches real bugs #194 (8 October, 18:56) expected the first PR after develop run 37828101157 to read a warm cache. Both runs above came after it and read nothing.
• scripts/ci-rust-cache-smoke.py proves local reuse and read-only GHA lookups in a fixture. It does not prove that a develop push stores remote entries (CI and tests: warm caches, fail fast, keep only what catches real bugs #194, finding 8).
• About 4.8 GB of the cache sits in entries last read before 7 October, 03:05 UTC:
  • ubuntu-22.04-arm-ARM64-elastos-bd-source-home-linux-… 2,561 MB;
  • Linux-elastos-test-bd-…-50088228-… 1,388 MB;
  • Linux-elastos-bd-lint-…-50088228-… 456 MB;
  • the 50088228 capsules entry, the macOS-elastos-bd-… entry, two kubo-macOS-ARM64-… entries (one in refs/pull/271/merge) and providers-bd-target-….

Work

Each item is one PR. Item 1 comes first. Items 2–6 are independent of each other, and item 7 comes last.

1. The compiler cache stores and serves entries

Find why no sccache entry persists. Test these candidates in order:

• (a) The GHA backend's write path under cache service v2 (ACTIONS_RESULTS_URL).
• (b) The source-home multi-level chain disk,gha with SCCACHE_MULTILEVEL_WRITE_ERROR_POLICY=l0, which can hide level-1 write errors ("set job build directories" step in source-home-linux and source-home-macos).
• (c) Eviction under the 10 GB cap.

Diagnose with one develop dispatch that sets SCCACHE_LOG=debug and SCCACHE_ERROR_LOG, and uploads the log as an artifact. Then list caches by the sccache- key prefix.

Remove the stale entries listed above. Agree a storage budget per cache family with Anders (#194, finding 8).

• A develop push stores sccache entries. Proof: the cache list after the run shows sccache-* keys in refs/heads/develop.
• The next PR run shows a Rust hit rate of 70 % or more in source-home-macos and source-home-linux. Proof: the sccache statistics in the job summary.
• Total cache use stays inside the agreed budget for three consecutive develop pushes, and current entries stay in the cache. Proof: the cache list after each push.

2. The macOS update hop runs as its own job

Today source-home-macos runs the source-home build, the model journey, "build two actual Runtime versions", fixture generation and the update journey in sequence. A separate update-hop-macos job needs a qualified source-home Runtime and support inputs as well as the pinned previous release: source-gate alone does not produce them. Choose a commit/tree-bound producer handoff or an independent build with qualified support. Both jobs use the same accepted source, toolchain and journey profile. Preserve exact N/N+1 stamps, the previous-release pin, refusal checks and Undo (#194, finding 6). Measure both the critical path and additional runner/artifact cost after the compiler-cache baseline is proved.

• The macOS critical path equals the longer of the two jobs. Proof: three PR runs.
• The hop job still proves install N, update to N+1, the refusals and Undo. Proof: its job summary.
• Anders adds the new job to the required checks. Branch protection is his setting.

3. Heavy jobs start without waiting for source-gate

Every heavy job waits for source-gate, which takes about 4–5 minutes. Starting builds earlier can reduce elapsed time and can spend runner minutes on candidates that fail the cheap gate. Keep the current dependency until measurements show that early start improves the agreed cost and latency goals. If adopted for pull requests, retain source-gate as a required check and prove cancellation at its first failure. Push and merge-group runs need their own cancellation evidence.

• On a PR, the heavy jobs start within one minute of the run. Proof: the job timeline.
• A failing source-gate cancels the PR run. Proof: one deliberate test PR.

4. Capsule workspace throughput is measured and improved

The 25 capsule workspaces already share intermediates through the repository Cargo build directory. First measure Cargo startup, compilation, test execution and shared-directory lock wait. Batch compatible full-package selections within one workspace and identical profile/flags before adding concurrent Cargo processes. Separate capsule manifests require separate workspace calls. Compare a bounded worker count or matrix shards only when the measured bottleneck supports them. Keep warnings as errors, per-package counts, zero-test refusal, failure propagation and readable logs.

• test-capsules finishes in 12 minutes or less. Proof: three runs.
• A failing capsule still fails the job, and the log names the capsule. Proof: one deliberate test PR.

5. The custody image build uses prebuilt or cached Rust layers

The image prebuild takes 17 of the 24 minutes (ci.yml:432-457; deploy/custody-host/Dockerfile). Choose one approach:

• Option A: a cargo-chef dependency layer, so dependency compilation survives source changes.
• Option B: build the binaries on the runner with the compiler cache, and COPY them into the image.

Either way, pin FROM rust:1.91-bookworm (Dockerfile:21) by digest, as the ARM engine container already is.

• A warm prebuild takes 6 minutes or less. Proof: three runs.
• The base image is pinned by digest. Proof: the Dockerfile and a policy test.

6. test-elastos uses nextest

cargo test takes 12–16 minutes. #194 Part 2 already targets the two 3-minute tests and the 17-minute goal; this item adds process-level parallelism. Compare existing compatible Cargo batching with a pinned, checksummed cargo-nextest experiment. Adopt a new runner only after process isolation, CPU/memory limits and expected fixture failure behavior pass. Keep doctests with cargo test --doc and retain the exact case inventory. Test count equality alone does not prove equivalent behavior.

• The test count is equal before and after. Proof: cargo nextest list against cargo test -- --list.
• test-elastos finishes in 12 minutes or less. Proof: three runs.

7. Documentation-only PRs skip the heavy matrix safely

Safe documentation-only changes can use an in-job skip that still reports each required check. Markdown and documentation may also be embedded product or build inputs, so file extensions alone cannot qualify a skip. Implement the input-coverage work in #317 first. Then prove negative cases for embedded Markdown, installer/release inputs, product data and mixed changes. Preserve formatting and product-data gates. Public PR #326 narrows the local gate; the hosted matrix still needs this separate proof.

• A docs-only PR completes in 10 minutes or less. Proof: one test PR.
• The policy test refuses a skip filter that matches any non-documentation path. Proof: the test.

Lower priority

• cancel-on-failure holds a runner for the whole PR run. A longer polling interval reduces API requests while the runner remains allocated. Runner-minute savings require a different cancellation design; keep write authority outside build jobs.
• A larger macOS runner (macos-14-xlarge) shortens items 1–2 further, at extra cost. This is Anders' budget decision.

Security and consistency rules for every item

• Only trusted producers write shared caches: develop and main pushes, and the merge group if Anders enables it. PR runs read only (CI and tests: warm caches, fail fast, keep only what catches real bugs #194 Part 0). The cache credential stays with the cache process; a private security advisory linked from the remediation parent issue tracks this.
• Shipped release builds stay hermetic: release-package.yml keeps RUSTC_WRAPPER: ''.
• Journey builds may use the CI journey profile (LTO off, 16 codegen units). Release acceptance uses the shipping artifacts (CI and tests: warm caches, fail fast, keep only what catches real bugs #194, finding 4).
• New tools (sccache, nextest, cargo-chef) are pinned and checksummed.
• The nine required check names stay the same, unless Anders changes branch protection in the same step.
• Measure three PR runs and three develop runs before and after each item. Record the median wall time and the macOS job time in CI and tests: warm caches, fail fast, keep only what catches real bugs #194.

Decision for Anders: one CI run per tree (merge queue)

Anders decided on 8 October to keep the merge queue off for now. This section records the current evidence for the next review of that decision.

• develop protection: required_status_checks.strict is false, required_pull_request_reviews is null, and the nine checks are required. main: strict is true and reviews are required.
• Ruleset "develop merge queue (CI and tests: warm caches, fail fast, keep only what catches real bugs #194)" is disabled.
• scripts/release-publish.sh:55-75 (checked_ci) now accepts a successful merge_group run for the exact package SHA, so the release gate is ready for queue evidence.
• From 6 to 8 October, 13 of 26 develop push CI runs ended cancelled and 13 succeeded (gh run list --workflow ci.yml --event push --branch develop --created 2026-10-06..2026-10-08). A cancelled develop run leaves its tree without the develop CI evidence that prepare requires.
• Anders' 8 October measurement: about 1,545 of 5,622 job-minutes in 24 hours re-proved a tree that CI had already tested.
• Because strict is false, two PRs that each pass against an older base can merge and break develop together. Their combined tree is first tested on the develop push.

Decision requested: enable the queue with lighter PR runs (branch ci/one-run-per-tree, 63bedd01), or keep the current setup and record the accepted risk.

Target

These estimates need measurement:

• Today: 50–76 minutes.
• After item 1: about 30–40 minutes.
• After item 2: about 25–30 minutes.
• After item 3: about 20–25 minutes.

Items 4 and 5 then keep the Linux jobs below that line.

How to work on this (any person or agent)

• Read AGENTS.md first. Follow its rules for branches, reviews, push approval and the acceptance record.
• Branch from current develop as fix/<slug>, one PR per item. Workflow changes conflict easily; check open PRs that edit ci.yml (for example CI and tests: warm caches, fail fast, keep only what catches real bugs #194's batches) before you start.
• Line numbers refer to develop 8941d478. If lines moved, find the code by the quoted step or job name.
• Local checks: just ci-local-prepush. Workflow edits also need python3 scripts/ci-release-policy-test.py and the pinned actionlint.
• Proof is CI evidence: link the run IDs and quote the job-summary numbers. Keep one acceptance record in this issue, as AGENTS.md describes.