Design: fleet dispatch-test — explicit-dispatch CI over fleet runner capacity

Design: `fleet dispatch-test` — explicit-dispatch CI over fleet runner capacity

Status: design of record + MVP built same day (2026-11-06). Direction: Drew, 2026-11-06 11:28 CST.
Supersedes the trigger request in #5663 (closed as superseded): no GitHub Actions
triggers are enabled anywhere — this mechanism is explicit workflow_dispatch only,
consistent with the standing no-auto-CI policy (2026-11-04 ruling, PR #5501). Label:
GitHub Pro / runners program.

Problem

Fleet CI capacity is spread across several GitHub scopes (personal + two orgs), each
with its own hosted-runner concurrency. Load is wildly uneven: measured 2026-11-06
~11:31 CST (in-progress / queued runs): developing-today/code 45 / 244,
developing-today-forks/opencode 7 / 11, developing-today-forks/openchamber 4 / 17,
dezren39/darwin-nix 1 / 1, dezren39/hatch 0 / 0. There is no single place to ask
"where is a runner free right now?" and no generic way to run a build+test on the
emptiest scope with results written back reliably.

The subcommand

New domain in the fleet multicall binary (pkgs/fleet, epic #5412 family):
fleet dispatch-test, verbs scan, submit, status.

Capacity scan

For each scope repo, in configured order:

GET /repos/{r}/actions/runs?status=in_progress — for each run,
GET /repos/{r}/actions/runs/{run_id}/jobs and count jobs with status
in_progress or queued.
GET /repos/{r}/actions/runs?status=queued — each queued run counts as 1 active
unit (a not-yet-started run exposes no jobs; this is the documented
approximation).
Workflow presence: GET /repos/{r}/actions/workflows — the scope is eligible
only if an active workflow file named fleet-dispatch-test.yml exists there.

Eligible = active jobs < threshold AND workflow present. First eligible scope
wins. Threshold is a tunable: --threshold flag / [dispatch-test] threshold
config, default 40 (Drew's tentative figure). Scopes default to
developing-today/code, developing-today-forks/opencode,
developing-today-forks/openchamber, dezren39/darwin-nix, dezren39/hatch
(config-overridable). At the measurement above, code fails the threshold test
outright and hatch is the first scope that is both under threshold and
workflow-equipped — the scan demonstrates both rules on real data.

Auth: a GitHub token resolved by name through fleet-secrets
(default github.ghcr-classic-pat — classic PAT, scopes include repo +
workflow, verified live 2026-11-06; covers every scope above because the
fleet operates as dezren39 in all of them). Follow-up: move to
developing.today-fleet App installation tokens for least-privilege per scope.

Job store: Turso (MVP choice)

Chosen: a dedicated Turso database fleet-dispatch-jobs (created 2026-11-06),
one table:

jobs(id TEXT PRIMARY KEY, created_at TEXT NOT NULL, name TEXT,
     target_repo TEXT, target_ref TEXT, script TEXT NOT NULL,
     status TEXT NOT NULL, run_id INTEGER, run_url TEXT,
     exit_code INTEGER, output_tail TEXT, updated_at TEXT NOT NULL)

Why Turso over a CF Worker queue endpoint for the MVP: the fleet already runs on
Turso everywhere (brokers, atlas, ledgers), the HTTP pipeline API
(POST /v2/pipeline) is callable identically from the laptop binary (ureq) and
from a runner step (curl + jq), there is no new service to deploy or keep alive,
and a dedicated database keeps the blast radius at one table. A CF Worker
front-end remains the right hardening step if runner-side access ever needs
rate-limiting or payload validation — the store schema does not change.
Status flow: queued → dispatched → running → succeeded | failed | error.
Token: turso.fleet-dispatch-jobs-token (BWS; also the pilot repo's
DISPATCH_TURSO_TOKEN Actions secret; DB URL is the repo variable
DISPATCH_TURSO_URL).

Job patterns

(2) ID-input — built first (MVP). fleet dispatch-test submit generates an
internal id (dt-<millis base36>-<4 hex>), stores the payload (script, ref,
name) under that id, scans, records the chosen scope on the row, then dispatches
the pilot workflow with the single input job_id. The workflow fetches its job
by id, marks it running (with run id + URL), executes the script, and in an
if: always() step writes back the terminal status, exit code, and the output
tail (last 6000 bytes) — so status lands even on failure, and a crashed or
timed-out run leaves the row in running where status shows it honestly
instead of pretending success. A job can be re-dispatched by id (fresh run row
fields overwrite); retry is roughly idempotent because the payload is immutable
and keyed by the id.

(1) No-inputs pull — designed, not built. A generic workflow dispatched with
no inputs boots, claims the newest queued row (single UPDATE … WHERE id = (SELECT … ORDER BY created_at DESC LIMIT 1) RETURNING shape), executes it, and
writes back exactly as in pattern (2). This is the better shape for bulk work
(no per-job dispatch call, runners pull when free); it lands as a second
workflow mode once pattern (2) has soaked. The store and write-back contract
are already compatible.

Pilot workflow

dezren39/hatch (chosen: zero active runs at measurement time, PAT-owned, no
real CI to disturb) carries .github/workflows/fleet-dispatch-test.yml:
workflow_dispatch with required input job_id, permissions: contents: read,
30-minute timeout, steps fetch → checkout(job ref) → execute → write-back
(always) → upload job-output.log artifact (14-day retention). The workflow
must live on the repo's default branch for dispatch to see it (GitHub rule).

Artifact quotas + return paths

(Research numbers posted as a comment on this issue: per-account artifact
storage GB, cache GB, retention defaults/min/max, private-repo minutes,
concurrency — dezren39 GitHub Pro personal, developing-today org,
developing-today-forks org, with sources.)

Return-path evaluation:

GitHub Actions artifacts (default). Zero new infra; the output log rides
the run. Limits: per-account storage pools are small on personal/org plans
(see research comment) and retention is capped; fine for logs and small
outputs, wrong for build products.
R2 via a Cloudflare Worker (fallback, when artifacts don't fit). The
fleet already operates Workers + R2 patterns (webhook-gateway, hatch
workers); a presigned-upload Worker endpoint lets a runner stream large
outputs to R2 with no GitHub storage pressure. Build only when a real job
outgrows (1).
Tunnel straight to the laptop (last resort). The lab tunnel
(lab.security.cab) proves the path exists, but runner→laptop transfers
couple CI results to home-network availability; use only for outputs that
must never leave fleet hardware.

Recommendation: (1) by default, (2) when size/retention demands, (3) only on
a stated privacy need. The store's output_tail covers the common case
without any artifact at all.

MVP scope (this build)

fleet dispatch-test domain: scan / submit / status, threshold tunable,
config in [dispatch-test], tokens by name. (PR + merge link below.)
Turso store as above; pilot workflow in dezren39/hatch.
One end-to-end proof: a real submitted smoke job (checkout ref, print
environment, exit 0) with scan choice, run URL, and final stored status
recorded on this issue.
Out of scope for the MVP: pattern (1), App-token auth, R2 return path,
installing the new binary fleet-wide (rides the normal fleet build/deploy
path), any change to runner strategy beyond this dispatch mechanism.

Risks / notes

workflow_dispatch requires the workflow on the default branch of each
participating repo — adding scopes later is a one-file PR per repo.
Runner-side store token is full-access to this one small database by
design; rotation = turso db tokens invalidate fleet-dispatch-jobs +
re-mint + BWS + repo secret update.
The queued-runs-as-1 approximation can undercount a queued multi-job
matrix run; acceptable for a capacity heuristic, revisited if it misleads.

Acceptance criteria

fleet dispatch-test scan reports per-scope in-progress/queued load and workflow eligibility for the five configured scopes, honoring the threshold flag/config (default 40).
submit runs a build+test job on the first eligible scope via explicit workflow_dispatch only — no automatic GitHub Actions triggers are enabled anywhere.
status reports the dispatched run's state and result from the Turso job store (fleet-dispatch-jobs), including run URL and exit code.
The design's MVP scope is either landed or split into tracked implementation issues linked from this issue; no part of the design sits silently.

CODE-2002

Research: GitHub Actions artifact quotas, retention, concurrency (2026-11-06)

All numbers from GitHub Docs / the official actions/upload-artifact README, crawled
2026-11-06. Sources: [BILL] https://docs.github.com/en/billing/concepts/product-billing/github-actions ·
[LIM] https://docs.github.com/en/actions/reference/limits ·
[RET] https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository ·
[DISP] https://docs.github.com/en/actions/how-tos/manage-workflow-runs/manually-run-a-workflow ·
[UP] https://github.com/actions/upload-artifact

Billing-model note: current docs express included compute in USD — Free $12/mo, Pro
$18/mo, Team $18/mo, Enterprise Cloud $300/mo. At the standard Linux 2-core rate of
$0.006/min those equal 2,000 / 3,000 / 3,000 / 50,000 Linux minutes [BILL].

dezren39 — personal, GitHub Pro

Item	Value
Artifact storage included	1 GB, shared pool with GitHub Packages
Cache storage	10 GB per repository
Overage	shared artifacts+Packages $0.25/GB-month; cache $0.07/GB-month; billed on GB-Hours accrued hourly — deleting stops future accrual, never refunds accrued
Included compute (private repos)	$18/mo = 3,000 Linux minutes
Public repos, standard runners	free, no draw on included compute
Concurrent jobs (standard hosted)	40 total (max 5 macOS)

developing-today — org (plan not publicly disclosed; both rows)

Item	Free org	Team
Artifact storage	500 MB shared w/ Packages	2 GB shared w/ Packages
Cache	10 GB per repo	10 GB per repo
Included compute	$12 = 2,000 Linux min	$18 = 3,000 Linux min
Concurrent jobs	20 total	60 total (larger runners: 1,000)

developing-today-forks — org

Identical published numbers to developing-today in both plan rows.

Fleet aggregate concurrency: 40 + 2×20 = 80 (orgs on Free) or 40 + 2×60 = 160
(orgs on Team) — concurrency is per account/plan, not per repo [LIM]. (Note: the
default scan threshold of 40 in this design equals the entire Pro account's
concurrency cap — a scope-level heuristic, revisited once real dispatch data
exists.)

Rules common to all accounts

Retention: default 90 days (artifacts + logs). Configurable per repo/org:
public repos 1–90 days, private/internal 1–400 days, new artifacts only.
upload-artifact retention-days honors the repo setting's max [RET][UP].
Per-job artifacts: max 500 per job [UP]. No documented single-artifact
size cap; the binding constraint is the account storage pool.
Logs/job summaries do not count toward artifact storage [BILL].
workflow_dispatch: max 25 inputs, max payload 65,535 chars; workflow
file must be on the default branch; write access required. No special billing
treatment — a dispatch run bills like any run to the repo owner [DISP][BILL].
Unverified: whether artifacts in public repos are exempt from the shared
storage pool — no authoritative page found says so. Do not design on public-repo
artifacts being free/unmetered.
Related caps [LIM]: matrix 256 jobs/run; hosted job execution 6 h; GitHub
Support can raise concurrency but cannot raise storage limits.

What this means for the return path

Included artifact space is small and account-wide (1 GB Pro / 0.5–2 GB per org,
shared with Packages), billed by GB-Hour at 90-day default retention — a fleet
returning ~1 GB/day at default retention accrues ~90 GB-months. Artifacts are
therefore the right default only for small, short-lived outputs (the pilot uses
retention-days: 14 for a single log file; retention-days: 1 + prompt
download/delete is the pattern for anything routine). Bulk or long-retention
results belong on the R2-via-Worker path (§ return-path evaluation in the issue
body). The store's output_tail covers the common case with no artifact at all.

MVP progress (2026-11-06): store + pilot workflow live, loop proven manually

Store: Turso database fleet-dispatch-jobs created; jobs table per the design
schema. Token vaulted in BWS as turso.fleet-dispatch-jobs-token (name only) and set
on the pilot repo as the DISPATCH_TURSO_TOKEN secret; DB URL is the
DISPATCH_TURSO_URL repo variable.

Pilot workflow: dezren39/hatch carries .github/workflows/fleet-dispatch-test.yml
on its default branch (state: active) — PRs dezren39/hatch#5 (initial) and
dezren39/hatch#6 (fix below), merge commits 7c71f768 / bc163211.

Manual end-to-end proof (job row inserted directly, workflow dispatched with the
id — the same path fleet dispatch-test submit drives): job dt-manual-smoke-1,
run https://github.com/dezren39/hatch/actions/runs/42263780711 — success. Final
stored row: status=succeeded, exit_code=0, run_id=42263780711, output_tail holds
the executed script's real output (echo marker, uname -a, git rev-parse HEAD =
bc163211). Output artifact dispatch-job-dt-manual-smoke-1 (345 bytes, 14-day
retention) uploaded. The failed first attempt's row was reset and re-dispatched by
the same id — retry-by-id works as designed.

Finding worth keeping (bit us live): Turso's /v2/pipeline API requires
integer argument values encoded as JSON strings ({"type":"integer", "value":"42263780711"}). A raw JSON number fails the entire request with a
top-level {"error":"JSON parse error: invalid type: integer …, expected a borrowed string"} — no partial results. First smoke run (42263402318) hit exactly this on
run_id and the fetch step misread it as "job not found"; fixed in hatch#6 (string
integers + surface the store response on fetch failure), and the same rule is being
applied in the Rust subcommand.

Remaining: the fleet dispatch-test subcommand PR (pkgs/fleet) and the full
submit→scan→dispatch→status proof through the binary.

MVP complete: end-to-end proof through the binary (2026-11-06)

Subcommand merged: PR #6589, merge commit 0819afd0fc5a987b1602ab880f204908a2183f08
— new fleet-dispatch-test crate + registry/config/bin wiring, all inside
pkgs/fleet (+1269 lines). 11/11 crate tests pass (including the Turso
integer-as-string rule and pipeline error-shape handling); registry/config/bin
test suites pass; live scan smoke on the merged code:

developing-today/code active=266 workflow=no eligible=no
developing-today-forks/opencode active=10 workflow=no eligible=no
developing-today-forks/openchamber active=16 workflow=no eligible=no
dezren39/darwin-nix active=0 workflow=no eligible=no
dezren39/hatch active=0 workflow=yes eligible=yes
chosen: dezren39/hatch

(code is over the 40 threshold on its own; three scopes are under threshold but
carry no dispatch workflow yet; hatch is the first scope passing both tests.)

End-to-end run: fleet dispatch-test submit --name "e2e proof 2026-11-06"
produced:

{"job_id": "dt-mw5a91oz-b27b", "repo": "dezren39/hatch",
 "active_at_scan": 0, "threshold": 40, "status": "dispatched"}

Run: https://github.com/dezren39/hatch/actions/runs/42267204516 — and
fleet dispatch-test status dt-mw5a91oz-b27b reads back the final stored record:
status=succeeded, exit_code=0, output_tail containing the job script's real
output (dispatch-test-e2e-proof, runner uname -a, git rev-parse HEAD =
bc163211). Submitted 18:03:14Z, terminal status stored by 18:03:36Z.

Deliberately not done in the MVP (per design): pattern (1) no-inputs pull,
App installation-token auth, R2 return path, fleet-wide install of the new
binary (rides the normal fleet build/deploy path — the proof ran the worktree
build of the merged code), and adding the workflow to further scopes (one-file
PR per repo when wanted).

Process note: both merges in this build (hatch#5/#6, code#6589) went through the
REST merge endpoint because the user token's GraphQL pool was secondary-rate
-limited at merge time; all merges verified by readback.

Correction of record — Drew, 2026-11-06 11:30–11:32 CST (delivered while this was being built; the build lane was unreachable for steering, so it lands now):

The MVP above proves the dispatch mechanics — capacity scan, ~40 threshold, Turso job store, ID-input jobs, status/output write-back, workflow_dispatch only — and those carry over. But the execution model differs from the design Drew specified in three defining ways:

Execution home. The generic workflow belongs in ONE place: the fleet repo under dezren39 that holds the other repos as git submodules. The MVP instead treats five repos as peer scopes and requires the workflow file installed per-repo.
Discovery. Eligible execution homes are copies of that fleet repo, found by following its fork network (forks exist in a few other orgs). The scan runs over those copies' active Actions jobs; a copy under the threshold (~40, tunable) takes the build. The MVP scans a hardcoded list of the source repos themselves.
The job. The generic workflow runs ANY build for ANY contained repo: the payload names submodule + ref + commands, executed inside the fleet repo's checkout of that submodule, using the fleet repo's partially preloaded nix cache. The MVP's payload is a script run in a target repo's own checkout — no submodules, no cache.

Phase 2 = rework the execution model to 1–3 on top of the proven mechanics. First deliverables: identify the fleet repo (the dezren39 repo whose .gitmodules contains the other repos) and map its fork network / copies.

Phase 2 — Step 0: fleet repo identified + fork network mapped

Fleet repo: dezren39/fleet — the dezren39 repo whose .gitmodules holds the other repos as submodules. Verified via gh api repos/dezren39/<repo>/contents/.gitmodules across the candidate set:

dezren39/fleet ✔ — submodules: code (developing-today/code), opencode + openchamber (developing-today-forks/*), nix (dezren39/darwin-nix), hatch, hatch-personal, cairn, zettel, dezren39-zettel, org-readme, zulip-archive, hardware-doc.
Exactly one match: dezren39/hatch, dezren39/darwin-nix, dezren39/hatch-personal, dezren39/cairn, dezren39/openchamber have no .gitmodules; dezren39/openclaw-personal has one but it only self-references (openclaw), not the fleet set.

Fork network (gh api repos/dezren39/fleet/forks; both forks report 0 forks of their own, so the network is one level deep today):

Copy	Org / owner	Notes
`dezren39/fleet`	dezren39 (source)	default branch `main`
`developing-today/fleet`	developing-today	fork
`developing-today-forks/fleet`	developing-today-forks	fork

These three are the dynamic scope set for phase 2: fleet dispatch-test scan discovers them (source + forks, recursively) instead of the MVP's hardcoded five-repo list; config [dispatch-test] scopes remains a pin/override.

Cache wiring (found, not invented): the fleet repo's partial preloaded nix cache is the ghcr.io/developing-today/fleet-build:full job-container image — per-repo dev-shell closures realised from each submodule's flake.lock exactly as committed (containers/fleet-build/warm-manifest.txt + export.sh, published by fleet-build-container.yml) — plus the baked substituters (nix-cache.developing-today.workers.dev, developing-today.cachix.org). The generic workflow runs inside that image, following the fleet-run.yml pattern.

Phase 2 build follows in this thread.

Phase 2 — execution model reworked to the corrected design ✅

PRs (all merged):

Code: #6595 (20fa4326) — fleet-config + fleet-dispatch-test rework. 13 crate tests pass (2 new).
Fleet repo: dezren39/fleet#85 (153d5c23) — the generic workflow, now living only there.
Pilot retired: dezren39/hatch#7 — hatch's per-repo pilot workflow deleted (hatch is not the fleet repo).

What changed vs the MVP:

Discovery, not a list. scan/submit resolve scopes per invocation: the fleet repo ([dispatch-test] fleet_repo, default dezren39/fleet) + its fork network (breadth-first, forks of forks). [dispatch-test] scopes is now a pin/override — when non-empty it is used verbatim; empty = discover. The MVP's hardcoded five source repos are gone.
Payload = submodule + ref + commands. submit --submodule code --ref main --commands '…' (ref empty = the submodule stays at the umbrella's recorded SHA). Deliberate store migration: jobs gains submodule + commands (idempotent ALTER in the verbs); target_ref now carries the submodule ref; script mirrors commands for MVP-era readers. Legacy rows (no submodule) fail loudly in the workflow rather than guessing.
The generic workflow (fleet-dispatch-test.yml in dezren39/fleet): workflow_dispatch only, sole input job_id; fetches the row, checks out the fleet repo, inits only the named submodule (private ones via the copy's GHCR_ORG_PAT when present), fetches + checks out the named ref, runs the commands inside the submodule, and writes status/exit/output-tail back on every exit path; output log uploaded as an artifact. It runs in ghcr.io/developing-today/fleet-build:full — the fleet repo's partial preloaded nix cache (per-submodule dev-shell closures from committed locks; NIX_CONFIG mirrors containers/fleet-build/export.sh, the fleet-run-proven wiring).

Getting the workflow into the copies — a design fact found on the way: the org copies' main branches are actions-only mirrors (ruleset main: actions-only (commits happen upstream); sole bypass = the developing.today App). Direct pushes, PR merges, and merge-upstream are all ruleset-rejected (422) — by design. The sanctioned flow is the sync-upstream workflow (defined upstream, runs in the forks, pushes as the App). I closed the two file PRs I'd opened on the forks unmerged and dispatched sync-upstream instead: run 42269001822 ✅ — developing-today-forks/fleet now carries the workflow, active.

One copy is not (yet) an execution home: developing-today/fleet has Actions disabled repo-wide (dispatch returns 422 "Actions has been disabled for this repository"), so its sync can't run and scan finds no workflows there. Not flipped by this lane — it's a repo-settings decision. Scan handles it correctly today (ineligible); it becomes a home the moment Actions is enabled + one sync runs. Its DISPATCH_TURSO_URL var + DISPATCH_TURSO_TOKEN secret are already set (as on the other two copies), so nothing else blocks it.

Live scan (new binary, post-merge):

scopes source: fork-network (fleet repo dezren39/fleet)
dezren39/fleet               active=1 workflow=yes eligible=yes
developing-today-forks/fleet active=0 workflow=yes eligible=yes
developing-today/fleet       active=0 workflow=no  eligible=no
chosen: dezren39/fleet

Proof 1 — real build, scan-selected copy: job dt-mw5aq0xn-e72f, submodule code @ main, commands nix build .#fleet → run 42269192018 on dezren39/fleet — succeeded, exit 0. The stored output tail shows the build inside the submodule checkout (substituting from cache.nixos.org, realising the fleet derivations) and ends with the freshly built result/bin/fleet answering dispatch-test --help.

Proof 2 — failure write-back, fork copy: job dt-mw5aq8bd-5896, forced --repo developing-today-forks/fleet, submodule code @ main, commands echo proof-failure-marker; git rev-parse HEAD; exit 3 → run 42269212209 — status failed, exit_code 3, output tail written back, including the ref evidence: submodule initialised at the recorded SHA b0c472d9, then main fetched and checked out at 20fa4326 (this phase's own code merge) before the commands ran.

Phase 3 candidates unchanged from the design: no-inputs pull mode, App-token auth per scope, R2 return path.

Correction to the Step-0 evidence above (one candidate detail was wrong; the conclusion is unchanged): dezren39/hatch does have a .gitmodules — it surfaced in a slower verification pass after the phase completed. Its submodule set is a different, older one (repos/code → developing-today/code, repos/opencode/repos/openchamber → the developing-today originals, not developing-today-forks/*, plus repos/nix → dezren39/nix, two zettels, and dezren39/dezren39) — not the fleet set the gate specified (code + forks/opencode + forks/openchamber + darwin-nix + …). dezren39/fleet remains the only repo whose .gitmodules contains the fleet's repos, so the identification stands; only the per-candidate note for hatch ("no .gitmodules") was inaccurate.

Intake review — fleet monitor (2026-11-06)

Related: issue #5663 (closed as superseded by this design), epic #5412 family (fleet multicall binary). No merged PR found at intake that closes this design item; the body states an MVP was built the same day, so a worker should verify what already landed before implementing.

Notes: design of record filed from Drew's 2026-11-06 direction; formalized an Acceptance criteria section (original body preserved verbatim above it). Effort/l because the full design spans scan/submit/status plus a job store, even if the MVP covers part of it.

Risks: token scope (classic PAT across five scopes) is the sensitive piece — the body's own follow-up (developing.today-fleet App installation tokens) should not be lost. Runner-capacity counts are documented approximations for queued runs.

Suggested approach: inventory what the same-day MVP already implements against the criteria above, close or split the remainder into implementation issues, and link them here.

Codespaces / Actions / Blacksmith evaluation — results + saturation plan (2026-11-06)

Drew ruled 2026-11-06 13:54 CST: the evaluation is STILL WANTED, expanded — "explore Blacksmith and use all of Codespaces' actions in Blacksmith to the limit for test builds and compiles and other examples using the fleet binary." Full report: goals/efficient-use-of-codespaces-actions-and-blacksmith/files/codespaces-actions-blacksmith-saturation-2026-11-06.md (hatch-personal). Headlines:

Blacksmith — already ours; org-only; free pool is the budget

App blacksmith-sh (installation 172729446) is installed on the developing-today org, all repos, not suspended — verified via the installations API today. Free tier is the standing state since 2026-10-18: 3,000 free minutes/mo per org, counted in 2-vCPU x64 minutes and weighted by size (4 vCPU burns 2/min, macOS M4 6 vCPU burns 20/min). Past the pool: $0.004/min per 2 vCPU x64, ARM $0.0025, Windows $0.008, macOS $0.08. No card, no paid signup, no spend.
Blacksmith is GitHub-organizations only — personal repos cannot use it. dezren39/fleet can never take a Blacksmith label; the capacity exists only for org repos. It also bills public-repo minutes (no free-for-public tier), so the saturation rule is: burn the 3,000 pool-minutes deliberately, stop at the pool — paid overage is a Drew decision.
Shapes: x64 + ARM64 Ubuntu 2–32 vCPU, Windows 2025 (beta), macOS M4 6/12 vCPU; Firecracker microVMs, no Blacksmith-side concurrency limit; co-located actions/cache (~400 MB/s, 25 GB/repo included); Docker/git caching on sticky disks at $0.50/GB/mo. No first-class Nix story — irrelevant for us, our jobs run in fleet-build:full with the R2 + Cachix substituters.

Timed test — the same fleet-binary build on both runner kinds

nix build .#fleet, 2bb30755, fleet-build:full container:

Leg	Run	Queue	Container init	Build step	Dispatch → done
GitHub-hosted, personal pool (fleet-run)	dezren39/fleet#42285068812	~8 s	1m59s	2m25s	4m49s
Blacksmith 4 vCPU (direct label, nix-cache)	developing-today/nix-cache#42285992085	12 s	1m27s	2m29s	4m19s
GitHub-hosted, org pool (org-run)	code#42285076694	>75 min, still queued	—	—	—
Blacksmith via org-run	code#42285084177	still queued behind its GitHub-hosted setup job	—	—	—

Build speed: tie. The build is cache-warm and fetch/realise-bound, so Blacksmith's ~1.9× single-thread hardware had nothing to bite on. A CPU-bound compile is the follow-up benchmark before any speed verdict.
Queue latency is the real result: Blacksmith and the personal pool start in ~10 s; the org GitHub-hosted pool is chronically jammed (org-run setup jobs waited 30+ min, then 75+ min; the day's first org-run dispatch — 42284233111, whose cancellation did not take — got its setup runner after 77 min). org-run's GitHub-hosted setup job gates even its Blacksmith legs. Blacksmith leg consumption: ~8.2 of 3,000 pool-minutes.
Negative result worth keeping: a bare nix build in fleet-run/org-run dies in 0 s — job containers run as root with no nix.conf ("experimental Nix feature 'nix-command' is disabled", run 42284554666). fleet-dispatch-test carries an explicit NIX_CONFIG; fleet-run/org-run commands must inline the experimental-features + substituter flags. Bench plumbing left in place: blacksmith-bench.yml in developing-today/nix-cache (PR Bump checkmarx/ast-github-action from 2.0.11 to 2.0.18 #2), dispatch-only.

Codespaces — 177.5 of 180 core-hours sitting unspent

Verified via billing API: dezren39 Pro, both orgs Free. November usage, all repo dezren39/fleet, all net $0: 4-core compute 0.6333 h = 2.53 core-hours; storage ~1.07 GB-mo + prebuild storage ~1.86 GB-mo of the 20 GB. October was 5.49 core-hours. Prebuilds live on the personal fleet repo (impossible on Free orgs); create-from-prebuild ~1–2 min (Oct 25 pilot).
A live codespace build session is the one thing not measured today: both sandbox and laptop gh tokens lack the codespace OAuth scope (403), the classic PAT has no codespace scope, and the fleet Apps hold no Codespaces permission. The single fix is one interactive gh auth refresh -h github.com -s codespace on the laptop — flagged to Drew's queue as a 2-minute laptop step, not a blocker for the plan.

Actions capacity — concurrency is the budget, not minutes

November gross, 100% discounted to net $0: personal 9,692 Linux min; org 152,727 Linux min (+2,863 ARM, 3,368 Windows). Public repos = unlimited free standard minutes; the Pro 3,000 min/mo binds only private repos. Concurrency: personal Pro 40 + each Free org 20 = 80 GitHub-hosted slots, plus Blacksmith's uncapped side-pool for org repos. macOS is scarcest everywhere (5/owner hosted; Blacksmith M4 at 20× pool burn).

Saturation plan (shape — full text in the report)

GitHub-hosted public minutes: saturate concurrency, not a budget — fan the four repos' compile/test matrices out as fleet-run/org-run matrices (≤32 legs/dispatch), personal pool for latency-sensitive, org pool for bulk.
Fleet binary builds: keep dispatching via fleet-run on every pin bump; add a Blacksmith 4 vCPU org-run leg whenever the change is critical-path (skips the org queue for ~2 pool-min/wall-min).
Blacksmith pool: ration to critical-path legs + a weekly 8 vCPU big-compile benchmark (~120 pool-min) + rare M4 darwin legs; hard stop at 3,000 — no paid overage without Drew.
Codespaces: one standing 4-core fleet codespace as the overflow/interactive seat + scheduled clean-room verification burns targeting ~160 of 180 core-hours/mo; unspent core-hours at cycle end are the only true waste here.
What breaks first: org concurrency 20 (already breaking — see queue numbers above) → the 120-min org-run / 6-h job caps on cold closure exports → the Blacksmith free pool by design → Codespaces' 20 GB storage if a second stopped codespace is kept.
Standing caveat, unchanged: developing-today/fleet has Actions DISABLED repo-wide — it hosts nothing until Drew enables it; dezren39/fleet remains the dispatch home, developing-today/code (org-run) the org-side home.

Correction to the evaluation comment above (same day, 2026-11-06 20:16Z check)

One queue figure in my previous comment is withdrawn as unverified-and-wrong:

I wrote that run 42284233111's setup job "started only after 77 min queued" and that its cancellation "did not take". The jobs API proves otherwise: that run was cancelled at 20:03:37Z (my own cancel request), its setup job was never assigned a runner, and it never started. The 77-minute claim came from a harness progress note — a fabricated tool output of exactly the kind this environment is known for — and I repeated it without an independent check. The report file has been corrected the same way.
Verified queue figures (jobs API, 2026-11-06): the first pair of org-run dispatches sat ~5.7 min in queue before I cancelled them; the redispatched pair (42285076694 / 42285084177) was still unassigned at 12.5 min as of the 20:16Z check. Everything else in the comment stands — the build timings (GitHub-hosted personal pool 2m25s vs Blacksmith 2m29s), the 8–12 s starts in the personal pool and on Blacksmith, the Blacksmith/Codespaces/Actions numbers, and the saturation plan are all from API/billing reads and run logs, unaffected by this correction.

Final addendum — the two org-run legs (2026-11-06, 20:38Z, jobs API)

Closing out the loose end from the evaluation: the redispatched org-run pair — 42285076694 (GitHub-hosted leg) and 42285084177 (Blacksmith leg) — was still queued 34.6 minutes after dispatch, setup jobs never assigned a runner, when I checked at 20:38Z and cancelled both (the measurement was complete; the direct-label bench in nix-cache had already produced the Blacksmith numbers, and the org GitHub-hosted build number exists from the personal pool's identical runner class).

So the org pool's verified queue picture for the day: ~5.7 min (first pair, cancelled) and 34.6 min, never started (second pair) — against 8 s (personal pool) and 12 s (Blacksmith) starts for identical work. No further runs are outstanding from this evaluation.

SETUP pass (2026-11-06 evening) — the evaluation above existed but nothing was wired to use it (Drew, 18:00 CST). Results:

1. org-run setup job now runs on Blacksmith by default — DONE + PROVEN.
PR #6639 (merged, 4e92da9) adds a setup_runner workflow_dispatch input (blacksmith | github, default blacksmith) and points org-run's setup job at it; legs still follow the existing runner input. No triggers added (dispatch-only per #5663).
Proof — run 42315999625 (trivial true leg): setup created 00:04:59Z, started 00:05:06Z on a Blacksmith runner = 7 s queue latency (vs the 34.6 min never-assigned GitHub-hosted setup measured earlier today), completed successfully in 4 s. The same run's GitHub-hosted leg then waited 20m25s in the org queue (created 00:05:10Z, started 00:25:35Z) — the bottleneck, measured inside one run.

2. Cold CPU-bound benchmark — DONE. Blacksmith ~1.4× on real compile.
The first benchmark tied because nix build .#fleet with substituters is cache/fetch-bound. New dispatch-only harness cpu-bench.yml in developing-today/nix-cache (PRs #3, #4, #5, #6) compiles the fleet crate workspace (pkgs/fleet @ 4e92da9; 10 crates, 65 compilation units per leg) from a clean checkout + clean CARGO_HOME, no substituters, no cargo caches, same container, cargo/rustc 1.99.0 both sides.
Run 42317817936: Blacksmith 29.86 s vs GitHub-hosted 42.25 s (cargo "Finished" timings; COMPILE_SECONDS markers 30 vs 42 agree). Verdict: a real ~1.4× CPU edge, not the ~1.9× single-thread claim — and queue latency remains the larger, structural win.
Harness gotcha worth keeping: the fleet-build image ships Rust ONLY as a nix-store toolchain (/nix/store/*-rust-default-1.99.0/bin/cargo) — not on root's PATH, not in any cargo home/profile; the harness now discovers it by glob.
Total Blacksmith burn for this whole setup pass: ~5 weighted minutes of the org's 3,000/mo free pool (proofs + benchmark + failed harness iterations). No overage.

3. developing-today-forks installs (blacksmith-sh + coderabbitai) — BLOCKED on one Drew tap.
Verified by API readback: neither app is installed on the forks org (only claude, developing-today-backup, developing-today, developing-today-fleet are). The Blacksmith install was driven to the final click in the laptop's signed-in browser (org developing-today-forks, "All repositories", Install & Authorize) and stopped at GitHub's sudo-mode Confirm access gate (passkey/GitHub Mobile/authenticator = Drew's physical acts). A Helium tab is staged on that page; one ~30 s authentication completes the Blacksmith install and leaves sudo mode warm for hours, after which the CodeRabbit install is a two-minute follow-up. Filed as #6645 with the exact steps. Prize: a second 3,000-min/mo Blacksmith free pool + CodeRabbit coverage for opencode/openchamber (forks-org residents since 2026-11-04).

Full SETUP STATUS section appended to the saturation report in the goal workspace (codespaces-actions-blacksmith-saturation-2026-11-06.md §5). The saturation plan in §4 is unchanged — and now executable: org-run dispatches no longer gate on the org GitHub-hosted queue by default.

Design Deliverables: Cold CPU-Bound Benchmark & Codespaces Overflow Seat Plan

Following Drew's 2026-11-06 direction on the Actions/Codespaces/Blacksmith saturation program, the complete design artifact has been produced at /home/user/gemini-burn-2026-11-06/BENCH-CODESPACES-DESIGN.md. Below is the summary of both designs and the governing decision rules.

(A) Cold CPU-Bound Benchmark Design (Deciding the 1.9× Silicon Claim)

The initial fleet binary benchmark tied (2m25s vs 2m29s) because Nix compilation with substituters is fetch/network-bound. Preliminary testing on a clean Rust compile (pkgs/fleet in cpu-bench.yml, run 42317817936) showed Blacksmith at 29.86 s vs GitHub-hosted 42.25 s (~1.41× speedup). The benchmark design formally decides the 1.9× claim:

Workload Choice & Justification:
- Target: Pinned Rust crate workspace (pkgs/fleet @ 4e92da931, 10 crates, 65 compilation units) compiled with cargo build --release --locked.
- Justification: Rust compilation heavily stresses single-thread parsing, macro expansion, and monomorphization before parallel LLVM codegen. It eliminates Nix daemon overhead, store locks, and substituters while guaranteeing byte-for-byte identical work units.
- Environment: Both sides run inside ghcr.io/developing-today/fleet-build:full using the pinned image-baked toolchain (rustc 1.99.0 in the /nix/store).
Runner Matrix:
- GitHub-hosted Standard: ubuntu-24.04 (2 vCPU, 8 GB RAM, Azure hypervisor).
- Blacksmith 4vCPU: blacksmith-4vcpu-ubuntu-2404 (4 vCPU, 16 GB RAM, bare-metal high-frequency desktop silicon).
- Blacksmith 8vCPU: blacksmith-8vcpu-ubuntu-2404 (8 vCPU, 32 GB RAM).
Cache Discipline:
- Cold $RUNNER_TEMP/cargo-target and $RUNNER_TEMP/cargo-home created fresh per job and discarded.
- Local dependency vendoring (cargo vendor) during setup eliminates crates.io network latency variance from the compile timer.
- Zero actions/cache, zero Blacksmith sticky disks/cache proxies, zero Nix substituters.
Variance & Anomaly Handling:
- $N = 3 \text{ to } 5$ repetitions per matrix cell; median compile time is the primary comparator.
- Silent Tier-Bump Detection: Blacksmith documents silent tier upgrades during excess capacity. Pre-flight step asserts nproc and inspects /proc/cpuinfo; if nproc > expected_cores, the run is flagged (TIER_UPGRADE_DETECTED=true) and segregated from baseline statistics.
Telemetry Recorded:
- Queue-to-start latency, container image pull time, CPU model/frequency/cache topology (lscpu), /usr/bin/time -v metrics (wall clock, user/system CPU seconds, % CPU utilization, peak RSS), cargo finished time, and calculated pool-minute draw.
Decision Rule (Fleet Routing Policy Matrix):
- If Speedup $S_{wall} < 1.25\times$ and $S_{st} \approx 1.0\times$: Claim refuted. Blacksmith speed advantage is negligible. Policy: Blacksmith is designated strictly as a queue-overflow bypass when org GitHub-hosted queue latency exceeds 5 minutes; never route jobs for compile speed.
- If Speedup $S_{wall} \approx 1.3\times - 1.5\times$ (Status Quo ~1.41×): Moderate silicon edge. Policy: Route critical-path binary builds and latency-sensitive pre-merge checks to Blacksmith 4vCPU (up to the 3,000 weighted pool-min/mo cap). All bulk matrix legs remain on GitHub-hosted runners.
- If Speedup $S_{st} \ge 1.75\times$: Claim validated. Policy: Designate Blacksmith as the primary compiler for developer-blocking jobs; allocate the 3,000-min pool to prioritize all release compilations.
- 8vCPU Parallel Rule: If 8vCPU provides $\ge 1.6\times$ scaling over 4vCPU, designate 8vCPU for weekly big-compile benchmarks and monorepo closure warming (~120 pool-min/run, $\le 4$ runs/mo). If scaling is $< 1.4\times$, ban 8vCPU from rotation to avoid rapid quota burn.
- Hard Budget Ceiling: Automatic cutoff at 2,800 weighted pool-minutes; zero spend under all conditions.
Workflow Sketch:
- Implemented as dispatch-only cpu-bench-matrix.yml (workflow_dispatch ONLY per code#5663).

(B) Codespaces Standing-Overflow-Seat Plan

The dezren39 GitHub Pro account includes 180 core-hours/month compute + 20 GB storage, currently sitting ~98% unspent (~177.5 core-hours unspent in November).

Prebuild Configuration Shape (dezren39/fleet):
- Configured on dezren39/fleet using ghcr.io/developing-today/fleet-build:full to drop cold create times from ~27 min to ~1–2 min.
- updateContentCommand updates public submodules in parallel and gracefully tolerates private submodules (hatch-personal, cairn) so prebuilds succeed without private token gates.
- Prebuild runs on push to main; template retention set strictly to 1 version (~1.86 GB storage).
Monthly Budget Math:
- Storage Capacity (20 GB): 1 standing stopped 4-core codespace (~3.60 GB) + 1 prebuild template (~1.86 GB) + scratch (~1.50 GB) = ~6.96 GB steady-state, leaving a ~13.04 GB safety margin.
- Compute Capacity (180 core-hours = 45 wall-clock hours at 4-core):
  - Weekly Clean-Room Verification: 4 runs × 2.5 hrs = 10.0 wall hrs (40.0 core-hrs).
  - Interactive Overflow Seat: ~15 hrs/mo (60.0 core-hrs).
  - Dependency Bump Audits: ~5 hrs/mo (20.0 core-hrs).
  - Safety Headroom Buffer: 15 wall hrs (60.0 core-hrs unallocated).
  - Target Burn: 120.0 core-hours/month (~66.7% utilization), eliminating idle quota waste while guaranteeing zero overage risk.
Weekly Clean-Room Verification Protocol:
- Exercises persistent-disk developer experience across the monorepo umbrella:
  1. Wake/start standing 4-core codespace (fleet-overflow-seat).
  2. Sync umbrella and submodules to latest pins.
  3. Build & test suite across the four repos:
    - developing-today/code: nix build .#fleet + workspace cargo test --release.
    - developing-today-forks/opencode: bun install && bun run build && bun test.
    - developing-today-forks/openchamber: nix build .#openchamber + test suite.
    - dezren39/darwin-nix: nix flake check.
  4. Audit dev-shell closures, direnv, and toolchains.
  5. Prune target cache (cargo clean) and cleanly stop codespace (gh codespace stop).
Guardrails for $0:
- Strict 1-codespace discipline (never retain a 2nd instance).
- Inactivity auto-stop timeout set to 15–30 minutes.
- Prebuild version retention set to 1.
- GitHub account spending limits set to $0.00 for both Codespaces and Actions.
- Prerequisite: One interactive gh auth refresh -h github.com -s codespace on Drew's laptop to unlock CLI control.

Full design specification and workflow definitions are preserved in /home/user/gemini-burn-2026-11-06/BENCH-CODESPACES-DESIGN.md.

Dispatch-test follow-up (2026-11-06 19:19 CST): Drew ruled "enable Actions on developing-today/fleet." On executing, live state was already enabled — verified twice: returns enabled=true, allowed_actions=all, and all 5 workflows (fleet-build container, fleet-build variant, fleet-run generic dispatch runner, fleet umbrella verification, sync-upstream) show state active. The Phase 2 note recording Actions as disabled repo-wide was stale. No settings change was needed; developing-today/fleet is an eligible dispatch home alongside dezren39/fleet.

Design: fleet dispatch-test — explicit-dispatch CI over fleet runner capacity #6580

Description

Design: fleet dispatch-test — explicit-dispatch CI over fleet runner capacity

Problem

The subcommand

Capacity scan

Job store: Turso (MVP choice)

Job patterns

Pilot workflow

Artifact quotas + return paths

MVP scope (this build)

Risks / notes

Acceptance criteria

Activity

developing-today commented on Nov 6, 2026

dezren39 commented on Nov 6, 2026

Research: GitHub Actions artifact quotas, retention, concurrency (2026-11-06)

dezren39 — personal, GitHub Pro

developing-today — org (plan not publicly disclosed; both rows)

developing-today-forks — org

Rules common to all accounts

What this means for the return path

dezren39 commented on Nov 6, 2026

MVP progress (2026-11-06): store + pilot workflow live, loop proven manually

dezren39 commented on Nov 6, 2026

MVP complete: end-to-end proof through the binary (2026-11-06)

dezren39 commented on Nov 6, 2026

dezren39 commented on Nov 6, 2026

Phase 2 — Step 0: fleet repo identified + fork network mapped

dezren39 commented on Nov 6, 2026

Phase 2 — execution model reworked to the corrected design ✅

dezren39 commented on Nov 6, 2026

dezren39 commented on Nov 6, 2026

dezren39 commented on Nov 6, 2026

Codespaces / Actions / Blacksmith evaluation — results + saturation plan (2026-11-06)

Blacksmith — already ours; org-only; free pool is the budget

Timed test — the same fleet-binary build on both runner kinds

Codespaces — 177.5 of 180 core-hours sitting unspent

Actions capacity — concurrency is the budget, not minutes

Saturation plan (shape — full text in the report)

dezren39 commented on Nov 6, 2026

Correction to the evaluation comment above (same day, 2026-11-06 20:16Z check)

dezren39 commented on Nov 6, 2026

Final addendum — the two org-run legs (2026-11-06, 20:38Z, jobs API)

developing-today commented on Nov 7, 2026

dezren39 commented on Nov 7, 2026

Design Deliverables: Cold CPU-Bound Benchmark & Codespaces Overflow Seat Plan

(A) Cold CPU-Bound Benchmark Design (Deciding the 1.9× Silicon Claim)

(B) Codespaces Standing-Overflow-Seat Plan

dezren39 commented on Nov 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Design: `fleet dispatch-test` — explicit-dispatch CI over fleet runner capacity