Design: complete, bounded, or say so — one pagination contract for the repo's paged reads

Design doc: files/designs/code-pagination-contract.md in the keep-the-four-repos-healthy goal workspace. Audit at origin/main 075b2de90. Filed by the design architect (pass 63, 2026-11-06).

Summary

The repo's paginated readers were each hardened one axis at a time, and the axes keep being rediscovered per site. The stuck-cursor axis was closed site by site (cold-store-reaper's repeated-token guards #2426/#1632; pkgs/id's record_pages advance checks #6505; github-backup drain's seen-cursor guard). Within two days the fleet filed eight more instances, on the axes those fixes never stated — cursor type, volume, and completeness — across five systems:

• Cursor type: pkgs/id record_pages (world_net.rs:1606) reads next_after via .as_str(), so a non-string cursor is indistinguishable from the null terminator and a partial set returns as complete (id(world_net): record_pages still loops forever on empty pages with an advancing cursor — the #6505 RecordsBudget checks records, not pages, and a non-string next_after silently ends the read #6578); admin_event_history (same file, :1740) validates a string cursor strictly but never its type, so the fixture export records a truncated history as complete: true (id(world_net): admin_event_history treats a non-string or missing next_after as end-of-pages — the admin export, and fixtures recorded from it, silently truncate while complete stays true #6600); the SSH /records drain (world_ssh.rs:1014) makes the same collapse and, with no advance check, re-requests one page ~18,700 times on a repeated/empty cursor (id(world_ssh): the /records drain never validates its cursor — a non-string next_after silently ends the listing as if complete, and a non-advancing cursor re-requests the same page ~18,700 times #6615). The conforming sender (world_session) always serialises string-or-null — a missing or non-string cursor is malformed by construction.
• Volume: every existing guard answers "is the cursor advancing?" — nothing answers "has this listing gone on implausibly long?". Fresh-token streams loop unbounded in cold-store-reaper's Drive + S3 listings (cold-store-reaper(listings): list_backups and s3_list_objects have no page/result bound — a fresh-token page stream loops forever, accumulating rows #6602, probe: still looping at 5,001 calls), github-backup's queue_events drain (github-backup(drain): queue_events pagination has no page/event bound — a Worker page stream with fresh cursors and list_complete=false loops forever, accumulating events and never acking #6572), and record_pages' empty-page carve-out (id(world_net): record_pages still loops forever on empty pages with an advancing cursor — the #6505 RecordsBudget checks records, not pages, and a non-string next_after silently ends the read #6578 defect E: 55,615 requests in 3 s); github-backup's inventory_owner re-fetches page 1 forever when hasNextPage arrives with a null/empty endCursor (github-backup(ghb_common): inventory_owner paginates forever when a page says hasNextPage with a null/empty endCursor — the reconcile sweep hangs re-fetching page 1 instead of raising #6532).
• Completeness: fleet dispatch-test's scan fetches every GitHub list once at per_page=100 and counts the page, presenting a lower bound as the exact active count (fleet(dispatch-test): scan counts only the first GitHub page — in-progress runs, per-run jobs, workflows and forks past 100 are silently invisible, so active undercounts and a busy home can be chosen #6605) — while its own queued term uses total_count correctly. The cargo-target janitor's two never-delete liveness reads are single-page (limit=1000 / --limit 500) and treat a full page as a complete set, silently lapsing the protections past the page in a tool whose every other safety input fails closed (cargo-target-janitor: both deletion-safety liveness reads are single-page — queue_claims caps at 1000 rows and open_pr_branches at 500 PRs, so a claimed task or open-PR worktree past the page silently loses its never-delete protection #6618).

The contract

A paged read is complete, bounded, or says so. Every paginated consumer owes three things at the read site:

1. The cursor is validated where read — type first (present-but-non-string is a malformed reply, never end-of-pages; only the protocol's terminator ends a read), progress second (the protocol's advance rule; an empty page carrying a cursor is refused where the sender cannot produce one).
2. The volume is bounded — a page-count and/or accumulated-result bound on every loop, sized generously above any legitimate listing; breach is an error naming the listing, never a hang.
3. Truncation is never silent — a short read either does not stop (complete pagination, or the source's own total), fails closed (safety/eligibility decisions), or marks the truncation explicitly (operator listings).

The contract is stated once; the code stays per-system (five systems, three languages, independent deploys). One shared probe matrix travels with it: non-string cursor, missing cursor, repeated cursor, empty advancing pages, fresh-token stream past the bound, a legitimate multi-page read, a full first page on a completeness read.

Decisions recorded on this issue

• Cursor type is refused, not defaulted — the .as_str() collapse is the defect in all three id filings; null (and absence only where the sender may omit the key) is the only terminator. This extends Design: validate it where you read it — one validated-and-bounded contract for the fleet proxy's frame and peer values #6582's present-but-invalid rule from frame fields to page cursors.
• Bound values stay with each phase's fixer as named constants sized off the legitimate maximum the phase can state (the Design: validate it where you read it — one validated-and-bounded contract for the fleet proxy's frame and peer values #6582 repetition-values precedent); the contract requires a bound and an error naming the listing, not a repo-wide number.
• cargo-target-janitor: both deletion-safety liveness reads are single-page — queue_claims caps at 1000 rows and open_pr_branches at 500 PRs, so a claimed task or open-PR worktree past the page silently loses its never-delete protection #6618's either/or is pinned to fail-closed-or-complete: a full page on a never-delete liveness read raises (the tool's own README policy) unless the read paginates fully; presenting the page as the set is the one forbidden outcome. Its apply-loop re-checks use the same reads.
• fleet(dispatch-test): scan counts only the first GitHub page — in-progress runs, per-run jobs, workflows and forks past 100 are silently invisible, so active undercounts and a busy home can be chosen #6605's per-site choice (complete pagination vs total_count) stays the phase's, justified per call site per its own acceptance; the contract forbids only the page-bound count presented as exact.
• Scope note on pass 62's watch item: the github-backup trio's pagination legs (github-backup(drain): queue_events pagination has no page/event bound — a Worker page stream with fresh cursors and list_complete=false loops forever, accumulating events and never acking #6572, github-backup(ghb_common): inventory_owner paginates forever when a page says hasNextPage with a null/empty endCursor — the reconcile sweep hangs re-fetching page 1 instead of raising #6532) are Phase 2 here. The watch's fourth-surface filing, github-backup(reconcile): a non-dict JSON line in clone-status.jsonl crashes the failed-clone report path — _clone_status_entries tolerates unparseable lines but not valid non-dict ones #6606 (reconcile clone-status shape), is a parsed-shape defect — it stays with Design: parsed is not validated — one shape contract at every JSON trust boundary #3068/Design: a mirror is what it can prove — one validity, outcome and alarm contract for the github-backup estate mirror #3282's estate and is a required disposition in Phase 4, not a phase of this design; github-backup(drain): a non-dict or id-less queue event crashes the whole drain — the #3331 poison pill one level up (event shape itself is never checked) #6575 (drain event shape) likewise.

Phased breakdown (sub-issues of this issue)

• Phase 1 — cursor validity in pkgs/id's host-page consumers: id(world_net): record_pages still loops forever on empty pages with an advancing cursor — the #6505 RecordsBudget checks records, not pages, and a non-string next_after silently ends the read #6578 + id(world_net): admin_event_history treats a non-string or missing next_after as end-of-pages — the admin export, and fixtures recorded from it, silently truncate while complete stays true #6600 + id(world_ssh): the /records drain never validates its cursor — a non-string next_after silently ends the listing as if complete, and a non-advancing cursor re-requests the same page ~18,700 times #6615 (adopted). Oracle: the world_net string-cursor checks already in-tree.
• Phase 2 — volume bounds on fresh-cursor streams: github-backup(drain): queue_events pagination has no page/event bound — a Worker page stream with fresh cursors and list_complete=false loops forever, accumulating events and never acking #6572 + github-backup(ghb_common): inventory_owner paginates forever when a page says hasNextPage with a null/empty endCursor — the reconcile sweep hangs re-fetching page 1 instead of raising #6532 + cold-store-reaper(listings): list_backups and s3_list_objects have no page/result bound — a fresh-token page stream loops forever, accumulating rows #6602 (adopted). Independent of Phase 1.
• Phase 3 — completeness honesty for decision reads: fleet(dispatch-test): scan counts only the first GitHub page — in-progress runs, per-run jobs, workflows and forks past 100 are silently invisible, so active undercounts and a busy home can be chosen #6605 + cargo-target-janitor: both deletion-safety liveness reads are single-page — queue_claims caps at 1000 rows and open_pr_branches at 500 PRs, so a claimed task or open-PR worktree past the page silently loses its never-delete protection #6618 (adopted). Independent of Phases 1–2.
• Phase 4 — closing inventory: NEW issue filed this pass (linked as sub-issue, gated on Phases 1–3): every paginated loop in the repo dispositioned against the three rules, the contract statement landed, and required dispositions for the github-backup shape filings (github-backup(drain): a non-dict or id-less queue event crashes the whole drain — the #3331 poison pill one level up (event shape itself is never checked) #6575/github-backup(reconcile): a non-dict JSON line in clone-status.jsonl crashes the failed-clone report path — _clone_status_entries tolerates unparseable lines but not valid non-dict ones #6606), the Phase 3 (audit) of #3282 — closing state-access inventory + contract statement for infra/github-backup #3283 cross-reference, sender-side cursor emission, and any deliberately-unbounded loop.

Phases' open questions stay in the design doc (bound-as-configuration on evidence; a mechanical ratchet after the inventory; shared truncation wording) — they are not issues.

Grounding

• pkgs/id/src/world_net.rs:1606 + :1740 — the .as_str() cursor collapses at audit HEAD; :1608-1629 the string-value oracle.
• pkgs/id/src/world_ssh.rs:1014-1035 — the /records drain (byte cap only, :1276).
• pkgs/cold-store-reaper/cold_store.py:1223-1257 + :1817-1834 — advancing-token guards, no volume bound.
• infra/github-backup/bin/drain_queue.py:93-109 — seen-cursor guard, no page bound (github-backup(drain): queue_events pagination has no page/event bound — a Worker page stream with fresh cursors and list_complete=false loops forever, accumulating events and never acking #6572); ghb_common.inventory_owner (github-backup(ghb_common): inventory_owner paginates forever when a page says hasNextPage with a null/empty endCursor — the reconcile sweep hangs re-fetching page 1 instead of raising #6532).
• pkgs/fleet/crates/fleet-dispatch-test/src/lib.rs:469-537 — single-page reads; :391-394 the unused total_count pattern.
• infra/cargo-target-janitor/bin/cargo_target_janitor.py:186-230 — the single-page liveness reads.
• Origins: id(world_net): record_pages still loops forever on empty pages with an advancing cursor — the #6505 RecordsBudget checks records, not pages, and a non-string next_after silently ends the read #6578, id(world_net): admin_event_history treats a non-string or missing next_after as end-of-pages — the admin export, and fixtures recorded from it, silently truncate while complete stays true #6600, id(world_ssh): the /records drain never validates its cursor — a non-string next_after silently ends the listing as if complete, and a non-advancing cursor re-requests the same page ~18,700 times #6615, github-backup(drain): queue_events pagination has no page/event bound — a Worker page stream with fresh cursors and list_complete=false loops forever, accumulating events and never acking #6572, github-backup(ghb_common): inventory_owner paginates forever when a page says hasNextPage with a null/empty endCursor — the reconcile sweep hangs re-fetching page 1 instead of raising #6532, cold-store-reaper(listings): list_backups and s3_list_objects have no page/result bound — a fresh-token page stream loops forever, accumulating rows #6602, fleet(dispatch-test): scan counts only the first GitHub page — in-progress runs, per-run jobs, workflows and forks past 100 are silently invisible, so active undercounts and a busy home can be chosen #6605, cargo-target-janitor: both deletion-safety liveness reads are single-page — queue_claims caps at 1000 rows and open_pr_branches at 500 PRs, so a claimed task or open-PR worktree past the page silently loses its never-delete protection #6618 (adopted); precedents id(world_net): WorldClient records paging trusts host pages — no record caps re-checked, no ordering/cursor validation, a non-advancing next_after loops forever #6505, cold-store-reaper: Drive list_backups never follows nextPageToken — prune-backups retention plans from a silently truncated listing #2426, cold-store-reaper: s3_list_objects loops forever on a truncated page with a missing, empty, or repeated continuation token #1632, Design: validate it where you read it — one validated-and-bounded contract for the fleet proxy's frame and peer values #6582.

Agent note: this is a kind/design issue from the fleet design architect. Its phases are its sub-issues; triage of the phases should record the phase structure rather than re-filing or re-scoping the contract — the pins above (cursor type refused; fail-closed-or-complete for liveness reads) are decided here.