Skip to content

fix(agent-work): preserve stale-AgentWork rows still owed a claim release (#1350) - #1389

Merged
joryirving merged 9 commits into
mainfrom
courier/misospace/dispatch/issue-1350
Oct 24, 2026
Merged

joryirving merged 9 commits into
mainfrom
courier/misospace/dispatch/issue-1350

Conversation

@itsmiso-ai

@itsmiso-ai itsmiso-ai commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Closes #1350

Summary

The pre-claim cleanup findAndReleaseStaleAgentWorkForIssue (called from src/app/api/issues/claim/route.ts) hard-deleted CLAIMED/IN_PROGRESS AgentWork rows whose agent has no active lease, with no awareness of the sweep's pending-release marker. A row the stale-work sweep had marked and left pending its claim release could therefore be hard-deleted by a claim attempt, dropping the release the sweep decided on.

Change

Mirrors the sweep's canonical pending boundary (activeStateFamilies in src/lib/stale-work.ts, #1316/#1334) into the claim-path cleanup:

  • Scan (src/lib/agent-work.ts): the candidate findMany now carries staleClaimReleasePending: false alongside the state filter, so a row still owed a claim release is never a cleanup candidate in any shape (a marked STALE row is already excluded by the state filter; the conjunct additionally refuses an active-shaped row carrying the marker).
  • Fenced delete: each delete is re-checked on the same conjuncts the scan matched (per-row CAS, mirroring the sweep's own practice) — the executing state family, the pending-release boundary, and the scanned ownership fields (issueId, agentName). A row that changed shape between the scan and the delete — a writer outside the per-issue mutation guard, e.g. the operator agent-work routes (release / release-all / reassign), the operator unclaim path and the worker agent-work start / checkpoint / finish paths — is left for whoever now owns that shape.
  • Honest accounting: deleted ids are collected only when the fence consumed exactly 1 row; the returned count and the stale_agentwork_cleanup audit notes report exactly the rows deleted, and no audit row is written when the fence consumed nothing (heightens the existing "a deletion is never left without its audit row" posture — and never claims a deletion that did not happen).

Scope note: park/BLOCKED preservation and lease handling were already covered by prior work (#1316, #1334) and are unchanged. The claim route is this function's only production caller and runs the cleanup inside the per-issue IssueMutationGuard; the concurrent writers it races with (operator agent-work routes, operator unclaim path, worker start/checkpoint/finish) do not take that guard, so the fence — not the guard — is the load-bearing protection.

Addressed review feedback

The AI review on d1c0a62 requested changes with one Major finding: the delete fence omitted the scanned ownership fields, so a row reassigned between the scan and the delete could still be deleted even though its owner changed. Fixed in 75351d4:

  • The delete fence now carries the scanned ownership fields (issueId, agentName) in addition to the state family and the pending boundary, so it matches the scan on every conjunct that identifies the row, and a reassigned row is left to its new owner.
  • A separate reviewer session (same model family — MiniMax-M3.1-Flash — so no provider diversity was available) then raised three items, all fixed in the same commit:
    1. minor — the state conjunct of the fence had no behavioral coverage: added a test whose concurrent write settles the row to RELEASED (the operator release route's shape) and asserts the fence consumes nothing and the row survives.
    2. nit — the scan's issueId conjunct was unverified: the test fake now honours where.issueId, and a new test pins that a row belonging to another issue is never a candidate.
    3. nit — the comment credited the per-issue mutation guard for the fence's safety, but the operator agent-work routes do not take that guard: the comment now states that the fence itself is the load-bearing protection.

A fresh reviewer session on the diff at 75351d4 (same model family — MiniMax-M3.1-Flash — so no provider diversity was available) raised three further items, all fixed in b7d6bd9:

  1. minor — the doc comment claimed "Live writers never leave the marker on an active-shaped row (the sweep pairs it with the STALE mark), so the boundary is defence in depth on top of the state filter". That is false: the operator agent-work reassign route rewrites a row to state: "CLAIMED" without clearing staleClaimReleasePending, so a sweep-marked STALE row can arrive at the claim-path cleanup as an active-shaped row that still owes its release — and the state filter alone would not keep it. The marker conjunct is load-bearing, not redundant; the comment now says so.
  2. nit — the same comment referred to "the other callers" of the lib function. There is exactly one production caller (src/app/api/issues/claim/route.ts), and it does take the per-issue mutation guard; the parties that do not take the guard are the concurrent writers this cleanup races with. The comment now names them correctly.
  3. nit — the accounting claim ("the audit names exactly the rows the fence consumed") was pinned only in the all-consumed case, so a regression back to reporting the scanned ids would have passed. A mixed-case test was added: with one candidate consumed and one refused by the fence, the returned count is 1 and the audit's id list names the consumed row only. Verified non-vacuous — reverting the notes to the scanned ids fails that test on its own.

A third reviewer session on the diff at b7d6bd9 (same model family — MiniMax-M3.1-Flash — so no provider diversity was available) raised five further items. Four are fixed in 496abdc; the fifth is answered on the PR thread:

  1. minor — the fence comment named "the self-heal routes", a route family that does not exist in this repository, and omitted the writers that actually race this cleanup: the operator agent-work routes (release / release-all / reassign) and the worker agent-work start / checkpoint / finish paths, none of which take the per-issue mutation guard. The comment now names them, and records that the stale-work sweep — the only writer that sets the pending marker — does take the guard, so the reachable pending shape is one that arrives pending.
  2. minor — the header comment claimed that a row still owed a claim release "belongs to the stale-work sweep's retry arm". That holds for a STALE pending row (the retry arm retires it) but is false for an active-shaped pending row: the retry arm requires state: "STALE", so such a row belongs to no sweep arm at all. The comment now distinguishes the two shapes, and states the premise the "no path drops the owed release" claim rests on — this cleanup is the repository's only agentWork.deleteMany (the sweep only marks rows STALE).
  3. minor — the "a deletion is never left without its audit row" invariant was not pinned: the in-memory fake's $transaction has no rollback semantics, so moving the deletes out of the transaction (or writing the audit after the commit) kept the suite green. A new test pins the transaction's shape — the delete and the audit row go through the same transaction client, the audit write follows the delete, and a failing audit write surfaces instead of the cleanup reporting a successful deletion. The real rollback needs a live PostgreSQL and is not covered by CI's Database integration job (a fixed file list that excludes this path); it is filed as test(agent-work): prove the claim-path cleanup's delete+audit rollback against a real PostgreSQL #1392.
  4. nit — the delete's issueId conjunct was not exercised by any test (no scenario moved a row between issues), so dropping it passed the suite — the mutation-coverage claim below was overstated for that one conjunct. A new test moves a row to another issue between the scan and the delete and pins that the fence refuses it; removing the conjunct now fails the suite.
  5. nit — the per-row CAS turns one bulk delete into N statements inside the interactive transaction (5s default timeout). Not changed: a bulk delete returns a count, not ids, so the per-row CAS is what lets the audit name exactly the rows the fence consumed, and N is bounded by the issue's leftover rows. Answered on the PR thread.

A fourth reviewer session on the diff at 496abdc (same model family — MiniMax-M3.1-Flash — so no provider diversity was available) returned SHIP with no blocker or major finding, and re-confirmed the three load-bearing claims (pending rows are never consumed; the fence protects the delete; the accounting is honest) and that every scan and fence conjunct has behavioural coverage. It raised two documentation-accuracy items, both fixed in 699d00d:

  1. nit — the fence comment enumerated the writers it races with but omitted one: the operator unclaim path calls releaseAgentWorkByAgentAndIssue (src/lib/lease.ts) after its guarded block, so it writes state: "RELEASED" on AgentWork rows without the per-issue mutation guard — the same unguarded shape as the operator agent-work routes. The comment now names it, so the fence's coverage of that shape is not an accident.
  2. nit — the comment said the stale-work sweep "cannot interleave with the claim path"; a maintainer force-claim can override that guard (src/app/api/issues/claim/route.ts), so the claim now reads "normally", and the comment records that the fence's pending conjunct refuses a row that became pending after the scan either way.

The same session flagged that docs/ownership-reconciliation.md still described the cleanup as deleting every AgentWork row with no matching active lease (no pending boundary) and pointed at a stale line number. Fixed in 699d00d: the doc now states the pending-release boundary and the fence, and the reference is corrected.

Validation (local, on commit 699d00d)

  • npx vitest run src/lib/agent-work.test.ts src/app/api/issues/claim/route.test.ts src/lib/stale-work.test.ts src/app/api/agent-work/route.test.ts — 4 files passed, 227 passed
  • npm run test (full suite) — 206 files passed / 5 skipped; 5325 passed, 0 failed, 27 skipped, 1 todo (147s)
  • npm run lint — 0 errors, 1 pre-existing warning in src/app/login/page.tsx (unrelated, unchanged since Fix: eliminate silently swallowed errors via .catch(() => {}) #744)
  • npm run typecheck — clean
  • Fence mutation check (re-run independently on the final head): removing each delete-fence conjunct in turn (state, agentName, issueId, staleClaimReleasePending) makes src/lib/agent-work.test.ts fail with assertion failures, not crashes — baseline 69 passed, failures 4 / 3 / 4 / 3 respectively
  • Accounting mutation check: the audit notes reporting the scanned ids → 1 failure; dropping the "no audit row when the fence consumed nothing" guard → 4 failures; the returned count regressed to the scanned count → 5 failures

Limitation: this workspace has no Postgres, so the DB-backed integration path (real generated SQL, the AgentWorkHistory cascade, transaction commit/rollback) was not exercised locally. CI's Database integration job covers the schema and its own integration files, but not this cleanup's transaction — that gap is #1392.

Remaining external verification

Courier added 3 commits October 23, 2026 23:12
npm audit --omit=dev (blocking CI gate) failed on main and on this PR
head: body-parser 2.3.0 was high (GHSA-27r3-jxmh-p828, DoS via trailing
bytes after a compressed body), hono 4.13.7 and ip-address 10.7.2 were
moderate.

npm audit fix --package-lock-only --include=prod bumps body-parser to
2.4.0, hono to 4.13.13, ip-address to 10.7.4 and qs to 6.16.1; npm also
re-resolved the dev tree (vite, rolldown, @oxc-project/types and
@rolldown/* bindings) inside the same existing ranges. package.json and
every range are unchanged; package-lock.json was already identical to
main before this commit, so the failure pre-dates this PR.

Local validation at this change state: npm ci --include=dev, npm run
audit (0 vulnerabilities), typecheck, lint (1 pre-existing warning) and
vitest all exit 0.
its-saffron[bot]

This comment was marked as outdated.

…1350)

The claim-path cleanup's fenced delete re-checked the state family and the
pending-release boundary, but not the ownership fields the scan matched
(`issueId`, `agentName`). A row reassigned between the scan and the delete
kept its id, state and marker, so the fence consumed it and the newly owned
row was deleted instead of being left to its new owner.

The delete now carries the scanned ownership fields as well, so the fence
matches the scan on every conjunct that identifies the row. Tests cover the
reassignment race, the concurrent state settle (operator release), the
scan's issue scope, and the pending boundary; each fence conjunct is
mutation-covered.

Also correct the comment that credited the per-issue mutation guard for the
fence's safety: the operator agent-work routes do not take that guard, so
the fence itself is the load-bearing protection.
@itsmiso-ai

Copy link
Copy Markdown
Contributor Author

Addressed the review feedback on d1c0a62 in 75351d4 (pushed to this branch).

Major — the delete fence omitted the scanned ownership fields. The fence re-checked id, state and staleClaimReleasePending, but not the agentName the scan matched, so a row reassigned between the scan and the delete kept the other predicates and was deleted out from under its new owner. The delete where in src/lib/agent-work.ts now carries the scanned ownership fields (issueId, agentName) alongside the state family and the pending boundary, so it matches the scan on every conjunct that identifies the row.

The claims that were reported as holding were left as they were and re-verified: the scan still excludes both pending shapes (staleClaimReleasePending: false covers the STALE and the active-shaped pending row), and the accounting is still honest (only consumed ids are returned and audited, and a zero-consumption fence writes no audit row).

Added coverage for the race the finding describes — a concurrent reassignment between the scan and the delete leaves the row in place, and the delete is pinned to carry the scanned owner. Two further gaps a reviewer session found in the same code are closed in the same commit: the fence's state conjunct now has a behavioral test (a concurrent settle to RELEASED — the operator release route's shape — is refused), and the scan's issueId conjunct is verified by a test plus a test fake that now honours where.issueId. A comment that credited the per-issue mutation guard for the fence's safety was corrected: the operator agent-work routes do not take that guard, so the fence itself is the load-bearing protection.

Local validation on 75351d4: npx vitest run src/lib/agent-work.test.ts (66 passed), npx vitest run src/lib/agent-work.test.ts src/app/api/issues/claim/route.test.ts src/lib/stale-work.test.ts (187 passed), npm run lint (0 errors, 1 pre-existing unrelated warning in src/app/login/page.tsx), npm run typecheck (clean), npm run test (5322 passed, 0 failed, 27 skipped, 1 todo).

The npm audit check failure on this PR is pre-existing and unrelated to this diff — it reproduces on main and is tracked in #1386. CI is left pending for operator verification; no merge is performed by this run.

its-saffron[bot]

This comment was marked as outdated.

@its-saffron
its-saffron Bot dismissed their stale review October 24, 2026 00:26

Superseded by a newer automated review for this pull request.

@itsmiso-ai

Copy link
Copy Markdown
Contributor Author

Takeover handoff — head 75351d4 re-verified; no code change was required.

State assessment

Independent local validation (re-run by this takeover, not inherited; workspace /workspace, revision 75351d4, deps installed with npm ci)

Check Command Result
Lint npm run lint 0 errors, 1 warning (src/app/login/page.tsx:39 — pre-existing, unrelated)
Typecheck npm run typecheck clean
Targeted suites npx vitest run src/lib/agent-work.test.ts src/app/api/issues/claim/route.test.ts src/lib/stale-work.test.ts 3 files passed, 187 tests passed
Full suite npm run test 206 files passed / 5 skipped; 5322 passed, 0 failed, 27 skipped, 1 todo (149.5s)
Fence mutation check removed each delete-fence conjunct in turn and re-ran src/lib/agent-work.test.ts baseline 66 passed; removing state, agentName, issueId or staleClaimReleasePending each fails the suite (assertion failures, not crashes) — the mutation-coverage claim in the PR body reproduces independently

Limitation: this workspace has no Postgres, so the DB-backed integration path was not exercised locally (CI's Database integration job covers it).

Remaining external verification (explicitly pending, not claimed green here)

No merge is performed by this run; the PR is left open for operator verification and human review.

@itsmiso-ai

Copy link
Copy Markdown
Contributor Author

Courier: already addressed (run courier-b7ad227788dfe837b5764bb83bb0e709)

Took over PR #1389 (misospace/dispatch, head 75351d4, branch courier/misospace/dispatch/issue-1350). No code change was required: all feedback present on the PR is already addressed at that head. (1) The AI review's Major finding on d1c0a62 (delete fence omitted the scanned ownership fields, so a row reassigned between scan and delete could still be deleted) is fixed in 75351d4 - the delete where now carries issueId and agentName alongside the state family and staleClaimReleasePending; the separate reviewer session's three items (state-conjunct test, issueId-conjunct test + test fake honouring where.issueId, corrected comment about the operator routes not taking the mutation guard) are also folded into 75351d4. (2) The AI review on head 75351d4 is clean/approved; get_review_threads reports 0 open threads and there are no unresolved comments. (3) CI on head: all check runs green except 'npm audit', which is pre-existing on main and tracked in #1386 - this diff touches only src/lib/agent-work.ts and src/lib/agent-work.test.ts with no dependency/lockfile change. (4) The PR body's out-of-scope exposure (delete fence does not re-read the lease set) is filed as open issue #1390. (5) Head verified published: remote ref courier/misospace/dispatch/issue-1350 sha == local HEAD == PR head sha; PR open and non-draft; working tree clean; no merge performed. Independent local validation re-run by this takeover (cwd /workspace, revision 75351d4, deps via npm ci): npm run lint = 0 errors + 1 pre-existing unrelated warning (src/app/login/page.tsx:39); npm run typecheck = clean; npx vitest run src/lib/agent-work.test.ts src/app/api/issues/claim/route.test.ts src/lib/stale-work.test.ts = 3 files passed / 187 tests passed; npm run test = 206 files passed / 5 skipped, 5322 passed / 0 failed, 27 skipped, 1 todo (149.5s). Additionally reproduced the PR's mutation-coverage claim independently: removing each delete-fence conjunct (state, agentName, issueId, staleClaimReleasePending) in turn makes src/lib/agent-work.test.ts fail with assertion failures (baseline 66 passed), with the file restored and the tree clean afterwards. Limitation: no Postgres in this workspace, so the DB integration path was not exercised locally. Pending external verification left explicitly pending: human review by joryirving, and the pre-existing npm audit CI failure (#1386). A concise takeover handoff comment with the settled decisions, check receipts and remaining external verification was posted on PR #1389.

…udit accounting (#1350)

The doc comment claimed "Live writers never leave the marker on an active-shaped
row (the sweep pairs it with the STALE mark), so the boundary is defence in depth
on top of the state filter". That is false: the operator agent-work reassign
route rewrites a row to `state: "CLAIMED"` without clearing
`staleClaimReleasePending`, so a sweep-marked `STALE` row can arrive at the
claim-path cleanup as an ACTIVE-SHAPED row that still owes its release — and the
state filter alone would not keep it. The marker conjunct is load-bearing.

The same comment also referred to "the other callers" of this function. There is
exactly one production caller (`src/app/api/issues/claim/route.ts`), and it does
take the per-issue mutation guard. The parties that do not take the guard are
the concurrent writers this cleanup races with, which is what the fence has to
survive.

Neither correction changes behaviour; both make the recorded rationale match the
code. The stranded-release gap the reassign route opens (the marker resurrected
onto an active-shaped row is inert for the sweep: its reclaim arms require
`staleClaimReleasePending: false`, its retry arm requires `state: "STALE"`) is
pre-existing and filed as #1391.

Adds a mixed-case regression test: with one candidate consumed and one refused
by the fence, the returned count and the audit's id list must name the consumed
row only. The accounting claim was previously pinned only in the all-consumed
case, so a regression back to reporting the scanned ids would have passed.

Local validation: `npx vitest run src/lib/agent-work.test.ts
src/app/api/issues/claim/route.test.ts src/lib/stale-work.test.ts
src/app/api/agent-work/route.test.ts` (4 files, 225 passed);
`npm run test` (5323 passed, 0 failed, 27 skipped, 1 todo);
`npm run lint` (0 errors, 1 pre-existing unrelated warning in
src/app/login/page.tsx); `npm run typecheck` (clean).
@itsmiso-ai

Copy link
Copy Markdown
Contributor Author

Takeover #2 — head advanced to b7d6bd9 (was 75351d4); one new commit, no behaviour change.

State assessment

  • Head b7d6bd9 on courier/misospace/dispatch/issue-1350 — remote ref sha == local HEAD == this PR's head sha. PR open, non-draft, no merge performed.
  • The AI review on the previous head (75351d4) was clean/approved, and there were no open review threads and no unresolved comments. The Major finding from d1c0a62 (delete fence omitted the scanned ownership fields) and the three items from the earlier reviewer session were already folded into 75351d4.
  • A fresh reviewer session on the final diff raised three further items — none of them a functional defect in the fence, the scan or the accounting. All three are fixed in b7d6bd9:
    1. minor — the doc comment claimed the pending marker is "defence in depth on top of the state filter" because "live writers never leave the marker on an active-shaped row". That is false: the operator agent-work reassign route rewrites a row to state: "CLAIMED" without clearing staleClaimReleasePending, so a sweep-marked STALE row can arrive here as an active-shaped row that still owes its release — the state filter alone would not keep it. The marker conjunct is load-bearing; the comment now says so. The fix itself was already correct; only the recorded rationale was wrong.
    2. nit — the same comment referred to "the other callers" of the lib function. There is exactly one production caller (src/app/api/issues/claim/route.ts) and it does take the per-issue mutation guard; the parties that do not take the guard are the concurrent writers this cleanup races with. The comment now names them.
    3. nit — the accounting claim ("the audit names exactly the rows the fence consumed") was pinned only in the all-consumed case, so a regression back to reporting the scanned ids would have passed. Added a mixed-case test: one candidate consumed, one refused by the fence → count 1, and the audit's id list names the consumed row only. Verified non-vacuous — reverting the notes to the scanned ids fails that test on its own.
  • The reassign route's leftover marker opens a separate, pre-existing gap (a resurrected active-shaped pending row is inert for the sweep: its reclaim arms require staleClaimReleasePending: false, its retry arm requires state: "STALE"). Out of this diff's scope — filed as fix(agent-work): settle the stale-release marker when an operator reassign rewrites a sweep-marked row #1391. The lease re-read exposure remains fix(agent-work): re-read the lease set inside the cleanup transaction so a lease taken mid-flight protects the row #1390; npm audit remains Security Audit gate red on main: 3 prod advisories (body-parser DoS high, hono, ip-address) #1386 (no dependency change in this diff).

Independent local validation (re-run by this takeover on b7d6bd9, cwd /workspace, deps via npm ci)

Check Command Result
Lint npm run lint 0 errors, 1 warning (src/app/login/page.tsx:39 — pre-existing, unrelated, unchanged since #744)
Typecheck npm run typecheck clean
Targeted suites npx vitest run src/lib/agent-work.test.ts src/app/api/issues/claim/route.test.ts src/lib/stale-work.test.ts src/app/api/agent-work/route.test.ts 4 files passed / 225 tests passed
Full suite npm run test 206 files passed / 5 skipped; 5323 passed, 0 failed, 27 skipped, 1 todo (147.7s)
Fence mutation check removed each delete-fence conjunct (state, agentName, issueId, staleClaimReleasePending) in turn, re-ran src/lib/agent-work.test.ts baseline 67 passed; each removal fails the suite with assertion failures (not crashes) — reproduces independently
Accounting mutation check reverted the audit notes to the scanned ids; separately dropped the "no audit row when the fence consumed nothing" guard the new mixed-case test fails on its own in the first case; the suite fails in the second

Limitation: no Postgres in this workspace, so the DB-backed integration path (real generated SQL, cascade of AgentWorkHistory, transaction commit/rollback) was not exercised locally — CI's Database integration job covers it.

Remaining external verification (explicitly pending, not claimed green here)

Review limitation: the reviewer session used the same configured model family (MiniMax-M3.1-Flash) as the coordinator, so it provides no model-family diversity.

No merge is performed by this run; the PR is left open for operator verification and human review.

its-saffron[bot]

This comment was marked as outdated.

…ction invariant (#1350)

A reviewer session on the final diff raised five items against the
claim-path cleanup's fence; four are fixed here and the fifth is answered
on the PR.

- The comments named a "self-heal" route family that does not exist and
  omitted the writers that actually race the cleanup: the operator
  agent-work routes (release / release-all / reassign) and the worker
  agent-work start / checkpoint / finish paths. They also claimed that a
  pending row always belongs to the sweep's retry arm, which is false for
  an ACTIVE-SHAPED pending row (the retry arm requires state STALE), so
  the comment now distinguishes the two shapes and says plainly that this
  cleanup is the repository's only `agentWork.deleteMany`.
- The delete's `issueId` conjunct was not exercised by any test, so
  dropping it passed the suite. A new test moves a row to another issue
  between the scan and the delete and pins that the fence refuses it.
- The "a deletion is never left without its audit row" invariant was
  unpinned. A new test asserts that the delete and the audit row share
  one transaction client, that the audit write follows the delete, and
  that a failing audit write surfaces instead of the cleanup reporting a
  successful deletion.

No behaviour change.
@itsmiso-ai

Copy link
Copy Markdown
Contributor Author

Takeover #3 — head advanced to 496abdc (was b7d6bd9); one new commit, no behaviour change.

State assessment

  • Head 496abdc on courier/misospace/dispatch/issue-1350 — remote ref sha == local HEAD == this PR's head sha. PR open, non-draft, no merge performed.
  • All feedback present on the PR is addressed at this head: the AI review's Major finding on d1c0a62 and the earlier reviewer sessions' items are folded into 75351d4 / b7d6bd9; the AI review on b7d6bd9 was clean/approved; there are no open review threads and no unresolved comments.
  • A third reviewer session on the diff at b7d6bd9 (same model family — MiniMax-M3.1-Flash — so no provider diversity was available) raised five items: four are fixed in 496abdc (two comment inaccuracies, the unpinned transaction invariant, the unpinned issueId conjunct), one is answered below. The PR body records each item and its disposition.

Answer to the remaining item — the per-row CAS statement count. Not changed, deliberately: a bulk deleteMany returns a count, not ids, so the per-row CAS is exactly what lets the audit row name the rows the fence consumed (the honest-accounting property an earlier review round pinned). N is bounded by the issue's leftover active-shaped rows (one per crashed attempt), and every delete plus the audit row shares one transaction, so a timeout rolls the whole cleanup back rather than leaving a partial delete without its audit row. The transaction and its timeout shape are unchanged by this PR — the pre-#1350 code put one bulk delete plus the audit write in the same prisma.$transaction; only the statement count grew, from 1 to N. If a bounded timeout is ever wanted, prisma.$transaction(fn, { timeout, maxWait }) at src/lib/agent-work.ts:481 is the single place to add it, at the cost of holding the row locks longer.

Independent local validation (re-run by this takeover, cwd /workspace, revision 496abdc, deps via npm ci)

Check Command Result
Lint npm run lint 0 errors, 1 warning (src/app/login/page.tsx:39 — pre-existing, unrelated)
Typecheck npm run typecheck clean
Targeted suites npx vitest run src/lib/agent-work.test.ts src/app/api/issues/claim/route.test.ts src/lib/stale-work.test.ts src/app/api/agent-work/route.test.ts 4 files passed / 227 tests passed
Full suite npm run test 206 files passed / 5 skipped; 5325 passed, 0 failed, 27 skipped, 1 todo
Fence mutation check removed each delete-fence conjunct (state, agentName, issueId, staleClaimReleasePending) in turn, re-ran src/lib/agent-work.test.ts baseline 69 passed; each removal fails the suite with assertion failures (not crashes) — 4 / 3 / 4 / 3 failures
Accounting mutation check audit notes regressed to the scanned ids; dropped the "no audit row when the fence consumed nothing" guard; returned count regressed to the scanned count 1 / 4 / 5 failures respectively

The issueId conjunct is newly covered: before this commit, removing it from the delete where left the suite green (the PR body's mutation claim was overstated for that one conjunct). The new test moves a row to another issue between the scan and the delete and pins that the fence refuses it.

Limitation: no Postgres in this workspace, so the DB-backed integration path (real generated SQL, the AgentWorkHistory cascade, transaction commit/rollback) was not exercised locally.

Follow-up filed

Remaining external verification (explicitly pending, not claimed green here)

Review limitation: every review session on this PR (the AI PR reviewer and the separate coordinator-run sessions) used the same configured model family (MiniMax-M3.1-Flash) as the coordinator, so no model-family diversity was available.

No merge is performed by this run; the PR is left open for operator verification and human review.

its-saffron[bot]

This comment was marked as outdated.

…c reference (#1350)

The claim-path cleanup's safety comment enumerated the writers it races
with but omitted one: the operator unclaim path calls
`releaseAgentWorkByAgentAndIssue` (`src/lib/lease.ts`) AFTER its guarded
block, so it writes `state: "RELEASED"` on AgentWork rows without the
per-issue mutation guard — the same unguarded shape as the operator
agent-work routes. It is now named, so the fence's coverage of that
shape is not an accident.

The comment also stated the stale-work sweep "cannot interleave with the
claim path"; a maintainer force-claim can override that guard, so the
claim now says "normally", and records that the fence's pending conjunct
refuses a row that became pending after the scan either way.

`docs/ownership-reconciliation.md` described the cleanup as deleting every
AgentWork row with no matching active lease and pointed at a stale line
number. It now states the pending-release boundary and the fence, and the
reference is corrected.
@itsmiso-ai

Copy link
Copy Markdown
Contributor Author

Takeover #4 — head 699d00d (one new commit, comments/docs only; no behaviour change).

State assessment at head 699d00d (cwd /workspace, branch courier/misospace/dispatch/issue-1350, base main @ 313bc5a)

Check Command Result
Lint npm run lint 0 errors, 1 warning (src/app/login/page.tsx:39 — pre-existing, unrelated, unchanged since #744)
Typecheck npm run typecheck clean
Targeted suites npx vitest run src/lib/agent-work.test.ts src/app/api/issues/claim/route.test.ts src/lib/stale-work.test.ts src/app/api/agent-work/route.test.ts 4 files passed / 227 passed
Full suite npm run test 206 files passed / 5 skipped; 5325 passed, 0 failed, 27 skipped, 1 todo (147.1s)
Fence mutation check removed each delete-fence conjunct (state, agentName, issueId, staleClaimReleasePending) in turn, re-ran src/lib/agent-work.test.ts baseline 69 passed; each removal fails the suite with assertion failures (not crashes) — 4 / 3 / 4 / 3 failures

What this takeover changed and why

The AI review on 496abdc was already APPROVED with no findings, and all four prior reviewer sessions' items were addressed. This takeover added one bounded review session on the final diff (same model family — MiniMax-M3.1-Flash — so no provider diversity was available), which returned SHIP with no blocker/major finding, re-confirmed the three load-bearing claims (pending rows are never consumed; the fence protects the delete; the accounting is honest) and that every scan and fence conjunct has behavioural coverage. It raised two documentation-accuracy items, both fixed in 699d00d:

  1. The fence comment enumerated the writers this cleanup races with but omitted the operator unclaim path — it calls releaseAgentWorkByAgentAndIssue (src/lib/lease.ts) after its guarded block, so it writes state: "RELEASED" on AgentWork rows without the per-issue mutation guard (same unguarded shape as the operator agent-work routes). Now named.
  2. The comment said the stale-work sweep "cannot interleave with the claim path"; a maintainer force-claim can override that guard, so it now reads "normally" and records that the fence's pending conjunct refuses a row that became pending after the scan either way.

Also fixed: docs/ownership-reconciliation.md still described the cleanup as deleting every AgentWork row with no matching active lease (no pending boundary) and cited a stale line number — it now states the pending-release boundary and the fence, with the reference corrected.

Not changed, deliberately (each is either settled or filed): the per-row CAS (N statements) — a bulk delete returns a count, not ids, and the audit must name exactly the rows the fence consumed; the lease-set re-read gap (#1390); the operator-reassign leftover marker (#1391); the live-PostgreSQL rollback test (#1392).

Remaining external verification — left explicitly pending

No merge performed by this run; the PR is left open, non-draft, with the current head pushed.

its-saffron[bot]

This comment was marked as outdated.

@its-saffron

its-saffron Bot commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

⏸️ This PR fix item is PARKED after repeated attempts without progress. It is not waiting on a human decision; an operator can requeue it to retry.

Reason: Failing check: npm audit (failure)

Outcome: no_progress

Worker executions: 5

Enqueue rows: 8

Enqueues by lane:

  • NORMAL: 8

Final failure: Agent reported failed; bounded at 5 fix attempts (PR_FIX_MAX_ATTEMPTS=5), parked instead of re-queuing (#1133).

Failing run(s):

Last attempt: CI check "npm audit" failed (failure) on this PR.

Error from the job log:
�[36;1mnpm run audit�[0m
shell: /usr/bin/bash -e {0}
##[endgroup]

dispatch@0.5.71 audit
npm audit --omit=dev --include=prod --audit-level=high --fetch-retries=5 --fetch-timeout=120000 --fetch-retry-mintimeout=20000 --fetch-retry-maxtimeout=120000

npm audit report

body-parser 2.0.0 - 2.3.0
Severity: high
body-parser vulnerable to Denial of Service via trailing bytes after a compressed request body - GHSA-27r3-jxmh-p828
fix available via npm audit fix
node_modules/body-parser

fast-uri 3.0.0 - 3.1.8
Severity: moderate
fast-uri vulnerable to URI scheme spoofing via percent-encoded scheme decoding - GHSA-w74x-v6f9-j3xg
fast-uri vulnerable to improper path equivalence in relative reference normalization - GHSA-xhq7-xmqj-769h
fast-uri vulnerable to host confusion via unchecked WHATWG host canonicalization - GHSA-wjr9-j795-jxcr
fast-uri vulnerable to scheme injection via an unescaped first-segment colon in suffix references - GHSA-333x-w4r6-6crp
fix available via npm audit fix
node_modules/fast-uri

hono 4.12.4 - 4.13.10
Severity: moderate
Hono: serveStatic decodes the request path a second time, leading to bypass of middleware on static paths - GHSA-5r4p-p66f-jhc7
fix available via npm audit fix
node_modules/hono

ip-address <=10.7.3
Severity: moderate
ip-address: Address6 matches a zone identifier in quadratic time ahead of its length limit, allowing a single long string to stall the process - GHSA-rxp7-6hxx-x8q5
fix available via npm audit fix
node_modules/ip-address

4 vulnerabilities (3 moderate, 1 high)

To address all issues, run:
npm audit fix
##[error]Process completed with exit code 1.
Post job cleanup.
##[start-action display=Setup Node;id=__960ac245-eb83-4f66-b61a-2d5777b71ea1.__actions_setup-node]
##[end-action id=__960ac245-eb83-4f66-b61a-2d5777b71ea1.__actions_setup-node;outcome=skipped;conclusion=skipped;duration_ms=0]
Post job cleanup.
[command]/usr/bin/git version
git version 2.55.0
Temporarily overriding HOME='/home/runner/work/_temp/95a59ff7-053b-4b42-837d-a31f05b93b56' before making global git config changes
Adding repository directory to the temporary git global config as a safe directory
[command]/usr/bin/git config --global --add safe.directory /home/runner/work/dispatch/dispatch
Removing SSH command configuration
[command]/usr/bin/git config --local --name-only --get-regexp core.sshCommand
[command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
Removing HTTP extra header
[command]/usr/bin/git config --local --name-only --get-regexp http.https://github.com/.extraheader

Full log: https://github.com/misospace/dispatch/actions/runs/40049264660/job/120738002702

Fix the root cause the log shows.

Posted automatically by Dispatch on 2026-10-24T01:46:11.593Z

joryirving
joryirving previously approved these changes Oct 24, 2026

@joryirving joryirving left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Secondary hard-blocker review on exact head 93934430e0bb28780e251de86e21882540b45f07: APPROVE. The claim-path cleanup now excludes rows still owed a release and fences each delete on the scanned state family, issue, owner, and pending marker. It reports/audits only rows actually consumed, and keeps delete plus audit in one transaction. The final diff against current main is limited to the intended three files. I updated the branch to include #1387's dependency audit fix; CI is rerunning on this exact head. The PostgreSQL rollback integration gap is documented and tracked separately in #1392, not a blocker for this bounded fix.

@joryirving
joryirving enabled auto-merge (squash) October 24, 2026 02:25
@itsmiso-ai
itsmiso-ai dismissed joryirving’s stale review October 24, 2026 02:29

The merge-base changed after approval.

@joryirving
joryirving merged commit 29d4168 into main Oct 24, 2026
13 checks passed
@joryirving
joryirving deleted the courier/misospace/dispatch/issue-1350 branch October 24, 2026 02:29
@its-miso its-miso Bot mentioned this pull request Oct 24, 2026

@its-saffron its-saffron Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Automated Review

Full PR review.

Analysis engine: MiniMax-M3.1-Flash (anthropic) — primary route · pr-reviewer-action v3.7.0

Recommendation: approve; the linked issue is addressed correctly and the exact head passes the repository’s full CI gates.

Linked Issue Fit

  • The pending-release invariant is implemented consistently with activeStateFamilies: both scan and delete require staleClaimReleasePending: false, preserving rows that still owe a release.
  • The delete fence includes id, issue, owner, executing state, and pending boundary; tests cover pending, reassigned, released, moved, and mixed-accounting rows.
  • Listed claims held: 3/3; repository search also confirmed this is the only production agentWork.deleteMany path.

Required Checks

  • check for breaking API changes in updated dependencies: satisfied — the lockfile upgrades were inspected, and the exact head’s tests, build, Docker builds, and npm audit succeeded without an observed API break.
  • run full test suite after upgrade: satisfied — the full test suite and all reported CI checks succeeded on the exact head.

Tool Harness Findings

  • Repository inspection confirmed the canonical pending boundary, the sole delete call site, the claim-route caller, the Node 24 host pins, and the updated dependency graph.

Unknowns or Needs Verification

  • Upstream changelogs were not retrieved before the tool budget was exhausted; exact-head Node 24 CI/build evidence is strong runtime compatibility evidence, but it is not a substitute for release-note review.

joryirving added a commit that referenced this pull request Oct 24, 2026
…gn rewrites a sweep-marked row (#1391) (#1393)

* fix(automation): surface failed sync responses on repo detail page (#1385)

* chore(deps): refresh lock within ranges to clear npm audit gate

npm audit --omit=dev (blocking CI gate) failed on main and on this PR
head: body-parser 2.3.0 was high (GHSA-27r3-jxmh-p828, DoS via trailing
bytes after a compressed body), hono 4.13.7 and ip-address 10.7.2 were
moderate.

npm audit fix --package-lock-only --include=prod bumps body-parser to
2.4.0, hono to 4.13.13, ip-address to 10.7.4 and qs to 6.16.1; npm also
re-resolved the dev tree (vite, rolldown, @oxc-project/types and
@rolldown/* bindings) inside the same existing ranges. package.json and
every range are unchanged; package-lock.json was already identical to
main before this commit, so the failure pre-dates this PR.

Local validation at this change state: npm ci --include=dev, npm run
audit (0 vulnerabilities), typecheck, lint (1 pre-existing warning) and
vitest all exit 0.

* fix(agent-work): settle stale-claim-release marker on operator reassign (#1391)

* fix(agent-work): give the reassigned row a live lease horizon (#1391)

The operator reassign rewrite to CLAIMED inherited the lease expiry the
preceding RELEASED write had stamped (now), so the row the new owner
received carried an already-expired lease — a shape no claim path
produces. The sweep's reclaim arm fires on
lastHeartbeatAt < cutoff OR leaseExpiresAt < cutoff, so an expiry that
is not ahead of the heartbeat lets the expiry arm fire at the same
instant as the heartbeat arm instead of strictly after it.

The handover is a fresh claim, so it now carries the claim path's lease
horizon (AGENT_WORK_LEASE_DURATION_MS, the value startAgentWork and
checkpointAgentWork stamp): the expiry stays ahead of the heartbeat and
the reclaim stays governed by the heartbeat window, exactly like a row
from startAgentWork. The horizon moves into an exported constant so the
claim path and the handover cannot drift.

Addresses the AI review findings on PR #1393 (lease expiry on the
CLAIMED rewrite, and the regression test not asserting it). The
staleClaimReleasePending settlement from bd3bd51 is unchanged.

Co-authored-by: Courier <courier@localhost>

* fix(automation): surface failed/in-progress per-repo sync responses (#1324) (#1384)

* fix(automation): surface failed per-repo sync responses (#1324)

* fix(automation): surface failed sync responses on repo detail page (#1385)

* chore(deps): refresh lock within ranges to clear npm audit gate

npm audit --omit=dev (blocking CI gate) failed on main and on this PR
head: body-parser 2.3.0 was high (GHSA-27r3-jxmh-p828, DoS via trailing
bytes after a compressed body), hono 4.13.7 and ip-address 10.7.2 were
moderate.

npm audit fix --package-lock-only --include=prod bumps body-parser to
2.4.0, hono to 4.13.13, ip-address to 10.7.4 and qs to 6.16.1; npm also
re-resolved the dev tree (vite, rolldown, @oxc-project/types and
@rolldown/* bindings) inside the same existing ranges. package.json and
every range are unchanged; package-lock.json was already identical to
main before this commit, so the failure pre-dates this PR.

Local validation at this change state: npm ci --include=dev, npm run
audit (0 vulnerabilities), typecheck, lint (1 pre-existing warning) and
vitest all exit 0.

---------

Co-authored-by: Courier <courier@localhost>
Co-authored-by: Jory Irving <46251616+joryirving@users.noreply.github.com>

* fix(agent-work): preserve stale-AgentWork rows still owed a claim release (#1350) (#1389)

* fix(automation): surface failed sync responses on repo detail page (#1385)

* chore(deps): refresh lock within ranges to clear npm audit gate

npm audit --omit=dev (blocking CI gate) failed on main and on this PR
head: body-parser 2.3.0 was high (GHSA-27r3-jxmh-p828, DoS via trailing
bytes after a compressed body), hono 4.13.7 and ip-address 10.7.2 were
moderate.

npm audit fix --package-lock-only --include=prod bumps body-parser to
2.4.0, hono to 4.13.13, ip-address to 10.7.4 and qs to 6.16.1; npm also
re-resolved the dev tree (vite, rolldown, @oxc-project/types and
@rolldown/* bindings) inside the same existing ranges. package.json and
every range are unchanged; package-lock.json was already identical to
main before this commit, so the failure pre-dates this PR.

Local validation at this change state: npm ci --include=dev, npm run
audit (0 vulnerabilities), typecheck, lint (1 pre-existing warning) and
vitest all exit 0.

* fix(agent-work): preserve stale-AgentWork rows still owed a claim release (#1350)

* fix(agent-work): fence the claim-path cleanup on scanned ownership (#1350)

The claim-path cleanup's fenced delete re-checked the state family and the
pending-release boundary, but not the ownership fields the scan matched
(`issueId`, `agentName`). A row reassigned between the scan and the delete
kept its id, state and marker, so the fence consumed it and the newly owned
row was deleted instead of being left to its new owner.

The delete now carries the scanned ownership fields as well, so the fence
matches the scan on every conjunct that identifies the row. Tests cover the
reassignment race, the concurrent state settle (operator release), the
scan's issue scope, and the pending boundary; each fence conjunct is
mutation-covered.

Also correct the comment that credited the per-issue mutation guard for the
fence's safety: the operator agent-work routes do not take that guard, so
the fence itself is the load-bearing protection.

* fix(agent-work): correct the pending-boundary rationale and pin the audit accounting (#1350)

The doc comment claimed "Live writers never leave the marker on an active-shaped
row (the sweep pairs it with the STALE mark), so the boundary is defence in depth
on top of the state filter". That is false: the operator agent-work reassign
route rewrites a row to `state: "CLAIMED"` without clearing
`staleClaimReleasePending`, so a sweep-marked `STALE` row can arrive at the
claim-path cleanup as an ACTIVE-SHAPED row that still owes its release — and the
state filter alone would not keep it. The marker conjunct is load-bearing.

The same comment also referred to "the other callers" of this function. There is
exactly one production caller (`src/app/api/issues/claim/route.ts`), and it does
take the per-issue mutation guard. The parties that do not take the guard are
the concurrent writers this cleanup races with, which is what the fence has to
survive.

Neither correction changes behaviour; both make the recorded rationale match the
code. The stranded-release gap the reassign route opens (the marker resurrected
onto an active-shaped row is inert for the sweep: its reclaim arms require
`staleClaimReleasePending: false`, its retry arm requires `state: "STALE"`) is
pre-existing and filed as #1391.

Adds a mixed-case regression test: with one candidate consumed and one refused
by the fence, the returned count and the audit's id list must name the consumed
row only. The accounting claim was previously pinned only in the all-consumed
case, so a regression back to reporting the scanned ids would have passed.

Local validation: `npx vitest run src/lib/agent-work.test.ts
src/app/api/issues/claim/route.test.ts src/lib/stale-work.test.ts
src/app/api/agent-work/route.test.ts` (4 files, 225 passed);
`npm run test` (5323 passed, 0 failed, 27 skipped, 1 todo);
`npm run lint` (0 errors, 1 pre-existing unrelated warning in
src/app/login/page.tsx); `npm run typecheck` (clean).

* fix(agent-work): correct the fence's stated racers and pin its transaction invariant (#1350)

A reviewer session on the final diff raised five items against the
claim-path cleanup's fence; four are fixed here and the fifth is answered
on the PR.

- The comments named a "self-heal" route family that does not exist and
  omitted the writers that actually race the cleanup: the operator
  agent-work routes (release / release-all / reassign) and the worker
  agent-work start / checkpoint / finish paths. They also claimed that a
  pending row always belongs to the sweep's retry arm, which is false for
  an ACTIVE-SHAPED pending row (the retry arm requires state STALE), so
  the comment now distinguishes the two shapes and says plainly that this
  cleanup is the repository's only `agentWork.deleteMany`.
- The delete's `issueId` conjunct was not exercised by any test, so
  dropping it passed the suite. A new test moves a row to another issue
  between the scan and the delete and pins that the fence refuses it.
- The "a deletion is never left without its audit row" invariant was
  unpinned. A new test asserts that the delete and the audit row share
  one transaction client, that the audit write follows the delete, and
  that a failing audit write surfaces instead of the cleanup reporting a
  successful deletion.

No behaviour change.

* fix(agent-work): complete the cleanup's racer list and correct its doc reference (#1350)

The claim-path cleanup's safety comment enumerated the writers it races
with but omitted one: the operator unclaim path calls
`releaseAgentWorkByAgentAndIssue` (`src/lib/lease.ts`) AFTER its guarded
block, so it writes `state: "RELEASED"` on AgentWork rows without the
per-issue mutation guard — the same unguarded shape as the operator
agent-work routes. It is now named, so the fence's coverage of that
shape is not an accident.

The comment also stated the stale-work sweep "cannot interleave with the
claim path"; a maintainer force-claim can override that guard, so the
claim now says "normally", and records that the fence's pending conjunct
refuses a row that became pending after the scan either way.

`docs/ownership-reconciliation.md` described the cleanup as deleting every
AgentWork row with no matching active lease and pointed at a stale line
number. It now states the pending-release boundary and the fence, and the
reference is corrected.

---------

Co-authored-by: Courier <courier@localhost>
Co-authored-by: Jory Irving <46251616+joryirving@users.noreply.github.com>

---------

Co-authored-by: Courier <courier@localhost>
Co-authored-by: Jory Irving <46251616+joryirving@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Claim-path stale-AgentWork cleanup hard-deletes rows still owed a claim release

2 participants