Repository navigation
fix(ownership): renew commits its lease CAS and AgentWork liveness write atomically (#1315) - #1336
Conversation
…ite atomically (#1315) renewOwnership wrote the fenced lease TTL extension and the matching AgentWork liveness write (heartbeat + renewed expiry + history) as two separate commits. A crash between them left the lease renewed while the AgentWork row kept its old lapsed 5-minute leaseExpiresAt; the stale-work sweeper reclaims on leaseExpiresAt too, so it marked the renewed worker STALE and deleted its lease — the renewal was lost. Since PR #1300 that write is load-bearing for the worker's survival, not best-effort evidence. Wrap the CAS (kept as the transaction's first statement so the lost-race early return commits nothing), the row re-read and the AgentWork write in one prisma.$transaction on the transaction's client. A failure rolls the whole renewal back; the route 500s and never reports success. Fencing (claimToken + disposition: null CAS), replay semantics (a retry just re-extends) and agentWorkHeartbeated's truthful absence reporting are unchanged.
…er/disposition) and sweeper (renew) Review follow-up on #1315: the shared closing sentence read as if the replay-guard rationale justified renew's transaction too. Renew has no replay branch; its constraint is that a committed lease CAS must never lose its liveness write.
There was a problem hiding this comment.
AI Automated Review
Full PR review.
Analysis engine: MiniMax-M3.1-Flash (anthropic) — primary route · pr-reviewer-action v3.7.0
Recommendation: Approve. The renewal now commits the fenced lease CAS, matching AgentWork heartbeat, renewed expiry, and history row through one transaction client, satisfying PR 1315 without changing the renewal contract.
Claims and Findings
- Held: 5/5 stated invariants are supported by the implementation. The CAS remains fenced by
claimTokenanddisposition: nulland is the transaction’s first write; lease and AgentWork use the same clock and expiry; rollback propagates to the route’s 500 response; retries remain ordinary TTL-extending renewals; andIssueMutationGuardstill surrounds the operation. - The new failure-injection tests cover update failure, history failure, transaction-client usage, retry convergence, and lost-race behavior.
Linked Issue Fit
- Satisfies the issue’s atomicity requirement, honest failure semantics, unchanged fencing contract, and unchanged replay semantics.
- No linked acceptance criterion is missing or misinterpreted.
Tool Harness Findings
- All 24 executed reads/searches succeeded and confirmed the production wiring uses the transaction client for all three renewal writes; the transactional test harness models rollback distinctly from root-client writes.
- The harness exhausted its budget before a broader sibling-path scan, but the supplied CI results report successful tests, database integration, lint, typecheck, and build.
Unknowns or Needs Verification
- No blocker-level unknown remains; the exhaustive tool investigation was truncated by the fixed budget, but the changed production path, its route consumer, transaction harness, mutation guard, stale-sweep integration, and relevant repository history were directly inspected.
joryirving
left a comment
There was a problem hiding this comment.
Secondary hard-blocker review on exact head 771bf14a1ef0fb7e69aa274f559caf76120cdd2d: APPROVE. The renewal CAS, lease re-read, matching AgentWork liveness write, and history record now execute through one Prisma transaction client, preserving the existing claim-token/disposition fence and honest rollback semantics. The lost-race path performs no prior write, retry still safely re-extends TTL, and failure-injection tests cover AgentWork/history rollback plus transaction-client usage. The primary reviewer is clean and all exact-head workflows are green. I found no material ownership/concurrency blocker.
What
renewOwnershipcommitted the fenced lease compare-and-set (TTL extension) and the matching activeAgentWorkliveness write (lastHeartbeatAt+leaseExpiresAt+ history row) as two separate writes. A crash between them left the lease renewed while theAgentWorkrow kept its old, lapsed 5-minuteleaseExpiresAt;sweepStaleWorkreclaims candidates onleaseExpiresAttoo, so it marked the renewed worker STALE and deleted its lease — the renewal was lost. Since PR #1300 that write is load-bearing for the worker's survival, not best-effort evidence.Fix: wrap the CAS, the post-CAS lease re-read, and the
AgentWorkheartbeat/history write in ONEprisma.$transaction, all on the transaction's client (heartbeatMatchingAgentWorknow takes thePrisma.TransactionClient). A failure rolls the whole renewal back; the route 500s — a rolled-back renewal never reports success.src/lib/worker-ownership.ts—renewOwnership+heartbeatMatchingAgentWork, module header rule 5 rewritten.src/app/api/agent-work/ownership/route.test.ts— new renew-atomicity suite (single-transaction-client proof, rollback at the liveness write and at the history write, lost-race CAS commits nothing), plus a newagentWorkHistory.createfailure-injection point in the existing harness.docs/worker-ownership-capability.md— atomicity section updated.Fixes #1315
Invariants this change must keep, and every other path enforcing each rule
claimToken+disposition: null. Kept verbatim inrenewOwnership; it is also the first statement of the transaction, so thecount === 0early return commits an empty transaction (same posturedispositionOwnershipdocuments). Other writers of this rule, all untouched:dispositionOwnership's terminal CAS (same fence),upsertLeaseclaim-token rotation/reset insrc/lib/lease.ts, and the stale sweep's claim-epoch-fenceddeleteManyon{claimToken, expiredAt, lastRenewalObservedAt}insrc/lib/stale-work.ts(The stale-work sweep deletes the lease for the whole (agent, issue) pair when any sibling row is stale #1317) — which still backs off if a renewal commits inside its decision window, now doubly so because the row expiry moves with the lease.AgentWorkrow move on the operation's own clock and expiry — no split state visible to the sweeper. Enforced byheartbeatMatchingAgentWorkriding the transaction client; the values written are the samenow/expiresAtthe lease received (tests assert equality withdb.lease.expiredAt/renewedAt). Other writers of both timestamps:startAgentWork/checkpointAgentWorkinsrc/lib/agent-work.tsmaintain their own 5-minute execution lease independently of the ownership capability — untouched and out of scope. The sweeper's candidate predicate (lastHeartbeatAt OR leaseExpiresAt) and its live-sibling complement predicate insrc/lib/stale-work.tsare unchanged; the feat(ownership): persist optional worker renewal and fenced claim evidence #1251 suite exercises them through the sweep's real clause, and those tests pass unmodified.register/disposition; unchanged.renew's replay semantics unchanged. A renewal is idempotent by nature — a retry re-extends the TTL. No replay branch was added.agentWorkHeartbeated: falsestill means "no matching active row existed" (absence), never a lost write; a lost write is now impossible (rollback + 500).IssueMutationGuardanchor (feat(recovery): add fenced capacity-exhaustion handoffs #1259) still surrounds the whole operation; the transaction narrows the window the sweeper's epoch fence is the belt for.Validation (run on head 771bf14; code identical at 0b140a0 which received the independent review)
vitest run src/app/api/agent-work/ownership/route.test.ts src/lib/stale-work.test.ts→ 79 passed (incl. 4 new renew-atomicity tests).vitest run→ 202 files / 5089 tests passed (24 skipped, 1 todo).npm run typecheck→ clean.npm run lint→ 0 errors (1 pre-existing warning insrc/app/login/page.tsx, untouched).