[itsmiso-ai]

Closes #1252

Implements the Snapshot API (approach D) of the #1237/#1239 design catalogue (docs/observability.md §7): GET /api/observability — a bounded, authenticated, read-only JSON snapshot of Dispatch's own work-supply, execution, ownership, PR-fix, admission, grooming, and pipeline signals. No exporter dependency, no /metrics, no coupling to Courier or any dashboard installation.

What lands

• src/lib/observability-signals.ts — pure derivation over loaded rows; all dimension allowlists live here; explicit unknown-vs-zero; as-of-scrape windowed gauges only (failures 900/3600 s, activity 1800/3600 s).
• src/lib/observability-snapshot.ts — read-only collector (findMany/findFirst/count only; no forge/GitHub calls, no scheduler mutation).
• src/app/api/observability/route.ts — maintainer-tier GET, Cache-Control: no-store.
• docs/observability-snapshot.md + minimal status notes in docs/observability.md; DISPATCH_OBSERVABILITY_STRANDED_ASSIGNMENT_MS in .env.example / AGENTS.md.
• Independent review pass done on head c0e02d7 (no blockers); all findings fixed in 6adc834 (presence-based A6 age evidence, AgentWork worker-identity pickup filter, canonical ack parsing, declared evidence horizon, unknown A4 state when the sync job is disabled).

Acceptance mapping

• Distinguishable scenario states: unit-tested matrix — healthy idle, busy steady churn, genuinely stranded assignment, no first successful sync, failing discovery, inactive worker, held dependency — asserted pairwise-distinct (digest collision test) and each with targeted assertion.
• Churn is not aged stranded / an actual aged assignment is: overlapping assignments that gain leases seconds later produce agedPastThreshold: 0 while a known-age un-executed assignment produces ≥ 1; GitHub-side assignments with no persisted age evidence stay ageUnknown and are never aged (fail-open). Re-assigning already-delivered work does not re-arm the clock.
• Windowed audit-failure arrivals at equal counts stay observable: as-of-scrape windows recomputed per request; the test shows a fresh failure visible in the 900 s window even as an older one expires — never delta()/increase()-able by construction.
• Read-only: collector + route tests mock a Prisma whose create/update/updateMany/upsert/delete/deleteMany all throw; a 200 response proves no write path executes. Claim release, requeue, status mutation, credential changes, and mandatory backends are absent.

Invariants this change must keep — and every other path enforcing the same rule

1. IssueSyncRun statuses are running/completed/failed (no success) — writers: src/lib/sync-lock.ts:160, src/app/api/sync/route.ts:67, src/app/api/sync/scheduled/route.ts (completed/failed paths). Kept: the snapshot matches exactly "completed"/"failed" and buckets anything else as unknown; the §4.9 source-drift check in docs/observability.test.ts is untouched and still passes.
2. PR-fix hand-out/ack identity: QUEUED + dispatchedGeneration === generation + "<agent>@<generation>" acks parsed at the LAST @ — writer/enforcer: src/lib/pr-fix-queue.ts (ackPrFixHandout, reclaimStalePrFixHandouts, parseAgentHandoutToken). Kept: the snapshot's ackedGenerations mirrors the canonical rule (integer ≥ 1, @-bearing agent names tested); the reclaimer itself is unchanged; overdue is age-tied to PR_FIX_HANDOUT_TIMEOUT_MS, matching A7's reviewed age-tied predicate.
3. Grooming admission is one rule — evaluateQueueAdmission in src/lib/queue-admission.ts, used by /queue//next-task via src/lib/agent-queue-fetch.ts. Kept: the snapshot reuses that exact function + loadAdmissionRuns (no reimplementation, no divergence); queue routes untouched.
4. PARKED is operational exhaustion, BLOCKED is the only human decision — enforced at write time by resolvePrFixOutcome/markPrFixItem in src/lib/pr-fix-queue.ts and surfaced by src/lib/pr-fix-surfacing.ts. Kept: settlement buckets them separately; parkedByOutcome never feeds humanDecisionsBlocked (test-pinned).
5. System emitters (hosted groomer, heartbeat, legacy runs) are not task execution — enforced elsewhere by the groomer's own lease check (src/lib/groomer/run.ts hasOtherLease) and the runType writers in tasks/report/heartbeat. Kept: execution/pickup/reported/started gauges filter by the agent/*-emitter exclusion and the task runType allowlist (implement, followup-pr), not subtraction.
6. No identity/secret leakage into metrics dimensions (design §9) — other surfaces (GET /audit, board API) legitimately carry drill-down identities; this snapshot must not. Kept: bounded allowlists with unknown buckets everywhere, and a test asserting repo names, issue numbers, agent names and acme never appear in the serialized snapshot.
7. New API routes default to maintainer tier — requiredTierForRoute in src/lib/auth.ts. Kept: no WORKER_ALLOWLIST entry was added; route test asserts 401 unauthenticated / 403 worker-tier.
8. Windowed values are gauges, not monotonic counters (design §4.1, pinned by docs/observability.test.ts no-rate-function rule). Kept: everything is recomputed from row timestamps per request; documented in the snapshot contract.

Local validation (run against head 6adc834, cwd /workspace)

• npm test equivalent — NODE_ENV=development npx vitest run: 202 files passed, 4 skipped; 5085 passed, 24 skipped, 1 todo (includes docs/observability.test.ts structural/citation checks and env-example.test.ts).
• npx tsc --noEmit: exit 0.
• npx eslint .: 0 errors (1 pre-existing warning in src/app/login/page.tsx, unrelated).
• Targeted suites for the new code: 52 passed.

CI is not asserted green — image build / Trivy (advisory) / AI review are external and left to the operator. No external verification was waited on.

Handoff

Settled decisions: snapshot JSON (approach D) only — a /metrics scrape exporter and OTel remain deferred per §10; A6 age evidence is presence-based with a declared 30-day evidence horizon (evidenceLookbackSeconds); null means unknown (absent sync series, off-mode admission, unevaluable A4 guard) and never zero. Remaining external verification: CI on this head, and an operator smoke of GET /api/observability with a maintainer token against a live instance.

[joryirving]

Blocker-only review on exact head 6adc8347453d70a8800b70d16757f580f7ec1899: APPROVE. The new GET /api/observability surface is maintainer-gated through the existing default-deny route tiering, carries no identity/error-text dimensions, and the collector is read-only (database reads plus pure derivation; no forge or scheduler mutations). The implementation reuses the canonical admission rule, preserves explicit unknown-vs-zero semantics, uses age-tied evidence for stranded assignments and PR-fix hand-outs, excludes non-task/system activity from execution signals, and keeps windowed values as gauges. I found no material correctness, security, ownership, or data-integrity blocker under the agreed merge-biased standard. CI, Security Audit, Build Dispatch Image, and PR Smoke are green; AI PR Review was still running when this approval was submitted.

[its-saffron]

AI Automated Review

Full PR review.

Analysis engine: MiniMax-M3.1-Flash (anthropic) — primary route · pr-reviewer-action v3.7.0

Findings (1 major)

Severity	Location	Finding
⚠️ Major	docs/observability-snapshot.md:21	The contract promises an allowlisted AgentWork checkpoint dimension, but the snapshot output exposes only AgentWork state, leaving the implemented signal catalogue incomplete.

Request changes: the snapshot promises an AgentWork checkpoint dimension but does not expose one.

Change-by-change findings

• Claim 1 violated: docs/observability-snapshot.md says the bounded dimensions include AgentWork state/checkpoint, but ObservabilitySnapshot.execution exposes only agentWorkByState; add the allowlisted checkpoint dimension or remove the contract promise if it is intentionally deferred.
• Claim 2 held: derivation remains pure, uses the documented fixed windows, and keeps absent baselines distinct from zero.
• Claim 3 held: the collector uses read queries only and does not invoke forge or scheduler mutation paths.

Sources

• src/lib/observability-signals.ts
• src/lib/observability-snapshot.ts
• src/app/api/observability/route.ts
• src/lib/queue-admission.ts
• src/lib/pr-fix-queue.ts
• src/lib/auth.ts

Linked Issue Fit

The read-only snapshot, distinguishability matrix, task-run allowlist, real sync statuses, age-tied assignment/PR-fix evidence, and window gauges align with PR 1252; however, the incomplete AgentWork state catalogue prevents the implementation from fully matching the referenced catalogue.

Tool Harness Findings

The harness confirmed maintainer-by-default route tiers, worker-token 403 handling, OIDC integration through authorizeRequest, canonical PR-fix ack parsing, shared queue-admission logic, and the relevant Prisma models; its tool budget was exhausted before a complete caller scan.

Unknowns or Needs Verification

The referenced PR 1237/PR 1239 source was not available for direct retrieval, so the exact catalogue fields were checked against the repository's docs/observability.md and the new snapshot contract rather than the external source.

Reviewed under the maintainer's blocker-only policy. The finding is technically valid: the snapshot contract/catalogue names an AgentWork checkpoint distribution that this slice does not expose. That is an observability completeness gap, not a material correctness, security, ownership, data-integrity, or irreversible-behavior defect. The endpoint's core purpose remains satisfied and the missing dimension is suitable for follow-up/dogfooding rather than a merge blocker.