[joryirving]

Summary

• New ci-failures scheduler job: files an issue when a workflow fails twice in a row on a repo's default branch, and closes it when that workflow goes green.
• Policy lives in a pure module (src/lib/ci-failure-ingestion.ts); the route is I/O around it.

Closes #931.

Why

pr-followup ingests a failing check when it happens on a pull request. The same failure on the default branch reached nothing — no PR, so nothing ingested it and nothing filed anything. A release build that broke after a merge showed on a dashboard and waited for someone to notice.

Most of the machinery already existed and pointed the wrong way: fetchRecentRunsAllWorkflows and fetchFailedJobLogExcerpt in github-ci.ts, and automation-sync.ts already fetching runs and default_branch for display.

Policy, and why each part

Decided on #931 rather than defaulted:

• Second consecutive failure files, never the first. One red run is weak evidence — network blips and rate limits clear themselves. Filing on the first would flood the queue, and a queue of issues nobody acts on trains people to ignore the label.
• Every workflow is eligible. The repeat rule is the noise filter. An allowlist would silently exclude whichever workflow nobody remembered to add, which is the failure this exists to catch.
• A green run closes the issue, commenting which run cleared it. An auto-filed issue nobody auto-closes goes stale the moment CI recovers.
• A recurrence files fresh and links the closed one. The earlier fix did not hold — new information, and it re-enters the queue with its own priority rather than reopening a body describing a different occurrence.

No new state

Both "failed twice in a row" and "already filed" are answered from GitHub: consecutiveness from the workflow's own recent runs, deduplication from a <!-- dispatch-ci-failure:<sig> --> marker in the issue body, in the same shape as the AI reviewer's. No Prisma model, no migration, and an instance that has never seen a repo behaves identically to one that has.

The signature normalises run-specific noise — ids, timestamps, hex digests, durations — before hashing. Without that no two failures ever look consecutive and nothing is ever filed; there is a test for exactly that.

Agnostic

Nothing about Foreman, gate profiles or lanes appears anywhere. "A failed workflow run on the default branch" is a plain GitHub concept. Labels come from DISPATCH_CI_FAILURE_LABELS rather than being hardcoded to one deployment's taxonomy.

Cost

The expensive calls — job listing and log fetch — happen only on the repeated-failure path, so a healthy repo costs one runs listing plus one issue listing per pass. Default interval 30m, per-repo isolation so one unreachable repo cannot abort the pass, and its own ci-failures lock key so it excludes only itself.

Verification

• npx vitest run — 2528 passed, 5 skipped (29 new, covering signature stability and normalisation, the classifier including cancelled/skipped, every branch of the decision table, and issue rendering)
• typecheck, lint and build clean; /api/ci-failures/sync appears in the route manifest

https://claude.ai/code/session_01YSuDvZq9ncvyX85Uzx3cQh

[its-saffron]

AI Automated Review

Full PR review.

Analysis engine: MiniMax-M2.7@https://litellm.jory.dev/v1 (anthropic) — primary route

Recommendation: Approve

This PR is a well-structured, thoroughly tested implementation of the ci-failures scheduler job that files issues from repeated default-branch workflow failures. The implementation directly addresses all acceptance criteria from linked issue PR 931.

---

Change-by-Change Findings

New Core Policy Module (src/lib/ci-failure-ingestion.ts)

The pure policy module is the centerpiece. It:

• Documents all noise-policy decisions explicitly (second-failure-only, all-workflows-eligible, green-run-closes, recurrence-files-fresh)
• Implements computeFailureSignature with log normalization (hex digests, timestamps, durations stripped) so consecutive failures match
• Provides classifyWorkflow to classify states as healthy, first-failure, repeated-failure, or unknown
• Implements decideAction as a pure function covering every branch of the decision table

Observations:

• The module header comment is the canonical policy reference and aligns verbatim with issue PR 931's "The part that decides whether this is any good" section
• cancelled, skipped, neutral, and timed_out conclusions are correctly treated as unknown (not failure), per the test treats cancelled and skipped as unknown, not failure
• In-progress runs are excluded from groupDefaultBranchRuns with a clear comment explaining why (a queued run between two failures would break consecutiveness)
• No new Prisma model or migration — "already filed" is answered from GitHub issue bodies via the <!-- dispatch-ci-failure:<sig> --> marker

New API Route (src/app/api/ci-failures/sync/route.ts)

The I/O wrapper is correct:

• Auth: authorizeRequest before anything
• Rate limit: 10 req/min per actor, returns 429 when exceeded
• Lock: acquireLock("ci-failures") with 409 on conflict — uses the new "ci-failures" sync type from sync-lock.ts
• Per-repo isolation: try/catch around each repo so one unreachable repo cannot abort the pass
• Expensive calls deferred: Job listing and log fetch happen only on the repeated-failure path; healthy repos cost one runs-listing call
• Local view updated: filed.push(...) after createIssue so a second workflow in the same repo with the same signature does not duplicate

GitHub Utilities (src/lib/github-issues.ts)

The new createIssue function is a thin wrapper around the GitHub API with retry via fetchWithRetry. It returns { number, html_url } which is all the caller needs.

Scheduler Integration (src/lib/scheduler.ts, src/lib/sync-lock.ts)

The ci-failures job:

• Path: /api/ci-failures/sync
• Default interval: 30 minutes (DEFAULT_CI_FAILURES_INTERVAL_MS = 30 * 60 * 1000)
• Configurable via DISPATCH_CI_FAILURES_INTERVAL_MS
• Lock key: "ci-failures" — its own lock key means it cannot starve other jobs

Tests (src/lib/ci-failure-ingestion.test.ts)

29 new tests covering:

• Signature stability and normalization (hex digests, timestamps, durations stripped)
• classifyWorkflow including cancelled/skipped/neutral/timed_out treated as unknown
• Every branch of decideAction — file, skip (already filed), skip (first failure), close, skip (green nothing open), skip (no signature)
• buildIssueDraft rendering (title, body, marker, supersedes link, log excerpt cap at ~6000 chars)
• buildCloseComment

---

Linked Issue Fit

Issue PR 931 Acceptance Criterion	Status
"A failed default-branch run files exactly one issue, carrying the failing job's name and a log excerpt sufficient to act on."	✅ buildIssueDraft includes job name and log excerpt
"The same failure recurring does not file a second issue."	✅ decideAction checks open issues by signature
"The noise policy above is decided and implemented, not left to a default."	✅ All four decisions documented in module header and PR body
"No Foreman-specific concept appears anywhere in the implementation."	✅ No Foreman, gate profiles, or lane references
"The labels applied should come from configuration"	✅ DISPATCH_CI_FAILURE_LABELS env var

Noise policy decisions match the issue's intent:

• First failure → no filing (transient filter)
• All workflows eligible (not an allowlist)
• Green run auto-closes with a comment naming the clearing run
• Recurrence files fresh, linking the closed issue as "did not hold"

---

Standards Compliance

Standard	Status
No secrets committed	✅ No .env or secrets in diff
Error handling uses error instanceof Error pattern	✅ Route catches errors per-repo; github-issues.ts throws new Error(...)
Return appropriate HTTP status codes	✅ 401 unauthorized, 409 lock conflict, JSON for success
Validation before database operations	✅ Auth checked before lock; lock checked before any work
API routes return JSON	✅ NextResponse.json(...) for all responses
No agent-specific names in generic docs	✅ N/A — this is a GitHub concept
Scheduler pattern matches existing jobs	✅ Same structure as pr-followup, groomer, etc.
Per-repo isolation for sync passes	✅ try/catch around each repo iteration
New sync type has its own lock key	✅ "ci-failures" added to SyncType and LOCK_KEY_BY_TYPE
Environment variable documented	✅ README.md updated with DISPATCH_CI_FAILURES_INTERVAL_MS and DISPATCH_CI_FAILURE_LABELS

---

Unknowns / Needs Verification

None. The corpus is complete: linked issue context is present, full diff is available, CI passed, tests cover all decision branches, and the implementation matches all stated acceptance criteria.