Review agent should check PR CI status before finalizing verdict

[fullsend-ai-retro]

What happened

On PR #211, the review bot (run 32017692283) issued request-changes with 10 findings (2 high, 4 medium, 4 low) focusing on stale references and config drift. Meanwhile, all functional tests were failing with HTTP 403 (GitHub connectivity broken from sandbox) — a critical runtime regression the bot did not mention. The human reviewer (waynesun09) discovered this by downloading CI artifacts and reading test output. The review agent has GitHub API access (it fetches file contents via gh api) but never checks statusCheckRollup or PR check status. The only CI-status-aware code in the repo is the merge-queue skill's await-and-enqueue.sh — the review pipeline has no CI integration at all.

What could go better

If the review agent had checked CI status before finalizing its verdict, it would have discovered that functional tests were 100% failing. This context would have: (1) elevated the review from config-drift findings to a critical connectivity regression, (2) prevented the bot from claiming verification of things CI proved were broken, (3) given sub-agents runtime signal to correlate with their static analysis. The human reviewer's most impactful contributions all came from CI artifact analysis — the agent could have replicated this with a simple API call. Confidence is high that this would have changed the review outcome. Note: agents#394 (CI workflow correctness checks) is distinct — it proposes checking CI workflow YAML files for correctness, not checking CI run status.

Proposed change

Add a CI status check step to skills/pr-review/SKILL.md in the orchestrator instructions, after fetching the PR diff and before dispatching sub-agents:

1. Add an instruction: "Before dispatching sub-agents, check the PR's CI status by running gh pr checks $PR_NUMBER --repo $REPO_FULL_NAME or querying the statusCheckRollup via GraphQL. If any required checks are failing, prepend a prominent notice to the review body: '⚠ CI checks are failing — findings below should be evaluated in light of runtime failures.' Pass a one-line CI status summary (e.g., 'CI: 3/7 checks failing — functional-tests FAILURE, lint SUCCESS') to each sub-agent's context so they can correlate static findings with runtime failures."

2. In the review result schema (schemas/review-result.schema.json), consider adding an optional ci_status field (e.g., {"passing": 4, "failing": 3, "pending": 0}) so downstream consumers (post-review.sh, merge-queue) can see the CI state at review time.

3. In the severity-filtering logic in scripts/post-review.sh, consider auto-elevating the review action to request-changes if CI checks are failing, regardless of finding count — a passing review with failing CI sends a contradictory signal.

Validation criteria

The next 5 review runs on PRs with at least one failing CI check should mention the CI failure status in their review body. Reviews on PRs with all-green CI should not contain spurious CI warnings. Measure by checking the review body text of the next 10 review runs for CI status references and correlating with actual CI status at the time of review.

---

Generated by retro agent from #211

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 12:30 PM UTC · Completed 12:34 PM UTC

Commit: 7aa8587 · View workflow run →

[fullsend-ai-triage]

Triage Summary

Severity: High — this gap caused a human reviewer to do work the agent could have automated, and allowed a contradictory review signal on a PR with 100% test failure.

Relationship to #394: As noted in the issue, #394 (CI workflow YAML correctness checks) is distinct — it proposes checking CI workflow files for correctness, while this issue proposes checking CI run status. Both can proceed independently.

Implementation reference: The merge-queue skill's scripts/await-and-enqueue.sh already queries statusCheckRollup via gh pr view --json statusCheckRollup and parses check states with jq. This pattern can be adapted for the review orchestrator's new CI status check step.

Proposed test case:

1. PR with failing CI checks → review body contains CI warning,
   ci_status field shows failures, sub-agents receive CI summary
2. PR with all-green CI → no CI warning in review body
3. PR with pending checks → review notes checks still running
4. PR with no CI checks → graceful handling, no errors
5. post-review.sh: failing CI elevates approve/comment → request-changes

---

Labels: This is a feature request for the review agent, matching the 'review-agent' and 'feature' labels.

---

Next steps:

• /fs-code — agent creates a PR to implement this issue
• /fs-code <your instruction> — agent implements with your specific guidance

[fullsend-ai-prioritize]

RICE Priority Score: 2.7

Score breakdown

Dimension	Score	Reasoning
Reach	2	The review agent runs on every PR in the repo (and potentially across multiple repos in the org). All contributors who submit PRs and receive automated reviews are affected — when CI is failing, the current review produces contradictory signals that every PR author and reviewer must work around. This is a platform-wide tool affecting most active users.
Impact	1.5	This is a significant pain point: the review agent can approve or issue findings that contradict CI reality, as demonstrated on PR #211 where 100% of functional tests were failing but the bot never mentioned it. The human reviewer had to manually download CI artifacts to discover the regression. Fixing this eliminates contradictory signals and automates work humans currently do manually, but workarounds exist (checking CI status separately).
Confidence	0.9	Very high confidence. The issue is exceptionally well-documented with a specific incident (PR #211, run 32017692283), concrete proposed implementation across three files (SKILL.md, schema, post-review.sh), validation criteria, and the triage agent confirmed the existing pattern in await-and-enqueue.sh can be reused. Scope and affected components are clear.
Effort	1	Medium effort. The change touches three components (SKILL.md orchestrator instructions, review-result schema, post-review.sh), but each modification is well-scoped. The triage agent confirmed that the merge-queue skill's await-and-enqueue.sh already implements the statusCheckRollup query pattern, so this is largely adapting an existing pattern rather than building from scratch. Five test scenarios are defined but are straightforward to validate.

Formula: (2 x 1.5 x 0.9) / 1 = 2.7

[ifireball]

Closing this as not planned - just like its the job of the human the wrote a PR to inspect CI and fix issues it should be the fix agent's job to inspect and fix the CI (#1333 ). A reviewer should only focus on its own review criteria.