docs(#7307): record PR 6935 as runtime-comparison review evidence

[fullsend-ai-coder]

Summary

Add a runtime/configuration-comparison evidence entry for PR #6935 to docs/problems/review-autonomy-evidence.md, kept distinct from the agent-vs-human split, plus a re-evaluation checkpoint for the #7116 review-runtime switch.

Related Issue

#7307

Changes

• New subsection Runtime and configuration comparison evidence recording the byte-identical fix(#6932): move seenResult flag after successful unmarshal #6935 re-review: claude/opus LGTM (2026-09-02, $2.50) vs pi/sonnet Medium test-adequacy finding (2026-09-14, $2.00), with the intervening chore(config): route agents across runtimes and models #7116 config change, N=1 caveat, and agents#1140 variance pointer.
• Re-evaluation checkpoint (20 test-adequacy-relevant review runs on this repo, or 2026-10-15) whose result should feed adaptive-agent-selection.md as a version-tagged fitness data point.
• Reciprocal links in adaptive-agent-selection.md, testing-agents.md, code-review.md, and trustworthiness-evidence.md.

Testing

• make lint-md-links passes (lychee offline, including fragments)
• Direct execution of applicable pre-commit hooks (trailing-whitespace, EOF, merge-conflict, private-key, gitleaks, lint-docs-links, lychee, lint-broken-symlinks)
• make lint (pre-commit) — skipped: hook repos returned HTTP 403 in this sandbox
• Tests added/updated for new or modified logic — N/A (docs-only)

Checklist

• PR title follows Conventional Commits (correct type, ! for breaking changes)
• Commits are signed off (DCO) — human and human-directed agent sessions only (autonomous agent commits are exempt)
• I wrote this contribution myself and can explain all changes in it

---

Closes #7307

Post-script verification

• Branch is not main/master (agent/7307-runtime-review-evidence)
• Secret scan passed (gitleaks — e0b37823c94a11970cf125996f2cb24a1ec2bd1f..HEAD)
• PR body secret scan passed (gitleaks — no-git)

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 10:03 AM UTC · Completed 10:19 AM UTC

Commit: b6c65fd · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $3.98

[github-actions]

Site preview

Preview: https://b9d884a8-site.fullsend-ai.workers.dev

Commit: ae5a83279aff3fb72d5f18962d6161969ac23ab5

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[fullsend-ai-review]

Risk Assessment: low (1/5)

Details

Tier 1 signals are unchanged from the prior assessment on this PR, git history shows unremarkable churn with no fix/revert activity on the substantively-edited files, and the issue remains well-scoped with matching acceptance criteria, so the prior low score of 1 is preserved.

Previous run

Risk Assessment: low (1/5)

Details

Docs-only PR (5 files, 54 lines) by a bot author with no protected/security-sensitive paths, no CI or dependency changes, moderate historical churn on the touched evidence doc, and a well-scoped linked issue with matching acceptance criteria, yielding a composite near the low end of the scale.

[fullsend-ai-review]

Review

Findings

Low

• [scope-authorization] docs/problems/testing-agents.md:35 — Issue Capture PR #6935 as runtime-migration evidence in review-autonomy-evidence.md and add a re-evaluation checkpoint for PR #7116 #7307 authorizes edits to review-autonomy-evidence.md and a fitness-function link in adaptive-agent-selection.md. The PR additionally adds reciprocal backlinks in testing-agents.md, code-review.md, and trustworthiness-evidence.md, which are not named in the issue's proposed change or validation criteria. This is small, docs-only surplus scope consistent with the repo's cross-referencing conventions (AGENTS.md), not a defect — no action required.

• [sub-agent-failure] N/A — The challenger sub-agent returned an empty adjudicated_findings array against a non-empty input set (it judged all three intent-coherence findings to be non-defect confirmations). Per policy this is treated as a challenger failure rather than trusted wholesale, so this review falls back to the pre-challenger finding set (the intent-coherence findings above, with info-level items suppressed by the severity threshold).

Both prior-review findings (technical-inaccuracy on the PR #6935 fixture arithmetic and incomplete-doc on the missing trustworthiness-evidence.md backlink) are resolved in this revision: the arithmetic is now correctly qualified as historical ("as originally reviewed in PR #6935, before a follow-up commit corrected it"), and a contextual backlink to review-autonomy-evidence.md#runtime-and-configuration-comparison-evidence was added at the "Historical track record" passage in trustworthiness-evidence.md.

Previous run

Review

Findings

Low

• [technical-inaccuracy] docs/problems/review-autonomy-evidence.md:142 — The entry states the TestParseClaudeStreamMalformedResultFallsBackToTokensEvent fixture summed to 3000+1600+400+100=5100, at/above tokenThreshold (5000). That arithmetic and threshold are correct and match what PR fix(#6932): move seenResult flag after successful unmarshal #6935's second review actually flagged (confirmed against internal/runtime/claude_progress.go and issue Capture PR #6935 as runtime-migration evidence in review-autonomy-evidence.md and add a re-evaluation checkpoint for PR #7116 #7307). However, it is not what the named test contains on current main: the test now uses 1000+300+200+50=1550, explicitly kept under tokenThreshold by a follow-up commit (test(#6932): fix fixture so regression test actually exercises deferred TokensEvent) that landed within the same PR fix(#6932): move seenResult flag after successful unmarshal #6935 branch before merge, in direct response to the second review's finding. The present-tense "the new test's fixture summed to 5100" with no qualifier will look wrong to anyone who opens the cited test today.
  Remediation: Qualify the 5100 sum as the fixture as originally reviewed in PR fix(#6932): move seenResult flag after successful unmarshal #6935 (before the follow-up commit corrected it), and note that the current test on main uses 1000+300+200+50=1550 — i.e., the hole the second review flagged was fixed in-tree before/at merge.

• [incomplete-doc] docs/problems/trustworthiness-evidence.md:78 — The new "Runtime and configuration comparison evidence" subsection's central point is that a runtime/model configuration change (PR chore(config): route agents across runtimes and models #7116) invalidated a prior review-quality baseline on a byte-identical diff. trustworthiness-evidence.md already substantively discusses this exact concept under "Historical track record" ("A configuration change resets the track record for the dimensions affected by that change.") and "Configuration drift detection". This PR added contextual body backlinks at the motivating passages in adaptive-agent-selection.md, testing-agents.md, and code-review.md, but only updated the top-level "Related" list in trustworthiness-evidence.md — the substantive passage was not backlinked.
  Remediation: Add a single contextual backlink from the "Historical track record" (or "Configuration drift detection") passage to review-autonomy-evidence.md#runtime-and-configuration-comparison-evidence, noting that PR fix(#6932): move seenResult flag after successful unmarshal #6935/chore(config): route agents across runtimes and models #7116 is a concrete instance of a configuration change resetting review-quality evidence.

---

Next steps:

• /fs-fix — agent addresses review findings automatically
• /fs-fix <your instruction> — agent fixes with your specific guidance
• Push commits directly — review re-runs automatically on push
• /fs-fix-stop — disable automatic fix runs for this PR

[fullsend-ai-coder]

🤖 Finished Fix · ✅ Success · Started 10:21 AM UTC · Completed 10:27 AM UTC

Commit: b6c65fd · View workflow run →

Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $0.71

[fullsend-ai-coder]

🔧 Fix agent — iteration 1 (bot-triggered)

Addressed both low-severity review findings: qualified the fixture-arithmetic claim in review-autonomy-evidence.md to match the corrected test on main, and added the missing contextual backlink in trustworthiness-evidence.md's Historical track record section pointing to the runtime/configuration comparison evidence.

Fixed (2):

1. technical-inaccuracy: fixture sum in review-autonomy-evidence.md:142 no longer matches the test on main (docs/problems/review-autonomy-evidence.md): Verified against internal/runtime/claude_progress_test.go and git history: a same-branch follow-up commit ('test(Move seenResult flag after successful result event unmarshal #6932): fix fixture so regression test actually exercises deferred TokensEvent') corrected the fixture before PR fix(#6932): move seenResult flag after successful unmarshal #6935 merged. Qualified the 5100 sum as the fixture 'as originally reviewed... before a follow-up commit corrected it', and added that the fixture on main today sums to 1000+300+200+50=1550, kept below tokenThreshold.
2. incomplete-doc: trustworthiness-evidence.md's substantive 'Historical track record' passage lacked a contextual backlink to the new runtime-comparison evidence section (docs/problems/trustworthiness-evidence.md): Added a contextual backlink from the 'Track records are meaningful only when the configuration is stable...' sentence to review-autonomy-evidence.md#runtime-and-configuration-comparison-evidence, noting PR chore(config): route agents across runtimes and models #7116/fix(#6932): move seenResult flag after successful unmarshal #6935 as a concrete instance, matching the backlink style already used in adaptive-agent-selection.md and testing-agents.md.

Tests: passed

Next steps:

• /fs-review — request a re-review of the changes
• /fs-fix <your instruction> — run another fix pass with specific guidance
• Push commits directly — review re-runs automatically on push
  _{Updated by fullsend fix agent}

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 10:29 AM UTC · Completed 10:41 AM UTC

Commit: ae5a832 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $3.06

Superseded by updated review

[fullsend-ai-review]

[low] scope-authorization

Issue #7307 authorizes edits to review-autonomy-evidence.md and a fitness-function link in adaptive-agent-selection.md. The PR additionally adds reciprocal backlinks in testing-agents.md, code-review.md, and trustworthiness-evidence.md, which are not named in the issues proposed change or validation criteria. This is small, docs-only surplus scope consistent with the repos cross-referencing conventions (AGENTS.md), not a defect.

Suggested fix: No action required if the team accepts backlink hygiene as in-scope for subsection-level additions; otherwise confirm this convention explicitly in AGENTS.md.