Run the merge head race through the real gateway and github-mcp-server

[arpanghoshal]

A reader of github/github-mcp-server#3230 ran the PR-merge head race against Control.execute with a fake provider and asked which row changes through the gateway. research/github-merge-head-race/ answers that through the real stack: ctrlrun gateway → github-mcp-server 1.14.0 (http) → a fake GitHub REST API that keeps the merge endpoint's sha semantics. Eleven scenarios, three with no gateway as the control. Results are from ctrlrun[gateway]==0.12.2 off PyPI and match main at 1c02d1e.

What it found:

• Without expectedHeadSha, an approved merge lands at a head nobody approved. The gateway has no precondition recheck (SPEC-v0.7 §6.4).
• With it, GitHub refuses the stale merge. The SHA is part of the approval hash, so an agent that changes it gets a new approval request.
• github-mcp-server reports a GitHub 409 and a dropped connection the same way (isError + text), so the gateway records both ambiguous.
• mcp: {not_executed_on_error: true} is false for this server: it recorded failed for a merge that landed.
• Requiring expectedHeadSha works only through the absent-argument fallback: Gateway: an absent argument is not a typo, so let a policy require one #239.
• The gateway's pre-check refusals leave no receipt and no event: Gateway pre-check refusals leave no receipt and no event #240.

The second commit is the github-merge-agent recipe, extracted from CTRLRun/ctrlrun-docs by render_cookbook.py; its page is in the companion docs PR.

Tests: test_cookbook.py, test_examples.py, test_repository_signals.py and test_packaging.py pass (273). The full suite locally has 14 failures, none from this change: 12 in test_hop.py, fixed by #241, and 2 T221 connect-timeout tests that only fail on macOS 27, fixed on macos-27-full-backlog-rst. Merge #241 first, or this PR's CI goes red on the hop tests.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added a GitHub merge-agent example demonstrating how to require approval for a specific pull request version and refuse merges when the reviewed version has changed.

• Documentation

  • Added an experiment report covering merge behavior when a pull request changes during approval, including stale requests, lost responses, retries, and recovery.
  • Documented the experiment setup, tested scenarios, and observed results.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 77612824-95e7-40d1-ac57-8bd648d50808

---

📥 Commits

Reviewing files that changed from the base of the PR and between 3f97ed8 and 8740397.

---

📒 Files selected for processing (8)

• examples/cookbook/github-merge-agent/ctrlrun.yaml
• examples/cookbook/github-merge-agent/main.py
• pyproject.toml
• research/github-merge-head-race/README.md
• research/github-merge-head-race/fake_github.py
• research/github-merge-head-race/results/2026-10-05.json
• research/github-merge-head-race/results/2026-10-05.md
• research/github-merge-head-race/run.py

---

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

---

---

📝 Walkthrough

Walkthrough

Adds a cookbook example with a policy for approving pinned GitHub merge requests. Adds a research harness with a fake GitHub endpoint to compare direct and gateway merge scenarios, and records observed outcomes for head changes, stale SHA responses, approvals, and retries.

Changes

GitHub merge-head behavior

Layer / File(s)	Summary
Policy-controlled merge example examples/cookbook/github-merge-agent/ctrlrun.yaml, examples/cookbook/github-merge-agent/main.py	The policy allows pull-request reads and requires approval for merge calls with a nonempty expectedHeadSha. The script checks unpinned and pinned calls, approval, a changed head, and refused retries against a simulated upstream.
Experiment services and runner research/github-merge-head-race/fake_github.py, research/github-merge-head-race/run.py, pyproject.toml	The fake endpoint tracks merge calls and mutations, and can return stale-SHA or already-merged responses or drop a reply. The runner executes direct and gateway scenarios and collects effect, approval, receipt, and endpoint state. Ruff per-file ignores now include ANN401 for the research scripts.
Recorded outcomes and experiment notes research/github-merge-head-race/results/*, research/github-merge-head-race/README.md	The dated results record direct and gateway scenario outcomes. The README describes the setup, observations, limitations, and run commands.

Priority: ⬇️ Low

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Other

Sequence Diagram(s)

sequenceDiagram
  participant ExperimentRunner
  participant CTRLRunGateway
  participant GitHubMCPServer
  participant FakeGitHub
  participant SQLiteStateStore
  ExperimentRunner->>CTRLRunGateway: Send merge tool call
  CTRLRunGateway->>GitHubMCPServer: Forward merge call
  GitHubMCPServer->>FakeGitHub: Send GitHub merge request
  FakeGitHub-->>GitHubMCPServer: Return result, error, or dropped response
  GitHubMCPServer-->>CTRLRunGateway: Return tool result
  CTRLRunGateway-->>ExperimentRunner: Return response
  ExperimentRunner->>SQLiteStateStore: Read effect, action, and receipt records

Loading

---

Merge Risk: ⚪ Minimal · up to 87403

No actionable merge-blocking issue remains; the documented experiment can proceed with normal checks.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 6.38% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 47 functions across 3 files. (5 skipped: 5…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly identifies the main change: running the merge-head race experiment through the real gateway and github-mcp-server.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

---

Full details: Docstring Coverage

Explanation

Docstring coverage is 6.38% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 47 functions across 3 files. (5 skipped: 5 unsupported.)

---

---

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 2

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

---

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

---

🛠️ Fix failing CI checks 💡

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

---

_{Comment @coderabbitai help to get the list of available commands.}