Skip to content

ci: run the full-suite job on docs-only PRs, where doc guards live - #511

Merged
cdeust merged 1 commit into
mainfrom
ci/doc-contract-tests-on-docs
Sep 8, 2026
Merged

cdeust merged 1 commit into
mainfrom
ci/doc-contract-tests-on-docs

Conversation

@cdeust

@cdeust cdeust commented Sep 8, 2026

Copy link
Copy Markdown
Owner

Root cause of the #509 → #510 incident. #510 fixed the symptom; this fixes why it could reach main at all.

The hole

The changes filter classifies *.md out of code:

code:
  - '**'
  - '!{*.md,docs/**/*.md,…}'

So a README-only PR sets code=false and every test job is skipped. But part of that suite takes documentation as its input: tests_py/scripts/test_codex_plugin_contract.py reads README.md and asserts on the canonical published identities — the hypermnesia-mcp-viz anchor, the viz/spec migration strings.

The guards were switched off exactly when their subject changed. #509 was a README-only diff, every test job skipped, it merged green, and the push to main went red on that test across five jobs (run 34238410970 — 7663 passed, 1 failed).

The fix, and why this shape

docs is added to test-sqlite's predicate only — not the 3.10–3.13 matrix.

That job already runs THE FULL SUITE (its own pytest step says so) in one ~5-minute job, so it covers the doc-contract tests at the smallest cost that actually closes the gap. Waking the whole matrix on every prose edit would undo the CI-efficiency work of #475–#481; doing nothing leaves the guard defeated. This is the middle that holds both.

The part I nearly got wrong

scripts/check_ci_gate_results.py mirrors ci.yml's predicates — _required_jobs computes full from ("code", "deps", "workflows"), and its own check_policy docstring requires the workflow and the executable policy to stay "in exact agreement".

Changing only the workflow would have left two sources of truth disagreeing about when test-sqlite may skip: the job would run, CI would pass, and the policy checker would still quietly bless a future regression that skipped it again. Exactly the duplicated-rule anti-pattern this repo builds against (craftsmanship_layer_table.py parses the table rather than copying it).

So _required_jobs gains the same condition, and both files carry the same sourced note pointing at each other. Caught by re-reading my own diff, not by CI.

Tests

Its two docs-only tests are brought to the new policy, and the first becomes a named regression test:

def test_docs_only_still_requires_the_full_suite_job(self):
    allowed = gate.CONDITIONAL_JOBS - {"test-sqlite"}
    needs = _skip(_needs("docs"), allowed)
    self.assertEqual(gate.check(needs, "pull_request", _event()), [])

    needs["test-sqlite"]["result"] = "skipped"
    self.assertTrue(gate.check(needs, "pull_request", _event()),
                    "skipping test-sqlite on a docs-only PR must not be justified")

It asserts the exact hole #509 fell through is now closed — a mechanical guard, not a comment asking future readers to be careful.

Verification

Check Result
actionlint 1.7.12 (the version pinned in ci.yml) on ci.yml clean
scripts/check_ci_gate_complete.py OK
test_check_ci_gate_results.py + test_check_ci_gate_complete.py 38 passed, 159 subtests
check_craftsmanship.py --base origin/main OK
ruff check, ruff format --check, check_doc_claims.py clean

This PR is its own proof: it touches .github/**, so workflows=true — the test jobs run here, unlike on #509.

🤖 Generated with Claude Code

https://claude.ai/code/session_01StMBvNd7eVJGtpnC2zNsx1


Generated by Claude Code

Root cause of the #509 -> #510 incident. The `changes` filter classifies
'*.md' out of `code`:

    code:
      - '**'
      - '!{*.md,docs/**/*.md,...}'

so a README-only PR sets code=false and every test job is skipped. But
part of that suite takes documentation as its INPUT:
tests_py/scripts/test_codex_plugin_contract.py reads README.md and
asserts on the canonical published identities (the hypermnesia-mcp-viz
anchor, the viz/spec migration strings).

The guards were therefore switched OFF exactly when their subject
changed. #509 was a README-only diff, every test job skipped, it merged
green, and the push to main went red on that test across five jobs (run
34238410970 — 7663 passed, 1 failed). #510 fixed the symptom; this fixes
the reason it could reach main at all.

Adds `docs` to test-sqlite's predicate only, not to the 3.10-3.13 matrix:
test-sqlite already runs THE FULL SUITE in one ~5-minute job, so it
covers the doc-contract tests at the smallest cost that closes the gap.
Waking the whole matrix on every prose edit would undo #475-#481.

scripts/check_ci_gate_results.py mirrors ci.yml's predicates — its own
check_policy docstring requires the workflow and the executable policy to
stay "in exact agreement". Changing only the workflow would have left two
sources of truth disagreeing about when test-sqlite may skip, so
_required_jobs gains the same condition and both files carry the same
note. Caught by re-reading the diff, not by CI.

Its two docs-only tests are updated to the new policy, and the first is
turned into a named regression test that asserts skipping test-sqlite on
a docs-only PR is NOT justified — the exact hole #509 fell through.

Verified: actionlint 1.7.12 (the pinned version) clean on ci.yml;
check_ci_gate_complete.py OK; both gate test files 38 passed, 159
subtests; craftsmanship, ruff check, ruff format, doc-claims all clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01StMBvNd7eVJGtpnC2zNsx1
@cdeust
cdeust marked this pull request as ready for review September 8, 2026 16:18
@cdeust
cdeust merged commit 9254e44 into main Sep 8, 2026
28 checks passed
@cdeust
cdeust deleted the ci/doc-contract-tests-on-docs branch September 9, 2026 07:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants