Conversation
run-eval.mjs runs the working-tree (or --code-ref) sync code against a pinned docs commit in a throwaway worktree and scores the result against the case's expectations: pages that must, may, and must not change, edit budgets for secondary pages, and restatement of full rules outside their owner pages. The first case is base-std#234 (3820cf0), a NatSpec-only clarification of isAuthorized that the sync spread across ~10 pages (#2054, #2069, #2072). Co-authored-by: Toshi <toshi-noreply@coinbase.com>
- Rule 5: guides and concept pages change only when the source change makes a step, outcome, revert, or recommended setting on the page wrong; otherwise they are returned unchanged. Add 'one owner per fact': full behavior lives on the owning reference page; other pages link to it. - Rule 6: inventory documented surfaces, not mentions. A manifest entry that matches only a mention is not an intersection. Comment-only diffs clarify behavior; edit only statements they show to be wrong. - Step 4: polish only edited paragraphs; callouts only for real behavior changes, never for clarifications. - The anti-noop rule now applies to reference and changelog-entry pages that document the changed symbol, not to pages that only mention it. Co-authored-by: Toshi <toshi-noreply@coinbase.com>
…istorical pages - isCommentOnlyChange: a dispatch whose source diff only edits Solidity comments is a clarification. Symbol-mention routing then reaches only function-reference and interface-index pages, and the prompt is told the change type. - symbolRouteGate: pages found only by symbol mention are not routed when they are changelog entries for an earlier hardfork or upgrades/<fork>/ pages for a fork other than the newest. Path-routed pages are unaffected. Co-authored-by: Toshi <toshi-noreply@coinbase.com>
1505323-token-self-recipient replays base-std#232, a real behavior change (transfers, mints, and seizes to the token's own address now revert) that reaches src/ only as NatSpec. It guards the other direction from the 3820cf0 case: the five function references that list InvalidReceiver must change, and their added lines must state the new condition. - must_mention: a regex the added lines of a page must match, so a cosmetic edit to a required page does not count. - unlisted_pages: "warn" reports pages outside the lists without failing. - README: how to run the evals and what each case guards. Co-authored-by: Toshi <toshi-noreply@coinbase.com>
…nd retry streams From the base-std@1505323 eval (0/2 before, 6/6 after across both cases): - natspecDocumentedSymbols: a NatSpec hunk usually stops above the declaration it documents, so mint.mdx was skipped whenever the model's manifest happened not to name mint. Walk the post-change source (already fetched for changelog entries) from each changed comment line to the next declaration; function-reference pages for those members are always called. - isCommentOnlyChange: reference mocks (test/lib/mocks/) count as code. Base Std is interface-only, so NatSpec plus a mock change is a behavior change, not a clarification. The prompt no longer lets the model infer "clarification" from a comment-only diff slice; only the flag decides. - restoreCodeSamples: on guide pages, when no signature changed upstream, restore fenced code blocks (not mermaid) the model altered. A run had rewritten send-a-payout's functionName to "simulateContract". - normalizeForNoop: table re-padding alone is a noop. - llm/client: retry mid-stream overloaded/api errors twice with backoff; the SDK's maxRetries only covers the initial response, and one such error failed a whole sync in the eval. Co-authored-by: Toshi <toshi-noreply@coinbase.com>
Usage lives in the run-eval.mjs header and each case documents itself; the README only needs to say when to run it. Co-authored-by: Toshi <toshi-noreply@coinbase.com>
Collaborator
🟡 Heimdall Review Status
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed? Why?
The base-std docs sync over-edits clarifications and, in places, misses real behavior changes. #2054, #2069, and #2072 spread a NatSpec-only
isAuthorizedclarification (base-std#234) across 9–14 pages: guides, older-fork changelog entries, andupgrades/beryl/. This PR fixes the causes and adds an eval to measure them.Prompt (
llm/prompts.mjs):Routing (
index.mjs,release-utils.mjs):upgrades/<older fork>/pages are no longer routed by keyword. Pages named in the route table are unaffected.mint.mdxwas being skipped whenever the model's change summary missed it, and this also happens on master.Guards:
functionName: "sendPayouts"to"simulateContract".overloaded_error) are retried twice. The SDK'smaxRetriesonly covers the initial request.Notes to reviewers
eval/run-eval.mjsreplays a recorded dispatch against a pinned docs commit in a throwaway worktree and scores which pages changed.--code-ref mastergives a baseline. There are two cases:3820cf0-isauthorized-natspec(precision): only the owning reference page and its index row should change.1505323-token-self-recipient(recall): a real behavior change that appears insrc/only as NatSpec must still reach all 5 function references, and their added lines must state the new revert.How has it been tested?
Eval results, as passing runs out of total:
npm --prefix scripts run test:base-std-sync: 186 pass, 1 fail. The failure is "upstream docs tree routes to the pages the IA guidelines assign", which also fails on master.Screenshots
N/A (no user-facing changes)