Skip to content

docs: grade the letter-count examples with an exact match - #718

Open
Mike-7777777 wants to merge 1 commit into
hud-evals:mainfrom
Mike-7777777:fix/readme-count-grader
Open

Mike-7777777 wants to merge 1 commit into
hud-evals:mainfrom
Mike-7777777:fix/readme-count-grader

Conversation

@Mike-7777777

@Mike-7777777 Mike-7777777 commented Oct 14, 2026 •

Copy link
Copy Markdown

Problem

The letter-count grader in the README checks str(word.count(letter)) in answer, which is a substring match. For strawberry (3 r's), the answers "13", "33" and "2 or 3" all get a reward of 1.0.

The same check appears in the quickstart, docs/skill.md, and twice in the "Creating Environments" guide. The guide's version is also the snippet shown in the "What is a good task?" post.

Change

Compare the stripped answer instead, the same check environments/blank/env.py already uses:

yield 1.0 if (answer or "").strip() == str(word.count(letter)) else 0.0

The README and quickstart prompts already say "Reply with just the number." The docs/skill.md and guide prompts didn't, so I added the same sentence there. Otherwise a correct answer like "3 times" would score 0 with nothing in the prompt asking for a bare number.

Trade-off: "3." and "The answer is 3." now score 0, which matches what the prompt asks for. If you'd rather be lenient, hud.graders.numeric_match(answer, word.count(letter)) checks the first number in the answer. It also rejects "13", at the cost of one import in the snippet. Happy to switch.

Check

  • Ran the README snippet with the local hud install and drove the generator directly, no model calls: "3" scores 1.0; "13", "2 or 3" and None score 0.0; "2" for blueberry scores 1.0.
  • Ran each edited grader line from the four files on "3", " 3\n", "3.", "The answer is 3.", "13", "33", "2 or 3", "not 2, it's 3", "three", "" and None. Only "3" and " 3\n" score 1.0.
  • ruff format --check . and ruff check . pass.

Side note, not changed here: the quickstart says hud init my-env --template blank creates a tasks.py containing count_letter(word, letter). The blank template actually ships count(sentence, letter) in env.py, and its tasks.py imports it.


Note

Low Risk
Docs-only changes to example graders and prompts; no runtime or SDK behavior is modified.

Overview
Documentation examples for the letter-count task now grade with an exact match on the stripped answer ((answer or "").strip() == str(count)) instead of a substring check (str(count) in answer), so false positives like "13" or "2 or 3" no longer receive full reward.

Where prompts did not already ask for a bare number, docs/skill.md and creating-an-environment.mdx (both snippets) add "Reply with just the number." so grading stays aligned with what users are told to return. README, quickstart, and the environment guide’s case-insensitive count template are updated the same way.

Reviewed by Cursor Bugbot for commit 18d2bc3. Bugbot is set up for automated code reviews on this repo. Configure here.

The README, quickstart, docs/skill.md and the environment guide graded
with str(count) in answer, so "13" scored 1.0 for strawberry. Compare
the stripped answer instead, as environments/blank does, and ask for
just the number where the prompt did not already.
@Mike-7777777
Mike-7777777 force-pushed the fix/readme-count-grader branch from 8b7bdb2 to 18d2bc3 Compare October 14, 2026 12:35
@Mike-7777777 Mike-7777777 changed the title docs: make the README letter-count grader reject answers that only contain the count docs: grade the letter-count examples with an exact match Oct 14, 2026
@Mike-7777777
Mike-7777777 marked this pull request as ready for review October 14, 2026 12:35

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant