Repository navigation
docs: grade the letter-count examples with an exact match - #718
Open
Mike-7777777 wants to merge 1 commit into
Open
Mike-7777777 wants to merge 1 commit into
Mike-7777777 wants to merge 1 commit into
Conversation
The README, quickstart, docs/skill.md and the environment guide graded with str(count) in answer, so "13" scored 1.0 for strawberry. Compare the stripped answer instead, as environments/blank does, and ask for just the number where the prompt did not already.
Mike-7777777
force-pushed
the
fix/readme-count-grader
branch
from
October 14, 2026 12:35
8b7bdb2 to
18d2bc3
Compare
Mike-7777777
marked this pull request as ready for review
October 14, 2026 12:35
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The letter-count grader in the README checks
str(word.count(letter)) in answer, which is a substring match. Forstrawberry(3 r's), the answers"13","33"and"2 or 3"all get a reward of 1.0.The same check appears in the quickstart,
docs/skill.md, and twice in the "Creating Environments" guide. The guide's version is also the snippet shown in the "What is a good task?" post.Change
Compare the stripped answer instead, the same check
environments/blank/env.pyalready uses:The README and quickstart prompts already say "Reply with just the number." The
docs/skill.mdand guide prompts didn't, so I added the same sentence there. Otherwise a correct answer like "3 times" would score 0 with nothing in the prompt asking for a bare number.Trade-off:
"3."and"The answer is 3."now score 0, which matches what the prompt asks for. If you'd rather be lenient,hud.graders.numeric_match(answer, word.count(letter))checks the first number in the answer. It also rejects"13", at the cost of one import in the snippet. Happy to switch.Check
hudinstall and drove the generator directly, no model calls:"3"scores 1.0;"13","2 or 3"andNonescore 0.0;"2"forblueberryscores 1.0."3"," 3\n","3.","The answer is 3.","13","33","2 or 3","not 2, it's 3","three",""andNone. Only"3"and" 3\n"score 1.0.ruff format --check .andruff check .pass.Side note, not changed here: the quickstart says
hud init my-env --template blankcreates atasks.pycontainingcount_letter(word, letter). The blank template actually shipscount(sentence, letter)inenv.py, and itstasks.pyimports it.Note
Low Risk
Docs-only changes to example graders and prompts; no runtime or SDK behavior is modified.
Overview
Documentation examples for the letter-count task now grade with an exact match on the stripped answer (
(answer or "").strip() == str(count)) instead of a substring check (str(count) in answer), so false positives like"13"or"2 or 3"no longer receive full reward.Where prompts did not already ask for a bare number,
docs/skill.mdandcreating-an-environment.mdx(both snippets) add "Reply with just the number." so grading stays aligned with what users are told to return. README, quickstart, and the environment guide’s case-insensitivecounttemplate are updated the same way.Reviewed by Cursor Bugbot for commit 18d2bc3. Bugbot is set up for automated code reviews on this repo. Configure here.