Skip to content

fix grammar rejection cache: store reject indices, not candidate pointers - #2434

Open
Reithan wants to merge 1 commit into
LostRuins:concedo_experimentalfrom
Reithan:fix-grammar-memoization-bugs
Open

Reithan wants to merge 1 commit into
LostRuins:concedo_experimentalfrom
Reithan:fix-grammar-memoization-bugs

Conversation

@Reithan

@Reithan Reithan commented Sep 5, 2026 •

Copy link
Copy Markdown

Overview

Fix a use-after-free in the grammar rejection cache (candidates_memos) in llama_grammar_reject_candidates_for_stack.

The cache stored full llama_grammar_candidate structs. Each one holds a raw code_points pointer into a decode buffer owned by the caller, which is freed when the sampling step ends. Since #2431 switched the cache key to a content hash, identical candidate lists now hit across steps, so a hit returns candidates whose code_points point into a previous step's freed buffer.

This PR is rebased onto current concedo_experimental and builds on #2431. #2431 already provides the content-based key and the cache write in the TOKEN / TOKEN_NOT branch, so this PR now only changes what is stored.

Changes

  • Cache values are the indices of rejected candidates (std::vector<size_t>), not full candidate structs.
  • On a cache hit, the rejects are rebuilt from the current call's candidates, so code_points always points into live data.
  • Both cache writes (the TOKEN / TOKEN_NOT early return and the char-rule path) store indices through one small helper.

Testing

koboldcpp has no grammar unit tests, so I checked this with a scratch driver linked against koboldcpp_default.so. It calls llama_grammar_reject_candidates_for_stack three times with identical content but freshly allocated code-point buffers, so the third call is a cache hit. It covers TOKEN, TOKEN_NOT and char rules.

  • This branch: the rejects match the uncached result, and every returned code_points points into the current call's buffers.
  • Current concedo_experimental: the rejects match, but on a cache hit code_points points into the previous call's freed buffers (fails for all three rule types).
  • This branch with the token-rule cache write removed: cache hits return an empty reject set for TOKEN and TOKEN_NOT (the bug Fix two memoization bugs in llama_grammar_reject_candidates_for_stack #2431 fixed).

End-to-end smoke test with Qwen3.5-0.8B on CPU: JSON-schema, word-alternation and character-class grammars, 3 seeded runs each, all grammar-valid with no asserts. The JSON outputs are identical to concedo_experimental for the same seeds.

Requirements

  • I have read and agree with the contributing guidelines.
  • AI usage disclosure: AI was used to identify the bug, draft the fix and write the test driver; the change was reviewed and approved by the author before push.

@LostRuins
LostRuins changed the base branch from concedo to concedo_experimental September 5, 2026 12:48
@LostRuins

Copy link
Copy Markdown
Owner

you should PR to the experimental branch, and i think this PR conflicts with your previous one. Also I don't actually think there is any performance improvement in fact I think that this actually performs worse.

@Reithan

Reithan commented Sep 6, 2026

Copy link
Copy Markdown
Author

Let me re-target to experiment and I'll mark this as draft and run a few more tests. 👍

@Reithan
Reithan marked this pull request as draft September 6, 2026 02:54
@reithan-local-agent
reithan-local-agent Bot force-pushed the fix-grammar-memoization-bugs branch from c1211a7 to 8788edb Compare October 10, 2026 02:48
@Reithan Reithan changed the title fix grammar rejection cache: content-based keys, index-only values fix grammar rejection cache: store reject indices, not candidate pointers Oct 10, 2026
…ters

Cached candidates held code_points pointers into decode buffers freed after
each sampling step, so content-keyed cache hits (since LostRuins#2431) returned
dangling pointers. Store reject indices instead and rebuild the rejects from
the current candidates on a hit, in both the token-rule early return and the
char-rule path.
@Reithan
Reithan force-pushed the fix-grammar-memoization-bugs branch from 8788edb to e069a43 Compare October 10, 2026 03:02
@Reithan

Reithan commented Oct 10, 2026

Copy link
Copy Markdown
Author

I benchmarked this to check the "performs worse" concern and couldn't find a regression.

Setup. I used CPU-only make koboldcpp_default builds, Qwen3.5-0.8B Q8_K_XL, 3 threads and greedy sampling (top_k=1). Each mode ran 5 times for 256 tokens. Scratch timers around SampleLogits and sample_grammar (not part of this PR) measured the time. Builds compared:

  • concedo_experimental at 97670a1 (this PR's base)
  • this PR at e069a43
  • the original PR head and its old base, to check whether the branch being far behind concedo_experimental explained the difference

All builds produced byte-identical output in every mode.

The heavy grammar is a hand-written JSON-like grammar with nested bounded repetitions and character classes, and no token rules:

heavy grammar
root ::= "[" ws entry (ws "," ws entry){0,40} ws "]"
entry ::= "{" ws "\"name\":" ws str ws "," ws "\"tags\":" ws "[" ws (str (ws "," ws str){0,5})? ws "]" ws "," ws "\"score\":" ws num ws "}"
str ::= "\"" [a-zA-Z0-9 _'-]{1,24} "\""
num ::= "-"? [0-9]{1,6} ("." [0-9]{1,3})?
ws ::= [ \t\n]{0,4}

Median microseconds per generated token:

Mode base: grammar PR: grammar base: whole sampler PR: whole sampler
no grammar – – 1956 1872
JSON schema 1103 1066 3346 3224
heavy grammar 893 888 3100 3067

The original PR against its old base was just as flat on grammar time: 1074 against 1110 on the JSON schema and 914 against 892 on the heavy grammar.

This PR doesn't touch the no-grammar row, and it still moves about 4% between builds, so differences of that size are noise. Both grammar modes are within that.

Why it doesn't move. I counted memo activity per 256-token JSON run. There were about 22.9k candidate-filter calls and about 10.3k lookups (the memo only stores lists under 80 bytes), but only about 2.7k hits. The cached path is a small part of grammar time, so changing what it stores has no visible cost.

What it does fix. On the base build the memo counts change between identical runs: hits were 2692, 2696 and 2694 on the same prompt, seed and grammar. With this PR they're 2724 every time. On the base, cached rejects still point at code_points buffers that the caller frees after the call that stored them, so later filtering reads freed memory. The output matched in these runs. Once that memory is reused, the filter can reject a valid token, which changes the output. It can also allow a token the grammar forbids, which aborts the server at the GGML_ASSERT in grammar_accept_token.

Could you share the grammar and model where you saw it run slower? I'll run that case.

AI-generated: benchmarked and written by Claude Opus 5.5 on behalf of @Reithan.

@Reithan
Reithan marked this pull request as ready for review October 10, 2026 03:51
@LostRuins LostRuins added the needs review needs review label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs review needs review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants