Repository navigation
Conversation
|
you should PR to the experimental branch, and i think this PR conflicts with your previous one. Also I don't actually think there is any performance improvement in fact I think that this actually performs worse. |
|
Let me re-target to experiment and I'll mark this as draft and run a few more tests. 👍 |
c1211a7 to
8788edb
Compare
…ters Cached candidates held code_points pointers into decode buffers freed after each sampling step, so content-keyed cache hits (since LostRuins#2431) returned dangling pointers. Store reject indices instead and rebuild the rejects from the current candidates on a hit, in both the token-rule early return and the char-rule path.
8788edb to
e069a43
Compare
|
I benchmarked this to check the "performs worse" concern and couldn't find a regression. Setup. I used CPU-only
All builds produced byte-identical output in every mode. The heavy grammar is a hand-written JSON-like grammar with nested bounded repetitions and character classes, and no token rules: heavy grammarMedian microseconds per generated token:
The original PR against its old base was just as flat on grammar time: 1074 against 1110 on the JSON schema and 914 against 892 on the heavy grammar. This PR doesn't touch the no-grammar row, and it still moves about 4% between builds, so differences of that size are noise. Both grammar modes are within that. Why it doesn't move. I counted memo activity per 256-token JSON run. There were about 22.9k candidate-filter calls and about 10.3k lookups (the memo only stores lists under 80 bytes), but only about 2.7k hits. The cached path is a small part of grammar time, so changing what it stores has no visible cost. What it does fix. On the base build the memo counts change between identical runs: hits were 2692, 2696 and 2694 on the same prompt, seed and grammar. With this PR they're 2724 every time. On the base, cached rejects still point at Could you share the grammar and model where you saw it run slower? I'll run that case. AI-generated: benchmarked and written by Claude Opus 5.5 on behalf of @Reithan. |
Overview
Fix a use-after-free in the grammar rejection cache (
candidates_memos) inllama_grammar_reject_candidates_for_stack.The cache stored full
llama_grammar_candidatestructs. Each one holds a rawcode_pointspointer into a decode buffer owned by the caller, which is freed when the sampling step ends. Since #2431 switched the cache key to a content hash, identical candidate lists now hit across steps, so a hit returns candidates whosecode_pointspoint into a previous step's freed buffer.This PR is rebased onto current
concedo_experimentaland builds on #2431. #2431 already provides the content-based key and the cache write in the TOKEN / TOKEN_NOT branch, so this PR now only changes what is stored.Changes
std::vector<size_t>), not full candidate structs.code_pointsalways points into live data.Testing
koboldcpp has no grammar unit tests, so I checked this with a scratch driver linked against
koboldcpp_default.so. It callsllama_grammar_reject_candidates_for_stackthree times with identical content but freshly allocated code-point buffers, so the third call is a cache hit. It covers TOKEN, TOKEN_NOT and char rules.code_pointspoints into the current call's buffers.concedo_experimental: the rejects match, but on a cache hitcode_pointspoints into the previous call's freed buffers (fails for all three rule types).End-to-end smoke test with Qwen3.5-0.8B on CPU: JSON-schema, word-alternation and character-class grammars, 3 seeded runs each, all grammar-valid with no asserts. The JSON outputs are identical to
concedo_experimentalfor the same seeds.Requirements