Repository navigation
Save SmartCache / admin save states to a file, and keep them across an admin model reload - #2596
Open
Letrange-g wants to merge 4 commits into
Open
Letrange-g wants to merge 4 commits into
Letrange-g wants to merge 4 commits into
Conversation
… keepstates parsing, stale hand-off removed
- File header KCPPSS2: n_vocab, model parameter count and context size; a file of another model or context is refused.
- Every size read from a state file is checked against the bytes left before allocating; a truncated file is refused.
- No empty file is left when the requested slot holds no state; a half-written file is removed.
- keepstates accepts true/false/numbers sent as text ("false" no longer enables it).
- The keep-across-reload hand-off file is deleted when a different text model is loaded.
- API docs for the file field of save_state/load_state and keepstates on reload_config.
Tested on Windows/CUDA 13.5 (RTX 5090) with a 4B Q8 model in admin mode with SmartCache: keepstates restore
(602-token prompt reprocessed as 1 token after reload), per-slot file save/load, refusal at another context size,
truncated file refused with the server alive, "false" writes nothing, hand-off deleted after loading another model.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Letrange-g
force-pushed
the
kv-states-file
branch
from
November 14, 2026 17:06
82c934e to
f022866
Compare
Letrange-g
marked this pull request as ready for review
November 14, 2026 17:09
Owner
|
Hmm I don't really want the save/load state system to interact with the local disk. This is partly because koboldcpp tries as much as possible not to change the local filesystem, and also to prevent remote users to initiating disk writes (kcpp mostly tries to be self contained and not allow persistent changes after loading) Moreover, save/load states are very ephemeral and also generated/discarded quickly all the time. It doesn't really make sense to explicitly write them to disk, when the same effect is already achievable through swap anyway. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2520.
Rebased onto
concedo_experimental(b0e38e9); no conflicts.What
POST /api/admin/save_stateand/api/admin/load_statetake an optional"file"name. The slot is also written to (or read from)<admindir>/<name>.kvstate. Basename only, jailed inside--admindir, so a client cannot write elsewhere.POST /api/admin/reload_configtakes an optional"keepstates":truefor every slot, or a number N for only the N most recently used ones (true/false/numbers are also accepted as text;"false"keeps nothing). Before the reload, those save-state slots are written to<admindir>/kcpp_savestates.bin(on Windows created short-lived,_O_SHORT_LIVED, so the cache manager keeps it in RAM instead of writing it to the SSD while memory allows; it falls back to disk on low-memory machines). After the child process loads the same text model again, they are read back and the file is deleted, so a later request gets a SmartCache match instead of reprocessing the whole prompt. If a different text model is loaded, the hand-off file is deleted unused.save_states_file(path, slot, keep_recent)/load_states_file(path, slot)(slot = -1means all slots). The file stores the KV buffer (and draft buffer if any), tokens, logits and media signature per slot.KCPPSS2): it carries the vocab size, the model's parameter count and the context size, and a file of another model or another context size is refused. Every size read from the file is checked against the bytes left before allocating, so a truncated or corrupt file is refused and the server stays up. No empty file is left when a slot holds no state, and a half-written file is removed.kcpp_docs.embd) for thefilefield of save_state/load_state andkeepstateson reload_config.Why
Apps that share one GPU between a text model and an image model can unload the text model with
reload_config {"filename":"unload_model"}while pictures are made, then bring it back withinitial_model. Today the reload ends the child process, so every SmartCache slot is lost and the next turn reprocesses the full context. Withkeepstatesthe cost is a file write and read instead.Tested
Windows, CUDA 13.5, RTX 5090 (32 GB),
--smartcache --admin:"keepstates": 2writes ~2 GB."false"writes nothing, hand-off deleted after loading another model.--smartcache 4 --admin --usemmap: "KV Save States File: 1 states, 3 MB written", reload 3.6 s, "[SmartCache Match of 1.00 restored from slot 3]";"file":"../x"is reduced tox.kvstateinside the admin dir.Honest note on the gain: on a 5090 the 2.8 s of prompt processing a kept slot saves is small next to the model reload itself, and in our app it was mostly hidden behind the next generation. The win grows with slower cards, CPU or partial offload, and long chats, where reprocessing the context after every swap takes tens of seconds.
Nothing changes when the new fields are absent.