Skip to content

Save SmartCache / admin save states to a file, and keep them across an admin model reload - #2596

Open
Letrange-g wants to merge 4 commits into
LostRuins:concedo_experimentalfrom
Letrange-g:kv-states-file
Open

Letrange-g wants to merge 4 commits into
LostRuins:concedo_experimentalfrom
Letrange-g:kv-states-file

Conversation

@Letrange-g

@Letrange-g Letrange-g commented Nov 11, 2026 •

Copy link
Copy Markdown

Closes #2520.

Rebased onto concedo_experimental (b0e38e9); no conflicts.

What

  • POST /api/admin/save_state and /api/admin/load_state take an optional "file" name. The slot is also written to (or read from) <admindir>/<name>.kvstate. Basename only, jailed inside --admindir, so a client cannot write elsewhere.
  • POST /api/admin/reload_config takes an optional "keepstates": true for every slot, or a number N for only the N most recently used ones (true/false/numbers are also accepted as text; "false" keeps nothing). Before the reload, those save-state slots are written to <admindir>/kcpp_savestates.bin (on Windows created short-lived, _O_SHORT_LIVED, so the cache manager keeps it in RAM instead of writing it to the SSD while memory allows; it falls back to disk on low-memory machines). After the child process loads the same text model again, they are read back and the file is deleted, so a later request gets a SmartCache match instead of reprocessing the whole prompt. If a different text model is loaded, the hand-off file is deleted unused.
  • New C exports save_states_file(path, slot, keep_recent) / load_states_file(path, slot) (slot = -1 means all slots). The file stores the KV buffer (and draft buffer if any), tokens, logits and media signature per slot.
  • Safety of the file format (header KCPPSS2): it carries the vocab size, the model's parameter count and the context size, and a file of another model or another context size is refused. Every size read from the file is checked against the bytes left before allocating, so a truncated or corrupt file is refused and the server stays up. No empty file is left when a slot holds no state, and a half-written file is removed.
  • API docs (kcpp_docs.embd) for the file field of save_state/load_state and keepstates on reload_config.

Why

Apps that share one GPU between a text model and an image model can unload the text model with reload_config {"filename":"unload_model"} while pictures are made, then bring it back with initial_model. Today the reload ends the child process, so every SmartCache slot is lost and the next turn reprocesses the full context. With keepstates the cost is a file write and read instead.

Tested

Windows, CUDA 13.5, RTX 5090 (32 GB), --smartcache --admin:

  • 31B Q4 text model sharing the card with an image model: a kept slot restored a 9,692-token prompt with 1 token processed (0.16 s instead of 2.84 s of prompt processing). A warm reload of the model took 12-14 s against ~55 s cold.
  • Keeping only the most recent slots matters: with 8 filled slots (~1.1 GB each on the 31B model) every swap wrote ~8 GB; "keepstates": 2 writes ~2 GB.
  • 4B Q8 model: keepstates restore (602-token prompt reprocessed as 1 token after reload), per-slot file save/load, refusal at another context size, truncated file refused with the server alive, "false" writes nothing, hand-off deleted after loading another model.
  • CPU build, stories15M, --smartcache 4 --admin --usemmap: "KV Save States File: 1 states, 3 MB written", reload 3.6 s, "[SmartCache Match of 1.00 restored from slot 3]"; "file":"../x" is reduced to x.kvstate inside the admin dir.

Honest note on the gain: on a 5090 the 2.8 s of prompt processing a kept slot saves is small next to the model reload itself, and in our app it was mostly hidden behind the next generation. The win grows with slower cards, CPU or partial offload, and long chats, where reprocessing the context after every swap takes tens of seconds.

Nothing changes when the new fields are absent.

@Letrange-g Letrange-g changed the title Save state slots to file; keep SmartCache Save SmartCache / admin save states to a file, and keep them across an admin model reloadstates across admin reload … Save SmartCache / admin save states to a file, and keep them across an admin model reload Nov 11, 2026
Letrange-g and others added 4 commits November 14, 2026 11:37
… keepstates parsing, stale hand-off removed

- File header KCPPSS2: n_vocab, model parameter count and context size; a file of another model or context is refused.
- Every size read from a state file is checked against the bytes left before allocating; a truncated file is refused.
- No empty file is left when the requested slot holds no state; a half-written file is removed.
- keepstates accepts true/false/numbers sent as text ("false" no longer enables it).
- The keep-across-reload hand-off file is deleted when a different text model is loaded.
- API docs for the file field of save_state/load_state and keepstates on reload_config.

Tested on Windows/CUDA 13.5 (RTX 5090) with a 4B Q8 model in admin mode with SmartCache: keepstates restore
(602-token prompt reprocessed as 1 token after reload), per-slot file save/load, refusal at another context size,
truncated file refused with the server alive, "false" writes nothing, hand-off deleted after loading another model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Letrange-g
Letrange-g changed the base branch from concedo to concedo_experimental November 14, 2026 17:06
@Letrange-g
Letrange-g marked this pull request as ready for review November 14, 2026 17:09
@LostRuins

Copy link
Copy Markdown
Owner

Hmm I don't really want the save/load state system to interact with the local disk. This is partly because koboldcpp tries as much as possible not to change the local filesystem, and also to prevent remote users to initiating disk writes (kcpp mostly tries to be self contained and not allow persistent changes after loading)

Moreover, save/load states are very ephemeral and also generated/discarded quickly all the time. It doesn't really make sense to explicitly write them to disk, when the same effect is already achievable through swap anyway.

@LostRuins LostRuins added the KIV for now Some issues prevent this from being merged label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

KIV for now Some issues prevent this from being merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request] Save/load KV state slots to a file, so long chats stay warm on low-RAM machines with fast NVMe drives

2 participants