You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(memory): Implement engine & model cache lifetime management with configurable idle TTL and auto-eviction #97
During live system operation, terminating LocalLLMServerManager (PID 37952) resulted in an immediate reclaim of 12.29 GB of host system RAM and ~870 MB of Windows Non-Paged Kernel Pool memory, even though the application's reported private working set was only ~1.85 GB.
Collected System Metrics
Metric
Before Stopping LocalLLMServerManager
After Process Termination
Delta
In-Use Physical RAM
25.00 GB
12.71 GB
-12.29 GB reclaimed
Free Physical RAM
6.73 GB
19.02 GB
+12.29 GB freed
Non-Paged Kernel Pool
1.92 GB
1.05 GB
-870 MB freed
Commit Charge
33.51 GB
21.10 GB
-12.41 GB
Root Cause
Unbounded Model Residency in Host & Driver Memory: When backend engines (Ollama, ComfyUI, Forge, Kokoro) load models or memory-map large GGUF/safetensors files, host memory pages and CUDA/driver host-pinned buffers remain allocated indefinitely.
Missing Cache Lifetime & Idle TTL Policies:VramOrchestrator currently only attempts to unload models reactively when another workload explicitly demands VRAM (EnsureVramForImageGenerationAsync(), EnsureVramForComfyUiAsync()). It does not enforce any idle expiration policy, TTL countdown, or keep-alive window.
Ollama Keep-Alive Unmanaged: Requests sent to Ollama default to retaining the model in memory indefinitely (keep_alive = -1 or engine default) unless explicitly told otherwise.
Proposed Changes
Configurable Model Cache Lifetimes: Add a user setting for model idle lifetime (ModelCacheIdleTimeoutMinutes, e.g., 5m, 15m, 30m, 1h, Never).
Idle Eviction Watchdog: Implement a background service in VramOrchestrator that monitors the last-used timestamp of loaded models and triggers an unload (/api/generate with keep_alive: 0 for Ollama, /free for ComfyUI) once the idle TTL expires.
Graceful Engine Hibernation: Provide an auto-stop or hibernation trigger for dormant background child engines after extended inactivity.
Activity
chore(release): bump version to v4.1.0 and refresh WASM bundle (#97, #98
feat(memory): add two-tier engine cache lifetime management and host …