Skip to content

fix(memory): Implement engine & model cache lifetime management with configurable idle TTL and auto-eviction #97

Closed

Description

@spelech

Description & Evidence

During live system operation, terminating LocalLLMServerManager (PID 37952) resulted in an immediate reclaim of 12.29 GB of host system RAM and ~870 MB of Windows Non-Paged Kernel Pool memory, even though the application's reported private working set was only ~1.85 GB.

Collected System Metrics

Metric Before Stopping LocalLLMServerManager After Process Termination Delta
In-Use Physical RAM 25.00 GB 12.71 GB -12.29 GB reclaimed
Free Physical RAM 6.73 GB 19.02 GB +12.29 GB freed
Non-Paged Kernel Pool 1.92 GB 1.05 GB -870 MB freed
Commit Charge 33.51 GB 21.10 GB -12.41 GB

Root Cause

  1. Unbounded Model Residency in Host & Driver Memory: When backend engines (Ollama, ComfyUI, Forge, Kokoro) load models or memory-map large GGUF/safetensors files, host memory pages and CUDA/driver host-pinned buffers remain allocated indefinitely.
  2. Missing Cache Lifetime & Idle TTL Policies: VramOrchestrator currently only attempts to unload models reactively when another workload explicitly demands VRAM (EnsureVramForImageGenerationAsync(), EnsureVramForComfyUiAsync()). It does not enforce any idle expiration policy, TTL countdown, or keep-alive window.
  3. Ollama Keep-Alive Unmanaged: Requests sent to Ollama default to retaining the model in memory indefinitely (keep_alive = -1 or engine default) unless explicitly told otherwise.

Proposed Changes

  1. Configurable Model Cache Lifetimes: Add a user setting for model idle lifetime (ModelCacheIdleTimeoutMinutes, e.g., 5m, 15m, 30m, 1h, Never).
  2. Idle Eviction Watchdog: Implement a background service in VramOrchestrator that monitors the last-used timestamp of loaded models and triggers an unload (/api/generate with keep_alive: 0 for Ollama, /free for ComfyUI) once the idle TTL expires.
  3. Graceful Engine Hibernation: Provide an auto-stop or hibernation trigger for dormant background child engines after extended inactivity.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions