Memory health permanently WARNs on memory-retrievals.jsonl: CortexHealth checks a stream MemoryRetriever never writes

[dth0m]

Title

Memory health permanently WARNs on memory-retrievals.jsonl: CortexHealth checks a stream MemoryRetriever never writes

Body

What happens

On a fresh 7.40.4 install, bun LIFEOS/TOOLS/MemoryHealthCheck.ts reports overall: warn with:

retrieval-missing: No retrieval evidence within 86400000ms freshness window.

The statusline renders this as 🧠 NEEDS ATTENTION · NO RETRIEVAL EVIDENCE WITHIN 86400000MS FRESHNESS… and it never clears, no matter how much retrieval actually happens. The <lifeos-ground> block is injected every turn, so retrieval is working; it just isn't recorded.

Why

The stream was designed but the writer was never built, and a health check was later added that assumes it exists.

• ObservabilitySystem.md:55 still carries the original note: MemoryRetriever.getRelevantContext() (ISC-107..112; not yet populated as of 2026-05-23; infrastructure ready).
• MemorySystem.md lists the stream as "Per-turn retrievals — written by MemoryRetriever".
• Release 7.40.4 (be9e8ef) added CortexHealth.ts, which reads MEMORY/OBSERVABILITY/memory-retrievals.jsonl and WARNs when the latest row is missing or older than 24h. CortexContract.md:203 codifies it: "Retrieval evidence freshness | 24 hours | WARN when missing/stale".
• git grep memory-retrievals over the whole tree returns only readers: CortexHealth.ts, MemoryStatus.ts, and three docs. MemoryRetriever.ts has never contained the string in any commit.
• No test references the stream, so a check shipping without its producer could not fail anywhere.

Fix

PR: #2094

Adds the writer to getRelevantContext(). Every uncached exit funnels through one finish() closure that appends a row shaped exactly as CortexHealth.validRetrievalRow requires: {ts, query_hash, returned_count, duration_ms} plus optional top_score.

Design choices worth a look:

• Empty-result paths are recorded. An empty retrieval is still evidence the retriever ran this turn, which is what the freshness check is asking about.
• Cache hits are not recorded. They do no ranking work; the stream is described as a "per-turn BM25 audit".
• The query is never persisted. It is the principal's prompt. The row carries a 16-hex sha256 prefix of the cache key, enough to correlate repeats.
• Writes are best-effort and swallowed. The retriever sits on every turn's critical path; a full disk must not turn into a thrown error.

Adds bun MemoryRetriever.ts test, following the self-test convention MemoryReviewer and seven other TOOLS already use. It runs the real hot-path entry point in a child process against a throwaway corpus and validates the rows with the real collectCortexEvidence / assessCortexEvidence. 12 assertions.

Updates the ObservabilitySystem.md:55 note from "not yet populated" to the landed behavior.

Verified

• bun MemoryRetriever.ts test → PASS (12/12)
• MemoryHealthCheck.ts on a live install: warn {critical:0, warn:1} → ok {critical:0, warn:0, ok:26}
• Existing CLI (--help, positional query, corpus-missing error) unchanged

Diff footprint

LIFEOS/DOCUMENTATION/Observability/ObservabilitySystem.md |   2 +-
LIFEOS/TOOLS/MemoryRetriever.ts                            | 166 +++++++++++++--

[michaelaye]

Confirming on a fresh 7.40.4 install, with a patch.

#1848 and #1910 are closed as completed, but the defect is still in the shipped 7.40.4
payload: skills/LifeOS/install/hooks/MemoryTurnStart.hook.ts has no writer either.
#1925 (raw ms in health messages) is the same, closed and still shipping.

memory-retrievals.jsonl has two references and no producer: CortexHealth.ts grades it,
MemoryStatus.ts displays it. Retrieval itself is fine — getRelevantContext runs every
turn from MemoryTurnStart.

Patch goes at the call site, not in MemoryRetriever, whose header declares it read-only:

const startedMs = Date.now();
const ground = getRelevantContext(prompt, { topK: 5, threshold: 0.20 });
logRetrieval(prompt, ground, Date.now() - startedMs);   // one row, fail-open

Two traps:

1. validRetrievalRow uses hasExactKeys — an extra field yields invalid, not ok.
   Omit top_score on zero results rather than sending null.
2. mkdirSync before the append; OBSERVABILITY may not exist on a fresh install.

Separately, retrieval-missing reads "within 86400000ms freshness window" but fires when
the log has zero rows — the lapsed case is retrieval-stale. That wording sends you
hunting a stalled retriever when nothing is stalled.