fix(memory): listings that serve content return chain heads by default

[cdeust]

Listings that serve memory content now return supersession chain heads by default, on both backends, so query_methodology no longer lists a retracted memory next to its replacement.

Symptom

On 2026-10-09, after remember(..., supersedes_id=4383843, force=True) answered "action":"superseded" and created 4383844, query_methodology returned both rows in hotMemories. The old row still carried a statement the new one corrects.

Root cause

The shared listing primitives defaulted to the physical table: heads_only: bool = False in pg_store_queries.py (get_memories_for_domain line 24, get_hot_memories line 54), sqlite_store_queries.py (lines 38 and 66), pg_store_entities.py, sqlite_store_entities.py, pg_store_stats.py and sqlite_store_stats.py, and get_memories_for_directory, get_memories_by_tag and get_recent_memories had no heads option at all. Each content-serving caller had to opt in; the earlier fix (docs/program/pr2-read-path-supersession-audit.json) opted in recall_hierarchical, drill_down and a few more. query_methodology.py:135-139 (get_memories_for_domain, get_memories_for_directory, get_hot_memories) did not, nor did narrative, get_project_story, sync_instructions, checkpoint, assess_coverage, detect_gaps. curate_wiki and curate_distill opted in on their first branch and then fell back to get_recent_memories, which read the whole table.

Fix

• The safe read is the default: heads_only defaults to True on get_memories_for_domain, get_memories_for_directory (parameter added), get_hot_memories, get_memories_mentioning_entity, get_recently_accessed_memories, and, after the sibling review, get_memories_by_tag, get_memories_for_entity and get_all_memories_for_validation (parameter added), on SQLite and PostgreSQL. get_recent_memories reads current_memories.
• wiki_extract selects candidates from current_memories, so claims are not mined from a retracted memory.
• Callers that need the physical chain say so with heads_only=False and are pinned by a test: validate_memory (all three selection paths), the pruning and plasticity consolidation passes, the staleness sweep (memory_staleness_pass) and its dry-run wrapper in scripts/memory_staleness_revalidate.py, and the four idempotency and duplicate-prevention lookups (memify_derive derived-rel, curate_distill distill-of, ingest_findings_writers find_existing_memory, ingest_document_writers already_ingested: a corrected fact need not carry the marker or dedup tag, so reading heads would derive, distill or write the same thing a second time; all four were physical reads before this PR).
• The pin is stricter again in round 4: it walks every non-test Python file of the repository (scripts/ and benchmarks/ included, the launcher's gitignored deps/ tree excluded), the listing set is derived from the store signatures, any listing call in a maintenance module (handlers/consolidation/, validate_memory.py, scripts/) that does not state heads_only fails, the heads_only defaults are compared between the SQLite and PostgreSQL stores from their declared signatures (no server needed), and every non-test class that defines a listing must accept the keyword. Its limit is in the test docstring: it reads source text, so it cannot see a run-time value.

Review round 4 (verdict Cortex-698d: B1, B2)

• B1: _DryRunStore.get_all_memories_for_validation in scripts/memory_staleness_revalidate.py lacked heads_only, so the script's default dry-run mode raised TypeError. It now accepts and forwards the keyword. Test: tests_py/scripts/test_memory_staleness_revalidate_script.py drives revalidate_staleness through the real script wrapper over a real SQLite store (2 tests).
• B2: find_existing_memory and already_ingested inherited the heads default and lost the retracted carrier, so after a finding was superseded, writing it again inserted the retracted text as a fresh head. Both pass heads_only=False (ADR-1100 cited at the sites), both are in the pin's _MAINTENANCE_SITES, one test per site: tests_py/handlers/test_ingest_dedup_after_supersession.py (real SQLite store, supersede the carrier, assert no second write).
• Sibling walks over ALL write-path existence checks of the whole repository (two AST walks, scripts and benchmarks included), dispositions in the table below and in /Users/cdeust/.claude/fleet/work/Cortex/698-r4-dispositions.md (walk outputs 698-r4-existence-walk.txt and 698-r4-checkwrite-walk.txt in the same directory). Doubles outside mcp_server/ that define a listing were checked by a class-level walk: one miss (the script wrapper), the four ingest fakes updated with B2.
• N1: the PostgreSQL default mutant is now killed without a server by the static defaults comparison (mutant: PostgreSQL get_recently_accessed_memories default flipped to False fails test_the_heads_only_defaults_are_the_same_on_both_backends).
• N2: docs/program/pr2-read-path-supersession-audit.json no longer says search_vectors or the episodic/semantic readers default to True (search_vectors keeps False on both backends, the readers have no keyword and read current_memories), and the amendments note says "ci-dessus".
• N3: ADR-1100 point 4 no longer lists tag lookups as physical exceptions; point 3 names the two ingest lookups.
• N4: the memify and curate_distill opt-outs stay protected by the pin (source text) plus their existing fakes; I did not add a behavioural test for them in this round.
• N5: the two SQL-substring session_start tests were replaced by tests_py/hooks/test_session_start_chain_heads.py, which runs both hook functions against a real PostgreSQL store and judges their results (the heads reach the curator, a retracted graph path is not served). Both hook functions are PostgreSQL-only (_connect_pg, a PG connection argument), so on a SQLite leg the substring tests pinned text that never executed; the new tests skip without PostgreSQL like the other hook SQL tests. tests_py/hooks/test_session_start.py is back to its content on main.
• N6: the pin scans the whole repository (scripts/ included).
• N7: dispositions file corrected (get_recent_memories moved from "already heads" to "fixed": both stats files changed from memories to current_memories).
• N8: see the PostgreSQL note below, labelled per head.
• N9: get_all_memories_for_validation is in the limit test (mutant: calling it with heads_only=False fails sqlite and pg legs).
• N10: unchanged, the statistics definition is left to the owner (stated above).

New mutants, each applied and run (raw output 698-r4-pin-mutants.txt and 698-r4-session-start-mutant.txt in /Users/cdeust/.claude/fleet/work/Cortex/): ingest opt-out removed fails the maintenance-sites pin; wrapper call dropping the keyword fails the silent-call pin; PostgreSQL default flipped fails the static defaults test; session_start SQL back to memories fails both behaviour tests.

Failing before, passing after (raw output in the same directory): B1 2 failed (698-r4-B1-before.txt, TypeError: _DryRunStore.get_all_memories_for_validation() got an unexpected keyword argument 'heads_only') then 2 passed (698-r4-B1-after.txt); B2 2 failed (698-r4-B2-before.txt, assert None == 1) then 2 passed (698-r4-B2-after.txt).

Round 4 dispositions (write-path existence, duplicate, marker and idempotency checks, whole repository)

Site	Disposition
ingest_findings_writers find_existing_memory	fixed: heads_only=False, pinned, tested
ingest_document_writers already_ingested	fixed: heads_only=False, pinned, tested
ingest_docs_content_writers find_existing_doc_memory	heads on purpose: filters superseded rows itself, a corrected document should make a new head
memify_derive _existing_derived_markers, curate_distill _existing_distill_markers	physical, already pinned
codebase_analyze_helpers load_existing_hashes	physical by contract (hash idempotency record), unchanged by the flip; see uncertain
wiki_migrate _existing_memory_ids, write_gate _collect_existing_embeddings, pg supersede_to_existing	by id, physical, unchanged
pg_store_memory_dedup list_exact_duplicate_groups	already reads current_memories
entity get-or-create helpers, checkpoint, triggers, cascade, ingest_codebase / ingest_docs_content passes	not memory-head existence checks (entity table, counters, physical stage cursor)
scripts/memory_staleness_revalidate.py wrapper	fixed (B1)
scripts/provenance_sweep.py, benchmarks/	no call of a listing primitive

Uncertain: load_existing_hashes builds {path: (id, hash)} from every physical row and the last row wins, so a stale hash on a superseded codebase memo could in principle win. It is behaviour on main, not changed by the default flip, and I did not reproduce it.

Sibling search (review round 2)

Commands: grep -rnE "(FROM|JOIN)\s+(memories|current_memories)\b" mcp_server | grep -v pg_schema for raw SQL, plus an AST walk over mcp_server/**/*.py that lists, per function, which of the two tables it reads (117 functions) and every call of a listing primitive outside infrastructure/ with its heads_only keyword (97 calls). Per-site dispositions: /Users/cdeust/.claude/fleet/work/Cortex/698-sibling-dispositions.md.

Site	Disposition
session_start _count_pending_curations (count shown to the user)	heads: now joins current_memories like the anchor and hot-memory queries
session_start _lookup_cached_graph_path (serves the cached graph path)	heads: FROM current_memories
write_post_store.py:127 synaptic tagging (get_hot_memories)	content-serving, heads, stated explicitly: a boost matters only for a memory a recall can serve
write_post_store.py:214 shared-entity fallback (get_memories_mentioning_entity)	heads, stated explicitly: the new memory is a head, retracted rows must not use the 50-row window
memify_derive.py:161 derived-rel marker scan (get_memories_by_tag)	maintenance: heads_only=False, pinned
memify_derive.py:195 provenance ids (get_memories_for_entity)	content-serving: provenance must cite live memories, heads_only=True stated, the primitive gained the parameter
curate_distill.py:157 distill-of marker scan	maintenance: heads_only=False, pinned
assess_coverage:255, change_impact:208 (get_all_memories_for_validation, served to the user)	heads; the primitive gained heads_only (default True); validate_memory and the staleness sweep pass False
consolidation cursors (decay, stage, neighbours, interference), by-id lookups, slot helpers, search_vectors, co-access and embedding signal readers, direct SQL by id or tag	physical by contract, unchanged; reasons in the disposition file
session_start _fetch_grooming_staleness	physical: MAX(created_at) of lessons; a head is never older than its predecessors

Owner decision left open: session_start _count_memories prints "(total: N)" from COUNT(*) over memories, so it includes retracted versions, and memory_stats.total has the same definition. Kept as a storage total; changing both is a statistics-semantics change outside this PR.

Failing before, passing after

Before (main with the new tests; raw output in /Users/cdeust/.claude/fleet/work/Cortex/B-before.txt):

E   assert 1 not in [2, 1]
FFFF   (query_methodology: domain, directory, global, handler)
24 failed, 10 passed in 61.07s   (listing primitives, sqlite and pg legs)
2 failed in 8.75s                (wiki_extract)

After, on the FINAL head d967c82 (PostgreSQL legs RAN, see the note below), pytest -q -p no:cacheprovider (full suite, raw output 698-r4-full-pytest.txt):

9685 passed, 19 skipped, 2 warnings, 373 subtests passed in 326.75s (0:05:26)
exit=0

On the final head: ruff check and ruff format --check clean, check_craftsmanship.py --base origin/main OK, check_project_wiki.py OK, check_no_deps_invariant.py clean, pyright mcp_server/ 0 errors in the locked environment (698-r4-gates.txt).

PostgreSQL, per head (this settles the two earlier contradictory notes): on 66aa07d the suite ran the PostgreSQL legs (9633 passed); on d3f04bb no PostgreSQL was reachable for the author's run, so 112 tests skipped (9584 passed) and the reviewer ran SQLite-only; on the final head d967c82 the suite ran against the repo's own throwaway database (conftest create_isolated_test_database, dropped at session end, no live database touched): 9685 passed, 19 skipped. The first full run on dfd18dc (698-r4-full-pytest-first-dfd18dc0.txt) was 9682 passed, 3 failed, 19 skipped: the new whole-repo pin tried to read a non-UTF-8 vendored file under the gitignored deps/ tree that the launcher tests create in the checkout; d967c82 excludes that tree and the three tests pass.

New in round 2, each shown to fail on a mutant:

• limit applied after the head filter: a listing of N returns N heads when the retracted rows outrank them (heat, recency and access), on both backends; mutant (SQLite get_memories_for_domain filters after the LIMIT) gives 0 = len([]).
• validate_memory keeps the superseded row on its three selection paths; mutant (heads_only=False removed) fails all three.
• a maintenance-module listing call without heads_only fails the pin; mutant (memify_derive provenance call drops the keyword) fails it.

PostgreSQL (head 66aa07d and d967c82): the pg legs of test_superseded_listing_defaults.py ran against the suite's throwaway database; they failed before and pass after (on d3f04bb they skipped). Two existing tests (test_supersession_read_path.py and its SQLite twin) asserted the old default and now pass heads_only=False explicitly.

Completion Ledger

Branch	Test
query_methodology domain, directory and global branches exclude the old row, return the head	tests_py/handlers/test_query_methodology_superseded.py (3 cases), test_handler_hot_memories_exclude_the_superseded_row
each of 7 listings excludes the old row by default, per backend	tests_py/infrastructure/test_superseded_listing_defaults.py::test_default_listing_excludes_the_superseded_row
heads_only=False still returns the physical chain	::test_maintenance_callers_can_still_see_the_physical_chain
explicit heads_only=True equals the default	::test_explicit_heads_only_matches_the_default
superseded row still readable by id	::test_the_superseded_row_stays_readable_by_id
only reviewed maintenance sites ask for the physical chain; maintenance modules state heads_only	tests_py/infrastructure/test_listing_callers_supersession_audit.py
limit of N returns N heads (6 listings, both backends)	tests_py/infrastructure/test_superseded_listing_defaults.py::test_a_listing_of_n_returns_n_heads_when_more_than_n_rows_exist
validate_memory keeps the physical chain	tests_py/handlers/test_validate_memory_physical_chain.py
session_start count and graph lookup read heads	tests_py/hooks/test_session_start_chain_heads.py (2 cases, PostgreSQL)
staleness script dry-run forwards heads_only	tests_py/scripts/test_memory_staleness_revalidate_script.py (2 cases)
ingest duplicate checks see a superseded carrier	tests_py/handlers/test_ingest_dedup_after_supersession.py (2 cases)
heads_only defaults equal across stores; whole-repo pin; doubles accept the keyword	tests_py/infrastructure/test_listing_callers_supersession_audit.py (5 cases)
wiki_extract backlog and explicit memory_id	tests_py/handlers/test_wiki_extract_superseded.py

Decision points for an ADR

Round 4: ADR-1100 point 3 now names find_existing_memory and ingest_document_writers.already_ingested among the physical-chain readers; point 4 no longer lists tag lookups as exceptions.

ADR-1100 is registered ("Listings that serve content return supersession chain heads by default"; amends ADR-0258, ADR-0537, ADR-0602, ADR-1014). It is applied on this branch (rebased on main after #697 merged). docs/program/pr2-read-path-supersession-audit.json, which recorded the opposite default, is updated here (the five entries that stated or implied it, plus an amendments note); no test reads that file.

Not verified

Windows. query_methodology still wraps its store read in a catch-and-return-empty block (pre-existing, outside this change).

Fixes #687

🤖 Generated with Claude Code

https://claude.ai/code/session_01DXMXvHLm94B6pA9FkWWmzu

[cdeust]

ZETETIC-REVIEW: REQUEST_CHANGES
d3f04bb

Scope: code, tests and text of PR #698 at the head above, base origin/main 02c3cf0. Read-only review in a registered throwaway worktree (review-698d). Load average 3.4 to 4.6, targeted tests only.

How tests were run: tests_py/conftest.py connects to PostgreSQL at import time (create_isolated_test_database at conftest.py:139, _pg_available at :181) and a server was listening on 5432, so the normal pytest entry would have touched the local server. I therefore ran pytest with --noconftest, a stub module standing in for tests_py.conftest (_USE_PG=False), HOME and every Cortex data path pointed at a throwaway dir, CORTEX_MEMORY_STORE_BACKEND=sqlite, DATABASE_URL unset, no database URL set. Every PostgreSQL leg was skipped; PostgreSQL behaviour below is by reading the SQL, not by execution. The PostgreSQL store classes were imported for signature inspection (and by the pin test); importing them opens no connection.

Blocking

B1. scripts/memory_staleness_revalidate.py:46 (_DryRunStore.get_all_memories_for_validation(self, limit, *, after_id, include_stale)) was not updated. The PR makes revalidate_staleness pass heads_only=False (memory_staleness_pass.py:85) and adds heads_only to the _StaleStore protocol. The script's default mode (no --apply) wraps the store in _DryRunStore, so it now raises. Reproduced on in-memory SQLite by calling revalidate_staleness(_DryRunStore(SqliteMemoryStore()), ...) with the module loaded from the script file: TypeError: _DryRunStore.get_all_memories_for_validation() got an unexpected keyword argument 'heads_only'. Contract drift on a changed signature (Move 3). Required: accept and forward heads_only, plus a test that drives revalidate_staleness through _DryRunStore.

B2. Two idempotency lookups silently moved from the physical chain to heads, the same class the PR itself opts out for memify derived-rel and curate_distill distill-of (ADR-1100 point 3: "must still see a superseded carrier so a corrected fact is not derived twice"):

• mcp_server/handlers/ingest_findings_writers.py:50 find_existing_memory: get_memories_by_tag(tag, limit=5), no heads_only. write_finding_memory's docstring promises "no duplicate row is inserted (idempotence)".
• mcp_server/handlers/ingest_document_writers.py:51 already_ingested: same call shape, dedup tag doc-ingest::@.
  Reproduced on in-memory SQLite at this head: write_finding_memory -> (1, True); supersede_atomic replaces row 1 with a corrected row that does not carry the finding tag; write_finding_memory again -> (3, True), i.e. the retracted finding text is inserted again as a fresh chain head, which is the query_methodology returns superseded memories in hotMemories #687 harm in reverse. already_ingested returned None after the summary was superseded, while get_memories_by_tag(tag, heads_only=False) still returns the carrier. Same probe with the three base SQLite store files (sqlite_store_queries.py, sqlite_store_entities.py, sqlite_store_stats.py from origin/main) checked out over the head, then restored: second write -> (1, False) and already_ingested -> 3 (the superseded carrier), so this is a regression introduced by the default flip. The author's table files these under "ingest_* cached lookups", which they are not; the ADR's maintenance list and the pin list in tests_py/infrastructure/test_listing_callers_supersession_audit.py omit them. Required: heads_only=False at both sites, add them to _MAINTENANCE_SITES and ADR-1100 point 3, and a test per site that supersedes the carrier and asserts no second insert. (ingest_docs_content_writers.find_existing_doc_memory already filters superseded rows in Python on purpose, so heads is right there; ingest_findings_resolve._from_graph_key and ingest_helpers cached-graph lookup are content/path lookups, heads acceptable.)

Verified (with command or source)

• Diff read in full: git diff origin/main...d3f04bb --stat (31 files, 806+/80-), every source hunk under mcp_server/ read.
• Store parity: signature defaults read via inspect.signature on both SqliteMemoryStore and PgMemoryStore: identical, nine methods carry heads_only, eight default True (get_all_memories_for_validation, get_hot_memories, get_memories_by_tag, get_memories_for_directory, get_memories_for_domain, get_memories_for_entity, get_memories_mentioning_entity, get_recently_accessed_memories), search_vectors False on both. get_recent_memories reads current_memories on both (pg_store_stats.py:100, sqlite_store_stats.py:113). In every one the view is the FROM source, so the head filter precedes ORDER BY and LIMIT on both backends, including both branches of get_memories_mentioning_entity and both branches of get_hot_memories. View defined on both (sqlite_schema.py:78, pg_schema.py:97).
• Targeted tests on head (SQLite legs): 77 passed, 29 skipped (PG legs) across test_superseded_listing_defaults, the pin test, test_memify_derive, test_curate_distill, test_query_methodology_superseded, test_validate_memory_physical_chain, test_memory_staleness_pass, test_wiki_extract_superseded, test_sqlite_supersession_read_path. test_session_start.py: the two new tests pass; 6 pre-existing main() tests fail only because skipping conftest leaves the composition root unwired (RuntimeError from core/environment.py:61), not related to the diff.
• Mutants (each applied, run, restored with git checkout; tree clean afterwards):
  • SQLite get_hot_memories default back to False: 4 failed (listing defaults x3, query_methodology global).
  • memify derived-rel heads_only=False dropped: 2 failed (pin test only).
  • memify derived-rel flipped to heads_only=True: 1 failed (pin test only).
  • SQLite get_memories_by_tag reads memories and filters heads in Python after the LIMIT: 1 failed (limit test, get_memories_by_tag leg).
  • validate_memory domain path heads_only=True: 2 failed (pin test, validate_memory by-domain).
  • SQLite get_recent_memories back to memories: 2 failed.
  • PostgreSQL get_hot_memories default back to False: SURVIVED locally (PG legs skipped). Caught only where PostgreSQL runs.
• python scripts/check_project_wiki.py: "Project wiki mirrors match their canonical sources." python scripts/check_craftsmanship.py --base origin/main: "Craftsmanship gate: OK". uvx ruff@0.16.6 check . and format --check . (scratch dir excluded): clean.
• docs/program/pr2-read-path-supersession-audit.json: valid JSON (json.load).
• Pre-existing touched test files, base -> head: test_memory_staleness_pass 3->3 tests, 6->6 asserts; test_curate_distill 11->11, 28->28; test_memify_derive 10->10, 35->35; test_session_start 82->84, 159->162; test_sqlite_supersession_read_path 12->12, 22->22; test_supersession_read_path 13->13, 25->25. The two read-path tests now pass heads_only=False explicitly, nothing removed.
• Five fakes: only signatures changed (heads_only added with default True, or required keyword in the staleness protocol fake); no assertion or behaviour removed. They ignore the keyword, as before.
• PR body: Fixes #687 present (line 89); PG-skip note present (line 57). The claim that only comments, docstrings and wiki changed since 66aa07d holds for the PR-owned files (git diff 66aa07d d3f04bb on handlers/consolidation, validate_memory, infrastructure, pin test: comment and docstring lines only). Issues Deleting the head of a supersession chain differs between SQLite and PostgreSQL #699 and Named degrade paths in remember, sqlite search and rebuild_profiles report only to an unread counter #685 cited by ADR-1100 point 6 exist and are open.
• ADR-1100 points 1, 2, 5 match the code; point 5 is pinned by the limit test (mutant above). Point 3 is incomplete (B2).

Non-blocking

N1. The PostgreSQL defaults are pinned only by the PG legs. A static check comparing inspect.signature defaults of every heads_only parameter across both stores would catch the surviving mutant without a server; the pin test today compares names only (test_the_listing_set_is_the_same_on_both_backends).
N2. Audit JSON, layer_mechanism: lists get_episodic/semantic_memories and search_vectors among primitives whose "kwarg heads_only (défaut True depuis ADR-1100)". search_vectors defaults to False on both backends and the episodic/semantic readers have no such kwarg. The amendments entry says "Les entrées ci-dessous" but sits at the end of the object.
N3. ADR-1100 point 4 names "id and tag lookups" among the deliberate physical-row exceptions, while point 2 makes get_memories_by_tag head-by-default. Reword point 4 (tag lookups are heads unless they opt out).
N4. The memify and curate_distill opt-outs are protected only by the source-text pin; the fakes ignore heads_only, so no behavioural test shows a superseded marker carrier still blocks a second derivation/distillation.
N5. The two new session_start tests assert on SQL substrings only.
N6. The pin test reads mcp_server/ only and lists modules by hand; scripts/ and ingest_* sat outside it, which is how B1 and B2 escaped. Consider widening _calls() to scripts/.
N7. The author's disposition table (fleet work file, not in the repo) lists get_recent_memories as "already filtered before this change"; the diff changes both stats files from memories to current_memories. The repo text (ADR, docstrings) is correct.
N8. PR body line 57 ("no PostgreSQL reachable") and line 64 ("a local server was reachable") describe different runs; label which head each refers to.
N9. get_all_memories_for_validation is absent from the limit test (id order makes the result moot today, but it is unpinned).
N10. memory_stats.total and the SessionStart banner total still count retracted rows; the PR states this and leaves it to the owner, no issue number. Not a defect dismissal in my reading (a stated statistic definition), but an issue would make it trackable.

Not verified

• Any PostgreSQL execution (all PG legs skipped by construction, see above); PG equivalence is by reading SQL and signatures.
• Full pytest suite, pyright, scripts/check_no_deps_invariant.py, CI, Windows, query plans on a large table.
• Whether any other script under scripts/ wraps a listing primitive (I grepped scripts/ for the nine primitives: only memory_staleness_revalidate.py:48 calls one).