Skip to content

The remote meta cache never expires, so local parity runs can silently pass or fail #138

Closed

Description

@LukasGold

The remote meta source is served from ~/.cache/oold/meta/remote-main, which is written once by oold meta fetch and then never revalidated. There is no TTL, no staleness warning, and no automatic refresh. meta_store.py records the fetch time in fetched.json and surfaces it (meta_store.py:534, meta_store.py:622), but nothing ever compares it against anything.

Since #133 the parity check gates the build, and CLAUDE.md tells contributors to run it locally before touching src/oold/validation/:

OOLD_SCHEMA_DIR=../oold-schema uv run pytest -m parity

That command compares a freshly cloned upstream compliance suite against a cache that may be arbitrarily old. The two halves drift independently.

Observed

A cache 15 days old produced three failures on a main that is in fact green:

FAILED test_parity_live.py::test_upstream_compliance_suite_passes
FAILED test_parity_live.py::test_vocabulary_coverage_matches_upstream
FAILED test_parity_live.py::test_verdict_agrees_with_the_reference_harness

1 keyword(s) defined in the meta-schemas but not tested: x-oold-reverse-default-properties

x-oold-reverse-default-properties was removed from the vocabulary in OO-LD/oold-schema#152. Upstream's compliance suite stopped covering it; the cached meta-schema still declared it. oold meta fetch --force alone turned all three green, with no change to this repository.

Why it is worth fixing rather than documenting

The false failure costs an investigation, which is annoying but self-correcting: the tests are red, someone looks.

The false pass is the real problem and it is silent. A stale cache can equally hide a genuine disagreement: the reference moves, the cached meta-schema does not, the local run stays green, and the contributor pushes believing parity holds. That inverts what #133 made the check for. Nothing in the output distinguishes the two situations, because the fetch date is printed but never judged.

CI is unaffected: a fresh runner has no cache and fetches every time. This is a local-verification trap only, which is precisely where the instruction to run parity by hand points people.

Options

  • warn when the cache is older than some threshold, naming the age and oold meta fetch --force
  • revalidate on use, cheaply, with a conditional request against the recorded base_url
  • have the parity tests themselves refuse a cache older than the upstream checkout they are comparing against

The third is the narrowest and matches the failure: parity is the only consumer that pairs the cache with an independently moving reference.

Found while verifying #137.

Activity

  1. simontaurus commented on Aug 29, 2026

    @simontaurus
    Contributor

    Hit this for real. The parity suite failed on main for a reason that had nothing to do with the code under test:

    FAIL coverage.vocab   compliance: 1 keyword(s) defined in the meta-schemas but not tested: x-oold-reverse-default-properties
    reference exit=0, this port passed=False
    

    The cached remote-main/oold-meta-schema-base.json still defined the keyword, while the compliance fixtures came from a current OOLD_SCHEMA_DIR checkout where OO-LD/oold-schema#152 had removed both. So the check compared a meta-schema from one commit against fixtures from another and reported a genuine-looking mismatch in the fixtures.

    oold meta fetch --force fixes it: 6 passed.

    This is the "silently fail" half of the title. The "silently pass" half is worse and would look like a keyword being covered because the cache predates its removal. Worth noting that the local reproduction is one merge old, so any parity run on a machine that has ever fetched remote is comparing against whatever main looked like the first time.

  2. added a commit that references this issue on Aug 29, 2026
    8e83643
  3. LukasGold commented on Aug 31, 2026

    @LukasGold
    ContributorAuthor

    Confirmed fixed by #143, and the fix is better than any of the three options this issue listed.

    All three of mine treated the problem as preventing staleness. Revalidating per file with a conditional request does that, but the part that matters more is "not revalidated: offline" and "not revalidated: upstream unreachable" in the origin string. That makes the unverified state visible rather than merely rare, which is the half I had not thought through: a cache that cannot be revalidated is still usable, and the run should say so instead of silently deciding either way.

    That also keeps the offline guarantee intact. My third option would have traded it away.

    Verified by re-running parity against a fresh clone of both references on current main: 6 passed. The specific drift that surfaced this, a cache 15 days old still declaring x-oold-reverse-default-properties after OO-LD/oold-schema#152 removed it, can no longer produce a silent verdict in either direction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions