Repository navigation
atlas-eval: a duplicate query key in a run's results object silently replaces that query's ranking — validation never sees the dropped entry #6390
Description
atlas-eval: a duplicate query key in a run's results object silently replaces that query's ranking — validation never sees the dropped entry
pkgs/atlas-search/crates/atlas-core/src/eval.rs is the deterministic offline evaluation harness (#5741): saved rankings are scored against judged qrels, and its reports decide whether a retrieval signal earns a place in the fused ranking and which parameters select_best_signal picks. Its module docs promise strict duplicate discipline — duplicate query ids, duplicate qrels, duplicate docs in a ranking, and duplicate signal names are all rejected at parse/validation time — and EvalData::validate() (called from EvalData::from_json, eval.rs:150) enforces each of them. One duplicate class escapes, because it vanishes before validation runs:
Run.results(eval.rs:125) isBTreeMap<String, Vec<String>>— query id → ranked doc ids. serde's map deserialization inserts key by key, so JSON text that names the same query twice in one run'sresultsobject keeps only the last ranking; the first is silently discarded during parsing.validate()iterates the post-parse map and cannot detect that anything was dropped.- Every other duplicate class is detectable post-parse (vectors keep both copies; struct fields are protected by serde derive, which errors on a repeated field) —
resultskeys are the only input that arrives through a map.
Verified on the laptop with a probe crate (~/git/fleet/probe-eval-dupkey, serde/serde_json, types mirrored field-for-field from eval.rs) against the exact shapes:
{"signal":"dense","results":{"q1":["a","b"],"q1":["c"]}}→ parses OK;results["q1"] == ["c"]— the["a","b"]ranking is gone, with no error at any layer.- Control, duplicate struct field (
"format"twice) → parse errorduplicate field 'format'— confirming the hole is specific to the map-typed field, not a general parser laxity.
Why it matters
The eval document is a file format (versioned fixtures such as pkgs/atlas-search/fixtures/eval-smoke.json, plus generated/hand-merged eval sets per pkgs/atlas-search/eval/README.md). A duplicated query entry — a merge artifact, a generator emitting a query twice, a hand edit — does not produce an error; it produces a wrong report: the signal is scored on the replacement ranking as if that were its output, and nDCG/recall/MRR means and select_best_signal parameter choices are computed from silently truncated evidence. The harness exists to be the trustworthy oracle for signal selection; a silent input-truncation path is exactly the failure its validation section was written to prevent (the same class the deploy registry fix #6384 addressed with an object_pairs_hook: discarded text must fail the load, not vanish).
Note the existing pinning test (validation_rejects_format_violations, eval.rs:~683) builds its inputs with serde_json::json!(...) values — a Value object cannot even represent duplicate keys, so that test style structurally cannot express this case; a regression test must feed from_json raw text.
Acceptance criteria
EvalData::from_jsonrejects, with an error naming the signal and query, anyatlas-eval/v1document whoseresultsobject repeats a query key — at any run.- All currently-valid documents still parse and score identically (including
fixtures/eval-smoke.jsonand the existing test corpus); every existing validation rejection (duplicate doc in a ranking, duplicate qrel, unknown query, grade > 30, duplicate signal) still fires. - A raw-text regression test covers the duplicate-
results-key case (not viajson!values, which cannot express it). - The fix does not change the report format (
atlas-eval-report/v1) or any metric definition.
Related: #6384 (same silently-discarded-duplicate-key class in the deploy registry; fixed with a pairs hook), #5741 (the eval harness's parent issue, closed), #6046 (eval-set/gate design that will build on this harness).
Activity
developing-today commentedon Nov 5, 2026
CODE-1904
dezren39 commentedon Nov 5, 2026
🤖 Agent note · filed
dezren39 commentedon Nov 5, 2026
🤖 Agent note · handed off
opencode73 remaining items
dezren39 commentedon Nov 6, 2026
🤖 Agent note · done
test(atlas-core): property coverage for duplicate results key rejecti…
fix(atlas-core): reject duplicate results keys in atlas-eval runs (#6433
dezren39 commentedon Nov 6, 2026
Agent note — done: fixed and merged
9e20032d0cf3034a2baea117caa60c115119138e); this issue auto-closed by the merge.EvalData::from_jsonnow rejects a repeated query key in any run'sresultsobject with an error naming the signal and query, via a manualDeserializeimpl forRun.origin/mainand passes with the fix; Codex P1 finding (property coverage) fixed in83d1362d36— 300 generated documents over escaped ids, field orderings, and duplicate positions — and Codex re-review of that commit completed with no findings.cargo test -p atlas-core: 31 lib + 5 parity pass;cargo fmtclean.