Skip to content

explore/query relevance: a generic token's exact name-match overboosts in peripheral dirs — follow-up to #746 #982

New issue
New issue
Closed
#1463
Closed
explore/query relevance: a generic token's exact name-match overboosts in peripheral dirs — follow-up to #746#982
#1463

Description

@michaelHMK
michaelHMK
opened on Jun 24, 2026

Summary

Follow-up to #746 (and #720 / #745). Those landed the project-name token down-weighting. #746 also floated a more general idea:

discount a token that matches a large fraction of indexed paths/symbols (it carries ~no signal but currently dominates).

That corpus-frequency discount is still not implemented on main (1.1.0). As a result, a generic non-stopword query token that happens to exactly match a symbol name (e.g. a bare usage) wins a large nameMatchBonus and crowds out the on-target product code — both in codegraph query / codegraph_search (no dampening at all) and, to a lesser but still user-visible degree, in codegraph_explore (where an existing dampening fires but is insufficient).

What actually happens today (corrected root cause)

There are two scoring paths, and they behave differently — the original write-up conflated them:

  1. codegraph query / codegraph_search (db/queries.ts searchNodes): score = bm25 + kindBonus + scorePathRelevance + nameMatchBonus. The only low-signal-token handling is dropping the project name (scorePathRelevance, via deriveProjectNameTokens). There is no corpus-frequency / generic-token discount, so a bare usage symbol ranks at the very top.

  2. codegraph_explore (context/index.ts findRelevantContext): this path does already demote generic exact-name matches — isDistinctiveIdentifier(token) is false for a plain lowercase word like usage, so a common-word exact match with no 2nd corroborating term gets score *= 0.3 (and a generic single-term match gets *= 0.6). This demotion fires for the reported case — and is still not enough, because nameMatchBonus hands a bare exact-name match a score so high that 0.3× of it still beats product symbols that don't literally contain the token.

So the accurate statement is not "generic-token downweighting isn't done" — it's "the existing demotion is a binary heuristic (distinctive-vs-common + 2-term corroboration) that can't counter a large nameMatchBonus, and the plain query/search path has no such demotion at all." The #746 general corpus-frequency discount is what's missing.

The lever (nameMatchBonus, src/search/query-utils.ts)

For a query token that exactly equals a symbol name:

  • whole query (whitespace-stripped) === name → 80
  • in a multi-word query, a token === name → 60 ← this is the case here (the earlier draft said "80")

STOP_WORDS deliberately doesn't filter common symbol names (get/set/usage/update/status), so these tokens keep full weight even when they match thousands of symbols. matchesNonProductionDir's de-prioritized set is hardcoded (integration/sample/example/fixture/benchmark/demo) and not extensible, so a peripheral tree like optional-skills/.../scripts/ gets no de-prioritization.

Minimal reproduction (public, no private repo needed)

Layout — product code with no symbol literally named usage, plus two standalone helpers that each have a module-level def usage():

apps/desktop/statusbar/StatusBar.ts            class DesktopStatusBar { render, refresh, mount }
apps/desktop/statusbar/StatusBarController.ts  class StatusBarController
apps/desktop/context/ContextWindowMeter.ts     class ContextWindowMeter { read, recompute }, estimateTokens
apps/desktop/context/format.ts                 formatTokens
gateway/server/server.ts, packages/core/util/strings.ts   (filler)
optional-skills/bodyfat/scripts/body_calc.py   def usage(): print("usage: ...")
optional-skills/nutrition/scripts/macro_calc.py def usage(): print("usage: ...")

codegraph init + codegraph index, then query desktop status bar context window usage:

Run Entry Where the usage() helpers land
A. codegraph query (no dampening) searchNodes ranks 1 and 2 (≈7626% vs product's top ≈5104%)
B. codegraph explore (0.3× demotion active) findRelevantContext top two Source Code blocks — push StatusBarController.ts + format.ts out of the section
C. control: same query with usage→tokens codegraph explore helpers don't appear; 4 product files fill the section
D. causal probe: rename usage()→print_usage(), re-index codegraph explore helpers vanish entirely; product code reclaims every slot

D isolates the cause to a single signal: removing the exact token match (so the 60-point nameMatchBonus no longer applies) removes the symptom. The existing explore-path demotion was active the whole time and didn't prevent B.

Why the existing user-side levers don't help

Proposal

Primary (the #746 follow-up — the part that's genuinely missing):

  • Add a corpus-frequency / IDF-style discount so a query token matching a large fraction of indexed symbols/paths contributes proportionally less to nameMatchBonus (and scorePathRelevance). This lowers the base an exact-name match starts from, which the explore path's existing 0.3× demotion then multiplies — fixing the "0.3× of a huge number is still huge" gap, and also fixing the query/search path that has no demotion at all.

Complementary / independently shippable, lower risk — keep these two distinct (they have different semantics):

  • Deprioritize / scope (a ranking lever): make the non-production-dir de-prioritization user-extensible via codegraph.json (e.g. deprioritize: ["optional-skills/", "scripts/"]), and/or add a --within <glob> / --exclude <glob> scope to explore / query + the MCP input schema.
  • Exclude (an index/recall lever) is a bigger product decision — name it separately if pursued; it's not required to fix the ranking symptom above.

Validation caveat

This touches the scorer for every query, so per codegraph/CLAUDE.md and the note in #746 it must be A/B'd with scripts/agent-eval/run-all.sh across the README repos plus a control before merge. Edge case to preserve: a query where the "generic" token genuinely IS the discriminating term (an actual query about a usage() reporter) — IDF should discount, not erase, the signal.

Activity

  1. michaelHMK
    changed the title [-]explore/query relevance: a generic non-stopword query token's exact name-match in a peripheral (non-test) dir buries on-target results — follow-up to #746[/-] [+]explore/query relevance: a generic token's exact name-match overboosts in peripheral dirs — follow-up to #746[/+] on Jun 24, 2026

  2. maxmilian commented on Jul 6, 2026

    @maxmilian
    maxmilian
    on Jul 6, 2026
    Contributor

    Reproduced the root cause on current main — the analysis holds. For anyone triaging, the three grounding points:

    • nameMatchBonus (src/search/query-utils.ts:343) hands a flat 80 (whole-query === name) / 60 (multi-word token === name), independent of how many symbols share that name.
    • searchNodes (src/db/queries.ts:1024-1030) is purely additive: bm25 + kindBonus + scorePathRelevance + nameMatchBonus, with no corpus-frequency term.
    • The exact-name supplement just above it (queries.ts:994-1020) re-injects any name = term node seeded at maxFtsScore, so a bare usage symbol enters rescoring at maxBM25 + kindBonus(function=10) + 60 — which is why it crowds out product code that doesn't literally contain the token. #746 closed with only the project-name down-weight landed; the general low-discriminative-token discount it floated is still absent.

    One correction to the issue's premise (it's ~2 weeks old): codegraph.json does have exclude now — it landed in 45d3293 as a gitignore-style, index-level skip (src/extraction/index.ts:489). So the immediate symptom has a workaround today: "exclude": ["optional-skills/"] keeps those usage() helpers out of the index entirely. That's the recall lever the issue set aside, though — it removes the symbols rather than ranking them correctly.

    So the genuinely-open, recall-preserving gap is still the ranking side, and it splits cleanly into two independently-shippable pieces:

    1. Extensible deprioritize (low risk, ranking-only): matchesNonProductionDir (query-utils.ts:325) is a hardcoded set (integration/sample/example/fixture/benchmark/demo). Making it user-extensible via codegraph.json (e.g. deprioritize: ["optional-skills/", "scripts/"]) is a small, contained change with no scorer-wide blast radius — it only lowers scorePathRelevance for opted-in trees.
    2. Corpus-frequency / IDF discount (the real Down-weight project-name / low-discriminative query tokens in search ranking (follow-up to #720) #746 follow-up, higher risk): scales nameMatchBonus/scorePathRelevance down by how large a fraction of indexed symbols a token matches. This touches every query, so per CLAUDE.md + the #746 note it needs an A/B via scripts/agent-eval across the README repos + a control that preserves the case where the "generic" token genuinely IS the discriminating term (a real query about a usage() reporter should discount, not erase).

    @colbymchenry — happy to take (1) as a self-contained PR (extensible deprioritize, with the config validation mirroring the existing exclude/includeIgnored handling in project-config.ts and an eval A/B to show no regression), if that direction is welcome and if @michaelHMK isn't already on it. Would you rather see (1) first as the low-risk step, or go straight at the IDF discount in (2)?

  3. JerryPeace commented on Jul 27, 2026

    @JerryPeace
    JerryPeace
    on Jul 27, 2026

    Another amplification surface for the same root cause, plus one negative data point that rules out a workaround.

    The prompt-hook path makes this non-optional for the user

    The existing analysis covers codegraph query / codegraph_search / codegraph_explore — paths where a human deliberately asked for a symbol. The UserPromptSubmit prompt-hook reuses the same ranking, but the "query" is the user's natural-language prompt. So a generic token that happens to be a common identifier triggers injection on prompts that reference no code at all, and the tokens are consumed silently on every turn with no user action to filter them.

    Concrete case, v1.5.0, a ~785-file multi-language monorepo (Python / TypeScript / C#):

    • Prompt: a meta discussion about indexing strategy and query relevance — zero references to any symbol, file, or feature in the repo.
    • Injected: 15.3 KB of source plus call graph and blast radius for unrelated UI/query-layer symbols, selected purely because the prompt contained the words index and query.

    The pattern is consistent for tokens that are simultaneously a domain concept and a common identifier: index, query, review, context, status. Those are exactly the words that dominate when the conversation is about the tooling or process rather than about product code — so the false-positive rate is highest in precisely the sessions where the injection has the least value.

    Negative data point: scoping the index does not help

    Before finding this issue I assumed the noise came from cross-project matches in the monorepo and that narrowing the index would fix it. Measured both ways, same query, same version:

    full index one 179-file sub-project excluded
    Files 785 606
    Nodes 12,633 8,928
    Edges 30,692 23,055
    Hits for the generic token 10 10 (identical set)

    Removing 23% of the corpus changed the hit set by zero, because the over-boosted matches were already first-party code that merely contained the token in a symbol name. So this is purely the ranking issue described above — not corpus composition, and not something users can work around by splitting or scoping their index.

    Worth noting the excluded sub-project also cost ~21% of the genuine cross-directory edges between the two remaining projects (566 → 449), so index scoping is actively harmful as a mitigation here, independent of it not working.

    Why the corpus-frequency discount from #746 would fix the injection case cleanly

    A flat nameMatchBonus of 60/80 is indifferent to how many symbols share the name, which is the whole signal that matters for this class of token. For the prompt-hook path specifically, a corpus-frequency term would also give a natural way to inject nothing rather than something weak: if every candidate's score is dominated by a high-frequency exact-name match, that is a good indicator the prompt was not structural at all.

    Happy to re-measure on a build with the discount if that helps validate it.

  4. maxmilian commented on Jul 28, 2026

    @maxmilian
    maxmilian
    on Jul 28, 2026
    Contributor

    @JerryPeace — the prompt-hook framing and the index-scoping table are both new to this thread, and the second one in particular closes off the workaround I would otherwise have suggested. Thanks for measuring it rather than asserting it.

    Two PRs are now up against #982. They are deliberately split, matching the distinction the issue draws between the two levers:

    One thing I should disclose before you spend time measuring, because it bears directly on your offer: #1462 is close to inert on this issue's own 8-file repro. Only two symbols there are named usage, so the IDF scale lands around 0.8 and the helper scripts still outrank the product code (62.8 vs 51.2). The repro demonstrates the mechanism — flat exact-name bonus dominating — but not the condition this particular lever keys on, which is the name actually being common. That is why the second PR exists; the 8-file shape is #1463's fixture, not #1462's.

    Your corpus is the one that would actually exercise it. index and query across 785 files and 12,633 nodes is exactly the high-df shape the discount is built for, and I have no way to produce that locally.

    If you are still up for it:

    gh pr checkout 1462   # branch: fix/982-idf-name-match
npm ci && npm run build
node dist/bin/codegraph.js index    # re-index so stats match

    The numbers that would tell us the most, same prompt as your 15.3 KB case, before and after:

    1. injected size (KB) from the prompt-hook path
    2. the hit set for the generic token — did the 10 change, or just reorder
    3. df for index / query (how many symbols carry that exact name) and the total node count, since the scale is driven entirely by that ratio

    If (3) turns out to be small even in a 785-file repo, that is itself a useful result: it would mean the discount is the wrong lever for the prompt-hook case and the path-based one carries it instead. Either way the measurement decides it, and a negative result is worth having on the record before a maintainer spends review time.

  5. maxmilian
    added a commit that references this issue on Aug 13, 2026

    fix(search): derive the deprioritize name-bonus damping instead of pi…

    8de1034
  6. colbymchenry
    added a commit that references this issue on Aug 22, 2026

    feat(config): add codegraph.json "deprioritize" for ranking-only path…

    1d9de88
  7. colbymchenry
    closed this as completedin #1463on Aug 22, 2026
  8. Goodhhw
    added a commit that references this issue on Aug 29, 2026

    feat(config): add codegraph.json "deprioritize" for ranking-only path…

    ce4a116
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions