Skip to content

feat(report): expose the file surface a finding landed on - #703

Open
Harbor404 wants to merge 1 commit into
NVIDIA:mainfrom
Harbor404:feat/finding-file-surface
Open

Harbor404 wants to merge 1 commit into
NVIDIA:mainfrom
Harbor404:feat/finding-file-surface

Conversation

@Harbor404

Copy link
Copy Markdown

Summary

Fixes #326.

Reports said which file and line a finding landed on, but not which surface
that line belongs to. The issue author manually labelled all 4415 findings from
65 real skill packages: code 1442, tests 981, docs 980, comments 412,
instructions 325, config 275.

The request is to expose, not filter. The reasoning is sound: tests/ looks
like noise but is also one of the easiest places to hide a payload, suppressing
it would trade real detections for cosmetics; a rule like SC1 hits config 100%
of the time and config is exactly where it should fire; and instructions
must stay distinct from docs because SKILL.md is what the agent executes,
not prose about it. Making the classification data lets consumers own their own
ranking and suppression risk, rather than baking that judgement into the
scanner.

Changes

  • New src/skillspector/surface.py — infer_surface(file_path, line_text=None) returning code | instructions | docs | tests | comments | config. Precedence: tests → instructions (SKILL.md) → comments →
    docs → config → code.
    • comments only applies within code/config: a comment inside a test file
      is still tests, and an HTML comment in README.md is still docs.
    • Manifest basenames are checked before docs, so requirements.txt is not
      read as prose because of .txt.
    • Extension-less doc stems only, so scripts/install.sh is not read as a doc
      because its name contains install.
  • models.py — Finding.surface, emitted by to_dict().
  • nodes/analyzers/static_runner.py — pass the matched line text into
    infer_surface.
  • nodes/deduplicate.py — record a surface per occurrence.
  • nodes/report.py — properties.surface in SARIF; expansion uses each
    occurrence's own label.
  • New tests/test_finding_surface.py — 34 parametrised cases.

A real bug found while testing

The first version passed unit tests but failed end-to-end: the same trigger
string placed in SKILL.md, docs/guide.md, tests/test_skill.py,
config/settings.json and a comment line in scripts/helper.py labelled the
helper as code. infer_surface returned comments when called directly, so
the label was being lost after conversion. Instrumenting _expand_occurrences
showed the cause: deduplicate() groups by rule_id + fingerprint + classification_metadata, so all five files collapsed into one finding
represented by SKILL.md, with the other four hanging off as occurrences. When
the report expanded them, only the representative's label survived.

Fix: each occurrence records its own label (same-file reuses the finding's,
cross-file falls back to pure path inference), and both expansion and SARIF
prefer the occurrence's own. The label is stored in the occurrence dict's
value, not its key — as a key, two labels on one (file, line) could add a
row and break the "do not change finding counts" constraint.

Testing

uv run --no-sync pytest tests/nodes/analyzers -q
# 4251 passed, 5 deselected, 4 xfailed

uv run --no-sync pytest -q
# 8513 passed, 17 skipped, 134 deselected, 4 xfailed

uv run --no-sync ruff check src/ tests/          # All checks passed!
uv run --no-sync ruff format --check src/ tests/ # 283 files already formatted
uv run --no-sync mypy src/skillspector/surface.py src/skillspector/models.py \
  src/skillspector/nodes/report.py src/skillspector/nodes/deduplicate.py
# Success: no issues found in 4 source files

(mypy reports one error in static_runner.py — _paragraph_ranges's
tuple[()] typing — which reproduces on unmodified origin/main.)

End-to-end on a purpose-built 5-surface skill package (--no-llm):

JSON issues: 10   score: 83 CRITICAL
  SKILL.md             -> instructions
  config/settings.json -> config
  docs/guide.md        -> docs
  scripts/helper.py    -> comments
  tests/test_skill.py  -> tests
SARIF results: 10, properties.surface matching

No behavior regression: scanning the same package on unmodified
origin/main and on this branch produced field-identical issues (apart from the
new surface and the per-run finding_id) and an identical risk_assessment
(score 83 / CRITICAL / DO_NOT_INSTALL, severity and rule distributions both
{'CRITICAL': 10} / {'P5': 10}), with identical suppression.

Notes for Reviewer

  • Size: 411 changed lines rather than the ~300 I usually aim for. Broken
    down: 223 lines of production code (13 of which are the repo's required
    license header, 17 a module docstring, ~40 the extension/filename constant
    tables), 173 lines of parametrised tests, 15 lines of edits. Compressing to
    300 would mean dropping one of the six surfaces or the comment detection —
    both of which the issue explicitly asks for. Flagging rather than silently
    cutting.
  • Comment detection is line-start only. Inline comments (x = 1 # payload)
    stay code, since no reliable column is retained; documented in the
    infer_surface docstring.
  • The comment extension table covers common languages; unlisted extensions fall
    back to path-based inference. Adding a language is one entry in
    _COMMENT_MARKERS.
  • docs vs config has some judgement in it: Makefile and Dockerfile are
    deliberately treated as executable build scripts (code), license files as
    docs. Changing the call is a constant-table edit.
  • Findings created by non-static analyzers (LLM/MCP/semantic) have no line
    text, so they use path-only inference and never land in comments. That is
    intentional — not guessing at information we do not have.

Add an advisory `surface` label to every reported finding: `code`,
`instructions` (`SKILL.md`), `docs`, `tests`, `comments`, or `config`. It is
derived from the finding's path, plus the matched line for the `comments`
case, and it is exposed rather than filtered on: severity, confidence, the
risk score, and the finding count are unchanged.

Both report formats carry it -- JSON `issues[].surface` and SARIF
`properties.surface` -- so a consumer can sort a report by surface, diff two
scans per surface, or write a local suppression policy that takes on the risk
explicitly instead of the scanner taking it on their behalf.

Exact-match dedup now records a label per occurrence, so aggregating the same
match across files does not mislabel the occurrences that live elsewhere.

Fixes NVIDIA#326.

Signed-off-by: Harbor404 <2657212322@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Expose the file surface a finding landed on (code / instructions / docs / tests / config)

1 participant