Skip to content

Audit intelligence evidence provenance and quote traceability - #129

Open
basicmi wants to merge 1 commit into
masterfrom
intelligence-evidence-quality-audit
Open

basicmi wants to merge 1 commit into
masterfrom
intelligence-evidence-quality-audit

Conversation

@basicmi

@basicmi basicmi commented Nov 17, 2026

Copy link
Copy Markdown
Owner

Motivation

The existing golden-set evaluation measures relevance classification and sentence extraction, but it does not audit the provenance carried by live extracted Claims. Add a separate operational audit to make quote traceability, source metadata completeness and source-origin annotations visible without pretending to score factual truth.

Changes

  • Add a read-only evidence-quality audit joining discovery candidates, extracted Claims and the evidence graph.
  • Check whether each Claim's quote is present in the fetched excerpt and whether its statement matches that quote.
  • Check source metadata consistency between each Claim and its originating discovery candidate.
  • Report provenance completeness, distinct source documents/publishers, declared independence labels, evidence/assertion classes and reviewer-assigned origin annotations.
  • Summarize duplicate wording clusters while explicitly retaining the non-corroboration warning.
  • Run the audit in the weekly discovery workflow, publish Markdown to the Actions summary and upload JSON/Markdown artifacts.
  • Add unit tests and document the metrics and limitations.

Safety and interpretation

This is not a composite quality score. Publisher diversity, declared independence labels, quote presence and source classes do not prove independent corroboration, peer review, semantic entailment, measurement validity or factual truth. Every Claim remains unverified; no product, company, benchmark or model-support registry is modified.

Tests

Covers grounded/missing quotes, statement/quote mismatch, source metadata mismatch, unknown independence, reviewer-origin annotations, duplicate wording across documents, fail-closed cross-stage checks and interpretation-limit messaging.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant