[v4.0.0] Phase 9: Cross-Agent Validator & Disagreement Resolution #280

Overview

Implement an adversarial Cross-Agent Validator that cross-checks proposed hypotheses against cluster facts to detect hallucinations and false positives.

Part of Milestone v4.0.0 — Phase 9 of the Multi-Agent SRE Architecture.

⚠️ IMPORTANT CONTRIBUTOR & PR INSTRUCTIONS

TARGET BRANCH: All Pull Requests implementing this phase MUST target branch v4.0.0 (DO NOT target main).
PR TITLE: feat(agents): Phase 9 - Cross-Agent Validator and Disagreement Resolution
PR SCOPE: Validation engine and contradiction detection.

1. Description of What Has to Be Done

LLMs exhibit confirmation bias—once a hypothesis is formed, models easily overlook contradictory facts. The Validator acts as an adversarial check: it specifically searches for facts in the EvidenceBundle that disprove the Lead Investigator's top hypothesis.

Contributors must:

1. Implement CrossAgentValidator with heuristic contradiction checks.
2. Check for hard factual contradictions (e.g. hypothesis claims OOMKill, but exit code was 1; hypothesis claims DNS failure, but CoreDNS is 100% healthy).
3. If contradictions are found, downgrade the confidence score and attach the contradiction reason.

2. Desired Outcome & Expected Behavior

Expected Outcome

• The validator either confirms the hypothesis (approved=True) or rejects/downgrades it (approved=False).
• If rejected, the reason is surfaced clearly:

  {
    "approved": false,
    "confidence_score": 0.25,
    "reason": "Hypothesis claims OOMKilled, but container exit code was 1 (application panic), not 137."
  }

3. The Implementation Plan & Architectural Blueprint

Files to Create and Modify

• [NEW] agents/validation/__init__.py: Package init.
• [NEW] agents/validation/validator.py: CrossAgentValidator and ValidationResult.
• [NEW] agents/tests/test_validator.py: Contradiction test suite.

4. Detailed Task Breakdown

• Task 9.1: Implement ValidationResult (validator.py)

  • Dataclass containing approved: bool, confidence_score: float, and reason: str.

• Task 9.2: Implement CrossAgentValidator (validator.py)

  • Implement ground-truth assertion rules against the EvidenceBundle.
  • Cross-check exit codes, restart counts, probe thresholds, and node conditions.

• Task 9.3: Unit Tests (test_validator.py)

  • Test valid hypothesis passes with approval.
  • Test invalid hypothesis with contradiction is flagged and penalized.

5. Technical Specifications & Concrete Code Signatures

# agents/validation/validator.py
from typing import Dict, Any
from agents.core.evidence import EvidenceBundle
from agents.synthesis.hypotheses import Hypothesis

class ValidationResult:
    def __init__(self, approved: bool, confidence_score: float, reason: str):
        self.approved = approved
        self.confidence_score = confidence_score
        self.reason = reason

class CrossAgentValidator:
    def validate(self, top_hypothesis: Hypothesis, bundle: EvidenceBundle) -> ValidationResult:
        if "OOMKilled" in top_hypothesis.description:
            status = bundle.get("ev.pod.container.status")
            if status and status.data:
                last_exit = status.data.get("lastState", {}).get("terminated", {}).get("exitCode")
                if last_exit and last_exit != 137:
                    return ValidationResult(
                        approved=False,
                        confidence_score=0.2,
                        reason=f"Hypothesis claims OOMKilled but container exit code was {last_exit}, not 137."
                    )
        return ValidationResult(approved=True, confidence_score=top_hypothesis.likelihood, reason="No contradictions found.")

6. Verification & Acceptance Checklist

• Run python3 -m unittest agents/tests/test_validator.py — all tests pass.
• PR targets branch v4.0.0.