Skip to content

feat(sandbox-worker): harness check — run hidden acceptance tests after GOAL_MET, report apart from the agent's claim #121

Closed

Description

@Alter-Igor

Why

svc-coding-agent#4 gate: all 90 runs said GOAL_MET, but about 10% failed the hidden acceptance tests. The service must be able to run tests the agent never saw after the agent claims GOAL_MET, and report the result separately from the agent's claim.

Design (fork, alterspective/sandbox-worker)

  • Nothing the harness runs comes from the box. The agent can write /work/repo and the supervisor's state.json. So the command, the tests archive (with SHA-256) and the agent's claim come from the service, in a harness request written as root after GOAL_MET.
  • supervisor/harness-run.ts (run as root through the E2B SDK):
    • checks the archive SHA-256;
    • copies the repo into a private (0700) folder owned by a separate sbxwharness user, and unpacks the hidden tests there;
    • freezes the agent's processes (SIGSTOP) during the run;
    • runs the command as sbxwharness with a clean env (no SBXW_*, no keys) and a timeout;
    • writes a result: harness.passed, exit code, output tails, plus agentClaim and agreement (claim-confirmed / claim-refuted / harness-error / …).
  • Fails closed: any harness error gives passed: false.
  • Manifest: optional harness: { required: boolean }. It is informational only; the box never supplies the command.

Not in scope

The service side: detecting GOAL_MET, uploading the request and tests, and acting on the result (svc-coding-agent#5).

Done when

  • Unit tests for request parsing, classification and env scrubbing.
  • An end-to-end run in a Linux container: tests pass → claim-confirmed; fail → claim-refuted; SHA mismatch and timeout → harness-error; the agent user cannot read the hidden tests.
  • The contract is posted on svc-coding-agent#5.

Refs Alterspective-Engine/svc-coding-agent#5, Alterspective-Engine/svc-coding-agent#4

Activity

  1. Alter-Igor commented on Oct 5, 2026

    @Alter-Igor
    OwnerAuthor

    🔒 HELD agent=opencodealt-claude-main

  2. Alter-Igor commented on Oct 5, 2026

    @Alter-Igor
    OwnerAuthor

    Project tracking (project-manager pass, Project 16, writer: opencodealt-claude-main, 2026-10-05)

    • WT-PM-05 Priority P2: the independent reviewer (Synapse MCP) is unreachable in this session, so P2 is applied by default. No judgement of my own was substituted.
    • WT-PM-01 Owner + Assignees: defaulted to Alter-Igor. WT-PM-02 Target Date: 2026-10-05, a review-by date.
    • Status In Progress, Stage Current: job 2 of the 2026-10-05 programme list (owner confirmed "A").
    • Where: branch sandbox-harness-check, worktree X:\opencode---sandbox-harness-check.
  3. self-assigned this
    on Oct 5, 2026
  4. Alter-Igor commented on Oct 5, 2026

    @Alter-Igor
    OwnerAuthor

    Done (agent=opencodealt-claude-main). Shipped in #122, squash-merged to dev as 2ff5a34d1b. The dev full run on that commit passed (4/4 shards). Project 16: Done / Completed. The contract is on svc-coding-agent#5, with a change note for timeout = harness-error and the symlink refusal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions