Skip to content

feat: add a command to check a policy file before creating a sandbox #4035

Open

Description

@mpereira

User Story

As someone writing a sandbox policy,
I want a command that tells me whether the file is valid,
so that I can fix mistakes before I try to create a sandbox with it.

Problem Statement

There's no command that checks a policy file on its own. openshell policy has set, update, get, list, and delete, and sandbox create has no dry run.

From reading the code, the CLI only parses the file, and the gateway does the remaining checks, such as rejecting a root run_as_user. So full validation needs a reachable gateway. sandbox create also connects before it reads the file, so with the gateway down, openshell sandbox create --policy bad.yaml fails with a connection error and never mentions the file.

Reproduction Steps

  1. Write policy.yaml with no version:
    process:
      run_as_user: "1000"
      run_as_group: "1000"
  2. openshell --gateway-endpoint http://127.0.0.1:1 sandbox create --policy policy.yaml, where nothing is listening on that port. It fails with failed to connect to gateway and doesn't mention the file.
  3. openshell-prover check --boundary policy.yaml policy.yaml correctly fails with missing field: version.
  4. Add version: 1 and change both IDs to "0". The same prover command prints result: within_boundary and exits 0.

Impact / Why This Matters

My first policy was missing version: 1. I couldn't tell whether the file was right without a running gateway and a sandbox create, which also pulls the image. And yes I should have read the documentation!

The workaround is openshell-prover check --boundary policy.yaml policy.yaml, with the same file as both arguments. The prover is meant for comparing two policies, not for validating one. On 0.1.2 it catches a missing version and unknown fields, but it accepts run_as_user: "0", which the schema reference says OpenShell rejects. It's also a separate binary that nothing points you to for this.

Proposed Design

A CLI command that takes a policy file, for example:

openshell policy check policy.yaml

It runs locally and doesn't need a gateway. It applies the same validation to the file as sandbox create --policy. A valid file exits 0. An invalid one exits non-zero and prints the same errors that creating a sandbox with it would. Alternative names for the subcommand: lint or validate. This is the same idea as terraform validate, helm lint, or opa check.

A pass means the file is valid, not that a sandbox will start. Checks that depend on the gateway or the image stay where they are, such as registered middleware, attached providers, or the image's USER. The command could also ask a running gateway to run its checks, the way kubectl has --dry-run=client and --dry-run=server. That could be done in follow-up work.

Acceptance Criteria

  • The command works with no gateway configured or running.
  • It rejects everything sandbox create --policy rejects in the file itself, including a missing version, unknown fields, and a root run_as_user.
  • When the file parses, it reports every validation problem in one run, not just the first.
  • The exit code is 0 for a valid file and non-zero otherwise.
  • The policy docs mention it where they describe writing a policy.

Alternatives Considered

  • The prover, as above. It doesn't apply all the validation and isn't meant for this.
  • policy update --dry-run. It previews a merge into a live sandbox's policy. It doesn't take a file, and it needs a sandbox.
  • A gateway-side check only, such as sandbox create --dry-run. The gateway can see more, such as registered middleware and attached providers. But it doesn't work before a gateway is up, or in CI or an editor, and it ties a file check to create's other flags.

Agent Investigation

Used the create-spike skill to check feasibility against main at 021400be8.

Feasible, medium complexity. Most of the file-only validation is already a pure function in openshell-policy, which the CLI already depends on. The work is a subcommand that skips gateway resolution, plus moving a few pure checks out of the gateway so both sides share them.

What validates a policy file today:

Check Where Needs gateway state Reports every problem
File read, YAML limits, unknown fields, version openshell-policy-schema/src/lib.rs No No, stops at the first
Endpoint TLS, enforcement, and access modes to_proto in openshell-policy/src/lib.rs No No, stops at the first
Semantic rules: identity, paths, endpoints, MCP, middleware structure validate_and_canonicalize_sandbox_policy, openshell-policy/src/lib.rs:1873 No Yes
Reserved _provider_ rule names openshell-server/src/grpc/validation.rs:1018 No No, stops at the first
Encoded size over 256 KiB grpc/validation.rs:267, MAX_POLICY_SIZE in grpc/mod.rs:160 No Single check
credential_binding structure grpc/policy.rs:1891 No No, stops at the first
Ambiguous endpoints openshell-policy/src/ambiguity.rs, called from grpc/policy.rs:1864 No, for the file's own rules Yes
Built-in middleware config openshell-supervisor-middleware-builtins/src/lib.rs:29 No No, stops at the first
External middleware, attached providers, global policy, compute driver grpc/sandbox.rs, grpc/policy.rs Yes Out of scope

The CLI runs only the first two rows. The rest run in the gateway on CreateSandbox (grpc/sandbox.rs:548-565 and 620).

What would change:

  • openshell-cli: a new PolicyCommands variant (main.rs:2100) with its own dispatch arm. The shared policy arm calls resolve_gateway and apply_auth at main.rs:3021 before it matches the subcommand. doctor check (main.rs:2715) is the existing pattern for a command that runs without a gateway.
  • openshell-policy: one shared function for file-only validation. None exists today. The gateway chains the checks itself on create and repeats the chain on the policy update paths (grpc/policy.rs:3606 and 3997).
  • openshell-server: the reserved-name, credential_binding, and size checks move into openshell-policy, and the gateway calls the shared function. Otherwise the command and the gateway only match by convention.
  • Docs and skills: docs/how-it-works/policies/manage-policies.mdx, a cross-reference from prover.mdx, and the openshell-cli and generate-sandbox-policy skills.

That's roughly 9 to 13 files. It doesn't touch gateway configuration or anything sensitive to SELinux or AppArmor.

Decisions for a maintainer:

  • Name. doctor check runs locally and profile lint goes through the gateway, so lint would end up meaning two things.
  • Exit codes and output. Plain exit 1, or the prover's split of 1 for an invalid policy and 2 for usage or I/O errors. Also whether to offer --output json.
  • Reporting every problem. The checks that stop at the first failure would need to return lists. The parse stage stopping early is consistent with the acceptance criterion.
  • Built-in middleware config. Checking it locally adds a CLI dependency on the builtins crate. External middleware can only be skipped with a note.
  • Reserved _provider_ names. If they're an error locally, the output of policy get --full fails the check and the output of --base passes.
  • No path argument. Require it, or fall back to OPENSHELL_SANDBOX_POLICY as sandbox create does.
  • Version skew. The result reflects the CLI's rules, which can differ from an older or newer gateway.

Tests:

  • Unit tests on the shared function, including a file with several violations.
  • CLI tests with no gateway configured, using run_isolated in crates/openshell-cli/tests/common/mod.rs.
  • A parity test against the gateway is practical. policy_only_process_identity_integration.rs already runs an in-process gateway.

Checklist

  • I've reviewed existing issues and the published docs
  • This is a design proposal, not a "please build this" request

Activity

  1. shiju-nv commented on Oct 1, 2026

    @shiju-nv
    Collaborator

    I'd call this openshell policy lint policy.yaml. It checks a file and reports errors without applying the policy, and it follows the existing profile lint name. Since profile lint uses the gateway today, the new command should say clearly that it works offline.

    We can reuse openshell-policy-schema for the policy types and bounded parser, then use the semantic checks in openshell-policy and the existing middleware validators. The schema parser alone doesn't cover everything though.

  2. elezar commented on Oct 2, 2026

    @elezar
    Member

    📋 triage-agent

    Classification: validated-feature

    The reporter describes first-hand difficulty checking an authored policy before a gateway is available. The proposed offline file-validation workflow is technically coherent and feasible, with sufficient impact, alternatives, and observable acceptance criteria.

    Independent source review on October 1 covered checkout 021400be8af471f8669369e679de3e18cf0bd672. PolicyCommands (crates/openshell-cli/src/main.rs:2100) has no standalone file validator, and dispatch resolves gateway/authentication before selecting the policy operation (main.rs:3021). Sandbox creation connects at run.rs:552 before loading the policy at run.rs:609, confirming why a connection error masks file errors. A read of remote-main CLI source that day confirmed the command remained absent.

    The bounded authored-policy parser and conversion can run locally. openshell-policy::validate_and_canonicalize_sandbox_policy (src/lib.rs:1873) already collects semantic violations, including unsafe explicit identities (src/lib.rs:1422). Gateway creation adds reserved provider rule-name, canonicalized size, safety, middleware, and credential-policy checks (openshell-server/src/grpc/sandbox.rs:550-568). Exposing only the schema parser would not satisfy the requested validation contract. Existing middleware/extension validation can supply relevant checks, but gateway-dependent services cannot by themselves satisfy the offline use case.

    The key design boundary is clear: validate everything ascertainable from the authored file, while retaining gateway registrations, provider attachments, image identity, and runtime capabilities as contextual checks. Collecting every post-parse problem requires accounting for validators that currently stop at the first error. Shared validation should keep local and gateway rejection behavior consistent; output should describe CLI/gateway version skew and any checks requiring live context. The suggested openshell policy lint policy.yaml interaction is reasonable and follows the maintainer's naming feedback; naming and detailed output remain design decisions for accepted work.

    The prover checks containment rather than equivalent file validity. Closed #4032 explicitly names #4035 as its replacement, so this is the canonical report. The release/duplicate searches performed during investigation found no implementing fix or canonical competing issue; the reported v0.1.2 remains the latest stable release. One impact qualification: rejecting an invalid policy need not pull an image, because gateway validation precedes compute creation; the requirement for a reachable gateway is nevertheless established.

    Impact: Policy authors and potential editor/CI consumers cannot obtain complete local validation through the current CLI. A reachable gateway or self-comparison with the separate prover supplies only an unsuitable or incomplete workaround. This is a feature gap, with no removed command identified. Evidence is strong for the absent workflow and reusable architecture. Binary reproductions were not rerun: the local development CLI wrapper attempted compilation and failed under sandbox restrictions in sccache.

    Next human decision: Decide whether to accept this feature, its ownership and sequencing. If accepted, settle the local validation coverage and output contract before implementation, and verify no-gateway operation, multiple simultaneous violations, and parity for file-only checks. Technical validation does not record acceptance; planning or implementation can be requested separately.

  3. added
    area:cliCLI-related work
    area:policyPolicy engine and policy lifecycle work
    and removed
    state:triage-neededOpened without agent diagnostics and needs triage
    on Oct 2, 2026
  4. letv1nnn commented on Oct 5, 2026

    @letv1nnn
    Contributor

    hey @mpereira, I'd like to take this one unless you are already working on it.

  5. mpereira commented on Oct 5, 2026

    @mpereira
    Author

    Hey @letv1nnn,

    I will work on this one today.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions