test(conformance): move scenario files into Cargo tests

.github/workflows/branch-checks.yml

@@ -141,6 +141,7 @@ jobs:
         shell: nix develop -c bash -euo pipefail {0}
         run: |
           cargo fmt --all -- --check
+          cargo fmt --manifest-path e2e/suites/conformance/Cargo.toml --all -- --check
           cargo fmt --manifest-path e2e/rust/Cargo.toml --all -- --check
           cargo fmt --manifest-path examples/governance-interceptor/Cargo.toml --all -- --check
           cargo fmt --manifest-path examples/supervisor-middleware-content-guard/Cargo.toml --all -- --check
@@ -196,6 +197,7 @@ jobs:
         shell: nix develop -c bash -euo pipefail {0}
         run: |
           cargo clippy --locked --workspace --all-targets -- -D warnings
+          cargo clippy --locked --manifest-path e2e/suites/conformance/Cargo.toml --workspace --all-targets -- -D warnings
           cargo clippy --locked --manifest-path e2e/rust/Cargo.toml --all-targets -- -D warnings
           cargo clippy --locked --manifest-path examples/governance-interceptor/Cargo.toml --all-targets -- -D warnings
           cargo clippy --locked --manifest-path examples/supervisor-middleware-content-guard/Cargo.toml --all-targets -- -D warnings
@@ -235,6 +237,7 @@ jobs:
           OPENSHELL_TELEMETRY_ENABLED: "false"
         run: |
           cargo nextest run --locked --profile ci --workspace --features openshell-server/test-support
+          cargo nextest run --locked --config-file .config/nextest.toml --profile ci --manifest-path e2e/suites/conformance/Cargo.toml --package openshell-test-conformance-cli -E 'binary(policy_advisor) & test(draft_assertion)'
           cargo nextest run --locked --config-file .config/nextest.toml --profile ci --manifest-path examples/supervisor-middleware-content-guard/Cargo.toml
 
   rust-build-modes:

AGENTS.md

@@ -22,7 +22,7 @@ Do not rely on this file for a full inventory. The detailed public and contribut
 | Path | Components | Purpose |
 |------|-----------|---------|
 | `crates/openshell-cli/` | CLI binary | User-facing command-line interface |
-| `crates/openshell-conformance/` | CLI conformance library | Reusable driver-agnostic scenarios and command runner |
+| `crates/openshell-conformance/` | Conformance test support | Command execution, polling, diagnostics, and cleanup |
 | `crates/openshell-server/` | Gateway server | Control-plane API, sandbox lifecycle, auth boundary |
 | `crates/openshell-sandbox/` | Sandbox runtime | Capability-free workload launcher, process identity, and seccomp-mediated I/O |
 | `crates/openshell-supervisor/` | Supervisor runtime | Gateway session, policy evaluation, credentials, and upstream networking |
@@ -61,6 +61,8 @@ Do not rely on this file for a full inventory. The detailed public and contribut
 | `python/openshell/` | Python SDK | Python bindings and CLI packaging |
 | `sdk/typescript/` | TypeScript SDK | Native Connect client, curated sandbox API, and generated protobuf types |
 | `proto/` | Protobuf definitions | gRPC service contracts |
+| `e2e/suites/` | Behavioral E2E suites | Directly authored conformance stories |
+| `tests/` | Test provisioning | Target configuration, artifact construction, and Ansible setup |
 | `deploy/` | Docker, Helm, K8s | Dockerfiles, Helm chart, manifests |
 | `docs/` | Published docs | MDX pages, navigation, and content assets |
 | `fern/` | Docs site config | Fern site config, components, and theme assets |

CI.md

@@ -575,6 +575,14 @@ merge.
 Do not add the informational Actionlint, Zizmor, Dependency Review, or CodeQL
 jobs to the required status list while they remain in observation mode.
 
+## Conformance source checks
+
+Branch Rust checks format and lint the separate
+`e2e/suites/conformance` workspace.
+The Rust test jobs include the conformance runner in the root workspace and select the pure policy assertion
+tests in the conformance CLI package with the CI nextest profile, without a gateway. Gateway-backed CLI tests retain their driver E2E and installed-artifact
+integration lanes; source unit-test jobs do not execute those entry points.
+
 ## Nix download recovery
 
 Jobs that enter the development shell enable `prepare-shell: "true"` on

TESTING.md

@@ -8,6 +8,20 @@ mise run e2e           # End-to-end tests (starts a Docker-backed gateway)
 mise run ci            # Everything: lint, compile checks, and tests
 ```
 
+## Test Families
+
+- Lint checks formatting, style, and static rules.
+- Unit tests verify a component in isolation.
+- Integration tests verify interactions between components.
+- End-to-end tests exercise a configured OpenShell target through a client.
+- Benchmarks measure performance or scale separately from behavioral correctness.
+
+Conformance stories under `e2e/suites/conformance/` verify required public behavior.
+The [conformance guide](e2e/suites/conformance/README.md) owns their authoring and
+execution guidance. Driver and feature suites remain under `tests/suites/`;
+existing mixed suites remain under `e2e/rust/`. `tests/` also supplies provisioning
+and artifact construction.
+
 ## Test Layout
 
 ```text
@@ -16,6 +30,9 @@ crates/*/tests/        # Rust integration tests
 python/openshell/      # Python unit tests (*_test.py suffix)
 e2e/python/            # Python E2E tests (test_*.py prefix)
 e2e/rust/              # Rust CLI E2E tests
+e2e/suites/conformance/ # Conformance stories
+tests/suites/          # Existing feature and driver suites
+tests/                 # Target provisioning and artifact construction
 ```
 
 ## Rust Tests
@@ -210,10 +227,16 @@ When more than one test needs this behavior, put the transport in the shared
 Rust e2e harness and require callers to use it instead of duplicating DNS,
 HTTP `Host`, TLS SNI, and mTLS handling.
 
+Conformance command execution, polling, diagnostics, and cleanup remain in
+`crates/openshell-conformance`. Story implementations live directly under
+`e2e/suites/conformance/cli/tests`. `mise run test:rust` includes runner and pure
+policy assertion tests without requiring a gateway. Existing Rust e2e tests
+retain their own harness.
+
 Suites:
 
 - Common suite (`--features e2e`) - driver-neutral CLI behavior, sandbox lifecycle, sync, port forwarding, policy, and provider tests.
-- CLI conformance (`tests/suites/conformance`) - portable Cargo tests for
+- CLI conformance (`e2e/suites/conformance`) - portable Cargo tests for
   lifecycle, mechanistic drafts, file transfer, and the sandbox-local API,
   including agent-authored permission requests. Driver E2E runs the complete
   Cargo test package. The installed-artifact conformance suite runs the same
@@ -251,7 +274,7 @@ cargo build --package openshell-cli
 OPENSHELL_BIN="$PWD/target/debug/openshell" \
   cargo test \
     --locked \
-    --manifest-path tests/suites/conformance/Cargo.toml \
+    --manifest-path e2e/suites/conformance/Cargo.toml \
     --package openshell-test-conformance-cli \
     --no-fail-fast \
     -- \

crates/openshell-conformance/Cargo.toml

@@ -3,7 +3,7 @@
 
 [package]
 name = "openshell-conformance"
-description = "Reusable OpenShell CLI conformance scenarios and runner"
+description = "OpenShell CLI conformance command runner and test support"
 version.workspace = true
 edition.workspace = true
 rust-version.workspace = true

crates/openshell-conformance/README.md

@@ -0,0 +1,10 @@
+# Conformance test support
+
+This crate provides the existing CLI command runner, bounded polling, structured
+output parsing, diagnostics, and explicit resource cleanup for conformance tests.
+
+Conformance stories and their assertions live directly in
+[`e2e/suites/conformance`](../../e2e/suites/conformance/README.md). This crate does
+not define scenarios or maintain a registry; Cargo discovers the tests.
+
+Runner consolidation with the existing Rust e2e harness remains separate work.

crates/openshell-conformance/src/lib.rs

@@ -1,17 +1,14 @@
 // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 // SPDX-License-Identifier: Apache-2.0
 
-//! Reusable support for portable `OpenShell` CLI conformance scenarios.
+//! Command execution, polling, diagnostics, and cleanup for `OpenShell` conformance tests.
 
 pub mod executor;
-mod scenarios;
 
 use std::collections::BTreeSet;
 use std::error::Error;
 use std::fmt;
-use std::future::Future;
 use std::path::PathBuf;
-use std::pin::Pin;
 use std::process::ExitStatus;
 use std::sync::Arc;
 use std::time::{Duration, Instant};
@@ -23,28 +20,6 @@ use tokio::time::sleep;
 
 use self::executor::{CliExecutionError, CliExecutor, ProcessCli};
 
-pub use scenarios::{
-    FILE_TRANSFER_GIT_FILTERING_SCENARIO, FILE_TRANSFER_PATH_SAFETY_SCENARIO,
-    FILE_TRANSFER_ROUND_TRIP_SCENARIO, MECHANISTIC_PROPOSAL_SCENARIO,
-    NEW_HOSTNAME_PROPOSAL_SCENARIO, POLICY_LOCAL_SCENARIO, SANDBOX_LIFECYCLE_SCENARIO,
-    SMOKE_SCENARIO,
-};
-
-/// A portable conformance scenario exercised by the Cargo test suite.
-#[derive(Debug)]
-pub struct Scenario {
-    pub name: &'static str,
-    run: for<'a> fn(&'a mut OpenShellRunner) -> ScenarioFuture<'a>,
-}
-
-pub type ScenarioFuture<'a> = Pin<Box<dyn Future<Output = Result<(), String>> + Send + 'a>>;
-
-impl Scenario {
-    pub async fn run(&self, runner: &mut OpenShellRunner) -> Result<(), String> {
-        (self.run)(runner).await
-    }
-}
-
 const CLEANUP_TIMEOUT: Duration = Duration::from_mins(2);
 pub const STATUS_TIMEOUT: Duration = Duration::from_secs(30);
 const GATEWAY_STATUS_ATTEMPT_TIMEOUT: Duration = Duration::from_secs(10);

e2e/suites/conformance/Cargo.toml

@@ -0,0 +1,35 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+[workspace]
+resolver = "2"
+members = ["cli"]
+
+[workspace.lints.rust]
+unsafe_code = "warn"
+rust_2018_idioms = { level = "warn", priority = -1 }
+trivial_casts = "warn"
+trivial_numeric_casts = "warn"
+unused_lifetimes = "warn"
+unused_qualifications = "warn"
+
+[workspace.lints.clippy]
+all = { level = "warn", priority = -1 }
+pedantic = { level = "warn", priority = -1 }
+nursery = { level = "warn", priority = -1 }
+
+# Allow certain pedantic lints that are too noisy
+module_name_repetitions = "allow"
+must_use_candidate = "allow"
+missing_errors_doc = "allow"
+missing_panics_doc = "allow"
+
+# Allow noisy nursery lints
+significant_drop_tightening = "allow"  # Often gives incorrect suggestions
+missing_const_for_fn = "allow"  # Too noisy for async code patterns
+
+# Allow noisy pedantic lints
+too_many_lines = "allow"  # Function length limits are subjective
+needless_pass_by_value = "allow"  # Common pattern in async handlers
+ref_option = "allow"  # Common pattern for optional references
+missing_fields_in_debug = "allow"  # Manual Debug impls often intentionally omit fields

e2e/suites/conformance/README.md

@@ -0,0 +1,43 @@
+# OpenShell conformance suite
+
+Conformance tests exercise required public behavior against a prepared gateway.
+Cargo discovers and runs the tests directly. Each area retains its existing
+scenario implementation and helpers in one integration-test file:
+`file_transfer.rs`, `lifecycle.rs`, `policy_advisor.rs`, and `smoke.rs`.
+
+The runner, process executor, polling, diagnostics, and cleanup remain in
+`crates/openshell-conformance`. That crate no longer defines scenarios or a
+registry. Existing Rust e2e, driver, and feature consumers retain their harnesses
+and locations. Story-module organization and shared-runner consolidation are
+separate follow-ups.
+
+## Run
+
+Supply the matching candidate CLI and a configured gateway:
+
+```shell
+OPENSHELL_BIN=/absolute/path/to/openshell \
+  cargo test --locked --manifest-path e2e/suites/conformance/Cargo.toml \
+    --package openshell-test-conformance-cli -- --test-threads=1 --nocapture
+```
+
+Use `--test file_transfer`, `--test lifecycle`, `--test policy_advisor`, or
+`--test smoke` to select an area. Existing test names remain available for
+focused selection. The lifecycle test retains combined stop/start persistence
+and stopped-sandbox deletion checks.
+
+## Source checks
+
+```shell
+cargo test --locked --package openshell-conformance
+cargo test --locked --manifest-path e2e/suites/conformance/Cargo.toml \
+  --package openshell-test-conformance-cli --test policy_advisor draft_assertion
+```
+
+The runner unit tests and three policy assertions run without a gateway. Other
+integration tests require target preparation and run in the existing driver E2E
+and installed nextest archive lanes. Tests register resources before creation,
+use unique names, and always call `OpenShellRunner::finish` so cleanup failures
+remain visible. Bounded polling observes asynchronous state; failed assertions
+are not retried. Live tests and public internet dependencies remain outside
+source unit checks.

tests/suites/conformance/cli/Cargo.toml → e2e/suites/conformance/cli/Cargo.toml

@@ -10,5 +10,11 @@ license = "Apache-2.0"
 repository = "https://github.com/NVIDIA/OpenShell"
 
 [dependencies]
+serde = { version = "1", features = ["derive"] }
+serde_json = "1"
+tempfile = "3"
 openshell-conformance = { path = "../../../../crates/openshell-conformance" }
-tokio = { version = "1.43", features = ["macros", "rt"] }
+tokio = { version = "1.43", features = ["macros", "process", "rt", "time"] }
+
+[lints]
+workspace = true