Skip to content

speedup: run conformance tests in parallel within each row #5860

Open

Description

@matthewgrossman

User Story

As a contributor waiting on Branch E2E, I want the conformance suite to run several tests at once, so that the conformance jobs stop taking 7–27 minutes and don't get slower with every test we add.

Description

openshell-conformance run runs tests one at a time, each in its own process. Most tests spend their time waiting on sandboxes, not using CPU, so the runner sits mostly idle.

Branch E2E runs the full suite 20 times per run: in 14 jobs that run directly on the runner and in 6 tmachine rows. In run 44803367985 (77 tests), one pass took:

Where Passes Time per pass
Kubernetes on kind 5 16.6–17.5 min
k3s tmachine rows 2 22.5 min of tests (row: 24–27 min)
Podman 5 7.9–13 min
Docker 5 5.8–10 min
VM driver 3 8.4–9.9 min

The k3s rows end every Branch E2E run. Running 4 tests at a time should bring their test time from about 22 min to about 6–8 min.

Proposed change

Context

Definition of Done

  • openshell-conformance run can run N tests at once, and tests marked to run alone never overlap with another test.
  • Each test's output appears as one block in the log.
  • Every test that changes gateway-wide state is marked, and the marking is documented in TESTING.md.
  • Branch E2E passes with N > 1 on the k3s, Docker and Podman rows, and the PR records before/after conformance times.

+++ Agent notes

  • Serial loop: tests/testkit/testkit/src/program/supervisor.rs ("running {to_run} tests one at a time"); per-test timeout and grace come from process::wait(&mut child, options.timeout, options.grace).
  • tests/testkit/testkit/src/program/mod.rs:268: "Tests share one gateway; run them one at a time unless asked otherwise" sets test_threads = Some(1) for in-process runs.
  • Known test that must run alone: settings::global_setting_overrides_sandbox_scope, kept out of conformance in test(conformance): port sandbox lifecycle and API e2e tests to testkit #5431 (see the comment on test(runner): apply per-family nextest config and a thread count to runner rows #5295).
  • Check for fixed sandbox, provider and policy names that would collide when tests run together.
  • Rows: .github/workflows/e2e-suites.yml; runner program args: tests/runner/src/execute.rs (Kind::Program). Jobs on the runner: e2e/rust/e2e-{docker,podman,vm,kubernetes}.sh via OPENSHELL_CONFORMANCE_BIN.
  • Per-test times: JUnit in the artifacts e2e-results-conformance-conformance-<env> of run 44803367985.

+++

Activity

  1. added
    area:buildCI/CD workflows, releases, build targets, packaging, and artifact publication.
    topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.
    on Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:buildCI/CD workflows, releases, build targets, packaging, and artifact publication.topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions