investigation: audit acceptance suite for reclassification — 272 nested runs, ~28% of suite CPU is boot cost

Context

Follow-up to #798 (closed) — the two structural directions it left unexplored, now with fresh numbers after #801-#818 and #830.

Current acceptance suite (61 files, macOS Bash 3.2, 2026-07-19):

Metric	Value
Wall, `--parallel` (CI mode)	17.7s
Wall, sequential	94s
CPU, `--parallel` run	~90s
Nested `./bashunit` spawn sites	~272
Cold start (`--version`)	53ms
Minimal nested run (1 noop test, `--no-parallel`)	91ms

Fixed floor: 272 × 91ms ≈ 25s CPU (~28% of the suite's CPU) is nested-run boot cost, before any fixture test executes. The rest is fixture bodies + parent overhead.

Task: audit, don't optimize (yet)

Phase 1 — categorize all 61 tests/acceptance/*_test.sh files. For each, decide:

A. True CLI-surface test — asserts on process-level behavior: exit codes, rendered console output/snapshots, flag parsing, env-file loading, signal handling, parallel worker behavior. Must stay acceptance. (Fork-census tests, snapshot tests, exit-code tests are all A.)
B. Function-level test in disguise — spawns ./bashunit but really asserts logic reachable by calling bashunit::* functions directly (like tests/functional/ and tests/unit/ do). Candidate for reclassification; each converted spawn saves ≥91ms CPU plus its fixture body cost.
C. Mixed — file contains both; note which tests are B.

Output: a table in this issue (file → A/B/C, spawn count, rough per-file wall time, notes). Per-file timing:

for f in tests/acceptance/*_test.sh; do
  s=$(python3 -c 'import time;print(int(time.time()*1e9))')
  ./bashunit "$f" >/dev/null 2>&1
  e=$(python3 -c 'import time;print(int(time.time()*1e9))')
  echo "$(( (e-s)/1000000 ))ms $f"
done | sort -rn

Phase 2 — file follow-up issues (one per coherent batch of B/C conversions, biggest measured win first). Do NOT convert anything in this issue's PR — the audit table and follow-up issues are the deliverable. A conversion batch only proceeds if Phase 1 shows ≥2s wall or ≥10s CPU win for the batch.

Secondary direction (measure only, likely reject)

Artifact sharing: sourcing src/ is ~31ms of the 53ms cold start; cap across 272 runs ≈ 8.4s CPU. Prior finding (#798/#761): the single-file bin/bashunit build is NOT faster to source (parse time dominates), and lazy-sourcing complicates the release build. Only pursue if Phase 1 leaves it as the dominant remaining cost — record the decision either way.

Constraints

Audit PR touches no runtime code; conversions (follow-ups) are strict TDD, one batch per PR
Reclassified tests must keep asserting the same behavior — no coverage loss; anything asserting on rendered output/exit codes stays acceptance
Fixture naming gotcha: files under tests/acceptance/fixtures/ must not match *[tT]est.sh (discovery-scan snapshots)
Bash 3.0+; full quality gate on every PR (./bashunit tests/, --parallel, --parallel --simple --strict, make sa, make lint)

Acceptance criteria

All 61 files categorized A/B/C with spawn counts and per-file timings, posted as a table on this issue
Measured estimate of total reclaimable wall/CPU from B+C conversions
Follow-up conversion issues filed (or explicit "not worth it" verdict with numbers)
Artifact-sharing direction: pursue/reject decision recorded with measurements

Phase 1 audit results (2026-07-19, macOS Bash 3.2)

Method: per-file wall timing (sequential, isolated runs) + spawn-site counts + content review. Note: spawn counts via ./bashunit/"$BASHUNIT" grep undercount files using $BASHUNIT_BIN/$BIN variables (e.g. rerun_failed, bashunitrc) — categorization was done by reading the files, not the counts.

Head (≥1s) — 20 files, ~77s of the 94s sequential suite

File	Wall	Spawns	Cat	Notes
bashunit_test.sh	29.7s	~7 doc runs	A	25s+ is the `bashunit doc` perf bug → filed #832 (echo\|sed forked per line of the 1580-line assertions.md, ~5.1s/invocation). After #832: ~5s
install_test.sh	9.9s	install.sh runs	A (network)	Real curl/wget downloads by design; skips without network. Irreducible locally
bashunit_upgrade_test.sh	4.0s	upgrade runs	A (network)	Same
bashunit_timeout_test.sh	3.7s	4	A	Real watchdog timing semantics
bashunit_summary_output_test.sh	3.5s	5	A	Rendered summary output
bashunit_run_forks_test.sh	3.3s	8	A	Fork-census budgets (CI-enforced)
bashunit_parallel_consistency_test.sh	2.7s	4	A	Parallel worker behavior
bashunit_rerun_failed_test.sh	2.4s	via $BASHUNIT_BIN	A	Cache file + process lifecycle
bashunit_random_order_test.sh	2.1s	4	A	Seed reproducibility across runs
bashunit_retry_test.sh	1.9s	6	A	Retry annotations in output
bashunit_shard_test.sh	1.8s	4	A	Shard partitioning across runs
bashunit_report_json_test.sh	1.7s	4	A	Report file emission
bashunit_risky_test.sh	1.7s	6	A	Exit codes + output
bashunit_fail_test.sh	1.6s	12	A	Failure rendering
bashunit_profile_test.sh	1.5s	3	A
bashunit_script_temp_file_cleanup_test.sh	1.4s	2	A	Process-level temp lifecycle
bashunit_fn_cleanup_test.sh	1.3s	2	A	#829 regression (main-shell state)
parallel_spy_parallel_test.sh	1.3s	1	A
bashunit_path_test.sh	1.0s	8	A	Discovery/path CLI
bashunit_setup_error_test.sh	1.0s	9	A	Hook failure rendering + exit codes

Tail (<1s) — 42 files, ~17s combined

All A (snapshot/output/exit-code/flag tests: assert_*, direct_fn_call, tag, filter, strict, diff, tap/junit/gha/html reports, no_color, syntax_error, init, coldstart_forks, …) except:

mock_test.sh — 142ms, zero spawns: cross-test clear_mocks runner behavior, already functional-style in practice. Moving it saves ~0ms. Leave.

No B files found. No C files found. Every spawning test asserts on process-level behavior: rendered output, snapshots, exit codes, report files, worker/timing semantics. This suite was already reclassification-clean — consistent with the #801-#818 campaign having moved the cheap wins.

Verdicts

Reclassification: NOT worth it. Zero B candidates; the conversion direction is empty. Criterion (≥2s wall / ≥10s CPU per batch) unreachable.
The actionable finding is perf(doc): bashunit doc forks echo|sed per line of assertions.md — ~5s to print a file (single awk pass) #832 (doc.sh per-line forks): removes ~25s sequential / big CPU slice from bashunit_test.sh — more than the entire theoretical reclassification ceiling.
Artifact sharing: REJECT. Sourcing src/ is ~31ms of the 53ms cold start; ceiling ≈ 8.4s CPU across 272 nested runs ≈ ~1s wall in parallel — and it complicates the single-file release build. Matches the perf: reduce test-suite execution time (tracking) #761/perf: reduce acceptance-suite cold-start cost (258 nested ./bashunit runs) #798 prior finding (parse time dominates; bin/bashunit is not faster to source).
Remaining floor after perf(doc): bashunit doc forks echo|sed per line of assertions.md — ~5s to print a file (single awk pass) #832: install/upgrade network tests (~14s sequential, skip-guarded) + ~272 × 91ms boot cost, both inherent to E2E coverage.

Closing: audit deliverable complete, follow-up filed (#832).

Audit complete (see table above): no reclassification candidates, artifact-sharing rejected with numbers, actionable finding filed as #832.

Navigation Menu

Repository navigation

investigation: audit acceptance suite for reclassification — 272 nested runs, ~28% of suite CPU is boot cost #831

Context

Task: audit, don't optimize (yet)

Secondary direction (measure only, likely reject)

Constraints

Acceptance criteria

Phase 1 audit results (2026-07-19, macOS Bash 3.2)

Head (≥1s) — 20 files, ~77s of the 94s sequential suite

Tail (<1s) — 42 files, ~17s combined

Verdicts

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

investigation: audit acceptance suite for reclassification — 272 nested runs, ~28% of suite CPU is boot cost #831

Description

Context

Task: audit, don't optimize (yet)

Secondary direction (measure only, likely reject)

Constraints

Acceptance criteria

Activity

Chemaclass commented on Jul 19, 2026

Phase 1 audit results (2026-07-19, macOS Bash 3.2)

Head (≥1s) — 20 files, ~77s of the 94s sequential suite

Tail (<1s) — 42 files, ~17s combined

Verdicts

Chemaclass commented on Jul 19, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions