fix(benchmarks): run bench_regression.sh under macOS bash 3.2 and sweep the shell scripts

[cdeust]

Symptom

benchmarks/lib/bench_regression.sh fails under the macOS system bash 3.2.57 with bash: wt_dir: unbound variable, so tests_py/benchmarks/test_bench_regression_datasets.py passes on a Mac only when a newer bash comes first on PATH (macos-latest run https://github.com/cdeust/Cortex/actions/runs/38008915302: 2 failures; with Homebrew bash 5 first on PATH, run 38015387291: 5 passed).

Measured here with the real library under /bin/bash (3.2.57): the failing runner case returns 1 instead of the runner's 17, and the missing-dataset case leaves its baseline worktree registered.

Root cause

benchmarks/lib/bench_regression.sh line 76 (before): the subshell that runs the baseline benchmarks sets trap '... "$wt_dir" ...' EXIT with a single-quoted body. wt_dir is local to run_baseline_benchmarks, and the body is only expanded when the trap fires. Reduced to a library of 8 lines (f() { local wt_dir; wt_dir="$(mktemp -d)"; ( trap 'echo "trap sees: [${wt_dir-UNSET}]"' EXIT; /usr/bin/false ); local status=$?; }) sourced by bash -c 'set -euo pipefail; source ./lib-src.sh; f':

/bin/bash 3.2.57 -c            trap sees: [UNSET]
/bin/bash 3.2.57 as a script   trap sees: [/var/folders/.../tmp.LgxgxgXd7U]
bash 5.3.15 -c                 trap sees: [/var/folders/.../tmp.W2k2ey2zwh]

So under bash -c bash 3.2.57 has already dropped the function's locals when the subshell's EXIT trap fires after a failing command; $wt_dir is unset, set -u aborts the trap before git worktree remove runs, and the shell's status becomes 1. The tests drive the library through bash -c, so they hit the failing mode.

Fix

The trap body is built where the trap is set: trap "$(printf 'git -C %q worktree remove --force %q ...' "$REPO_ROOT" "$wt_dir")" EXIT. printf %q keeps a path with spaces or quotes intact through the re-parse the trap does, so the trap no longer depends on any variable being alive when it fires.

Sibling search result that needed a change: scripts/mutation_check.sh took an empty first argument into "${TEST_ARR[@]}" (a different result per bash, measured below); it is now a usage error with exit 2.

Failing before, passing after

The test now runs the library under every distinct bash a user can have (/bin/bash and the first bash on PATH), and a third test uses a repository path with spaces, a quote and a $.

Before (the new parametrization against the old library, this Mac, /bin/bash 3.2.57 and Homebrew 5.3.15):

FAILED ...::test_missing_dataset_fails_before_runner_and_cleans_worktree[bash=/bin/bash]
FAILED ...::test_first_runner_failure_is_preserved_and_cleans_worktree[bash=/bin/bash]
E   assert 1 == 17   (stderr: '/bin/bash: wt_dir: unbound variable')
E   assert 2 == 1     (a registered baseline worktree left behind)
2 failed, 8 passed in 69.19s

After (head 11704b5, same machine, /bin/bash 3.2.57 and Homebrew 5.3.15 both exercised): tests_py/benchmarks/test_bench_regression_datasets.py and test_mutation_check_arguments.py: 14 passed.

shellcheck 0.11.0 on the two changed scripts (CI's shellcheck covers only workflow run blocks): on the base, bench_regression.sh reports 78:9 SC2164 (info level); on the previous head 982e6f8 it reported SC1073 and SC1072 errors for the directive # shellcheck disable=SC2064 -- early expansion ..., which shellcheck cannot parse. The directive now sits alone on its own line with the reason on the line above, as in scripts/lib/setup_py_step.sh, and the cd that SC2164 flagged exits on failure. On 11704b5 both scripts print nothing and exit 0; mutation_check.sh prints nothing on base and head.

Other gates on 11704b5: uvx ruff@0.16.6 check and format --check exit 0, check_craftsmanship.py --base origin/main prints OK, check_project_wiki.py exit 0, /bin/bash -n exits 0. pyright over mcp_server/ (untouched by this PR) was 0 errors on 982e6f8.

The full local pytest was NOT run for this PR: the machine was loaded by other sessions, and the diff changes two shell scripts and two test files. The nine test files that execute these scripts passed on 982e6f8 (62 passed, /bin/bash first on PATH); the later commit changes a comment line, one || exit 1 and the changelog. The first complete full run is CI.

Empty test list in scripts/mutation_check.sh, before: /bin/bash: assert 0 == 2 (mutmut ran with one empty test name and exited 0); Homebrew bash 5.3: assert 1 == 2 ("failed to collect stats"). After: both exit 2 with error: the test list (first argument) is empty. (tests_py/scripts/test_mutation_check_arguments.py, 2 passed).

Completion Ledger

New branch or error path	Test that asserts it
trap removes the baseline worktree when the first runner fails, bash 3.2 and newer	test_first_runner_failure_is_preserved_and_cleans_worktree[bash=...] (exit 17 kept, one worktree left)
trap removes the worktree when the dataset preflight fails	test_missing_dataset_fails_before_runner_and_cleans_worktree[bash=...]
success path unchanged, runners called in order	test_baseline_uses_exact_selected_ignored_inputs[...][bash=...]
trap body survives a path with spaces, a quote and $	test_cleanup_trap_quotes_a_repository_path_with_spaces_and_quotes[bash=...]
empty test list refused with exit 2 and the message, no "unbound variable"	test_an_empty_test_list_is_a_usage_error[bash=...]

Sibling search

Scope: every shell file in the repository, found by extension or by a bash/sh/zsh shebang over git ls-files (22 files, list in the first command), the 13 hook and install commands of .claude-plugin/plugin.json, the Codex plugin hooks, and .github/ workflows.

Command 1, files and syntax under bash 3.2.57:

$ /bin/bash --version | head -1
GNU bash, version 3.2.57(1)-release (arm64-apple-darwin25)
$ for f in <22 files>; do /bin/bash -n "$f"; done
OK on all 22: .clusterfuzzlite/build.sh benchmarks/energy/run.sh benchmarks/lib/bench_container.sh
benchmarks/lib/bench_only.sh benchmarks/lib/bench_regression.sh benchmarks/quick_test.sh
benchmarks/repro_longmemeval.sh benchmarks/reproduce.sh benchmarks/trust_factor_sweep.sh
docker/entrypoint.sh docker/run.sh scripts/bench_variance.sh scripts/docker_smoke.sh
scripts/install-plugin.sh scripts/lib/install_python_deps.sh scripts/lib/precache_embedding_model.sh
scripts/lib/setup_py_step.sh scripts/mutation_check.sh scripts/refresh_rustup_hash.sh
scripts/setup.sh scripts/spawn-agent.sh scripts/update-plugin.sh

Command 2, construct scan (grep -E over the 22 files, comments excluded). Zero hits for each of: declare/local/typeset -A, local -n/declare -n, declare -g, ${v,,} ${v^^} ${v,} ${v^}, ${v@Q} and the other @ transforms, mapfile, readarray, |&, &>>, coproc, ;& and ;;&, [-N] array indices, ${v: -N} negative offsets, [[ -v ]], wait -n, read -i/-N, fractional read -t, shopt -s globstar|lastpipe|inherit_errexit|compat*, exec {fd}>, $BASHPID $EPOCHSECONDS $EPOCHREALTIME $SRANDOM $BASH_ARGV0, printf '%(..)T', {01..10}, $'\u....'.

Hits that needed a decision (every other class above had none):

Site	Construct	Disposition
benchmarks/lib/bench_regression.sh:76	EXIT trap expanding a local	fixed in this PR
scripts/mutation_check.sh:25,86	"${TEST_ARR[@]}" under set -u, empty when the first argument is empty	fixed in this PR: exit 2 with a message
benchmarks/lib/bench_only.sh:42	"${raw[@]}" under set -u	no change: ONLY is non-empty there (the function returns above on empty) and an empty token is rejected
benchmarks/reproduce.sh:320,362	"${entry[@]}"	no change: always initialised with the script path
benchmarks/reproduce.sh:344,351	"${V4_MECHANISMS[@]}", "${mech_args[@]}"	no change: literal non-empty list at line 85, mech_args filled by the loop
benchmarks/reproduce.sh:516-525	${lm_args[@]+"${lm_args[@]}"} and siblings	no change: already the bash 3.2 safe form
benchmarks/reproduce.sh:527	"${be_args[@]}"	no change: initialised to (--split 100K) at line 509
benchmarks/reproduce.sh:504, benchmarks/repro_longmemeval.sh:130, scripts/mutation_check.sh:22	trap <function> EXIT	no change: each function reads script-level globals (LOCK_DIR, started_container, CONTAINER, BAK, PY, RUN_LOG, ROOT), none a function local
benchmarks/energy/run.sh	#!/bin/zsh, print -u2	not bash; zsh ships with macOS; out of scope
docker/entrypoint.sh, docker/run.sh	bash	run in the image or by docker, not the host /bin/bash; no flagged construct anyway
inline commands in .claude-plugin/plugin.json (11 bash -c hook commands)	$(command -v ...), ${VAR:-}, [ -z ]	no change: run under PATH=/bin:..., each stops at its own plugin-root guard message and exits 1, no syntax error
plugins/hypermnesia-mcp-codex/hooks/hooks.json	python3 ... direct	no shell construct
.github/workflows/*.yml	no macOS job (grep -rn "macos|macOS" .github/workflows/*.yml prints nothing)	Linux and Windows (Git Bash 5) bash only
111 inline shell snippets: the command strings of .claude-plugin/plugin.json (13), the Codex and deprecated hooks.json (14), and the run: blocks of the workflows and composite actions (84)	bash 4+ constructs	extracted with a script and each checked with /bin/bash -n (3.2.57) after replacing ${{ }} with a literal, plus a regex for associative arrays, mapfile/readarray, ${v,,} ${v^^}, &>>, `

The empty-array hazard itself, on this Mac, to show the class is real:

$ /bin/bash -c 'set -u; a=(); echo "${a[@]}"'
/bin/bash: a[@]: unbound variable
$ /bin/bash -c 'set -u; a=(); echo ${a[@]+"${a[@]}"}; echo ok-guarded'
ok-guarded

Command 3, execution under the system bash. The nine test files that run these scripts, with /bin first on PATH:

$ PATH=/bin:/usr/bin:$PATH pytest tests_py/benchmarks/test_bench_regression_datasets.py \
    tests_py/benchmarks/test_reproduce_only_validation.py tests_py/benchmarks/test_reranker_matrix_runner.py \
    tests_py/benchmarks/test_reranker_matrix_runtime.py tests_py/scripts/test_ci_codex_runtime.py \
    tests_py/scripts/test_install_plugin_marker_order.py tests_py/scripts/test_installer_upgrade_single_version.py \
    tests_py/scripts/test_precache_embedding_model_step.py tests_py/scripts/test_setup_py_failure_message.py
62 passed in 234.67s   (exit 0, on 982e6f8b, including test_mutation_check_arguments.py)

Adjacent, not changed here: BSD versus GNU userland differences (for example sed -i) are a separate class from bash versions; git grep -n "sed -i" -- '*.sh' prints nothing.

What could not be verified

• Only the library functions and the nine test files above were executed under bash 3.2.57. The long benchmark scripts (reproduce.sh, repro_longmemeval.sh, trust_factor_sweep.sh, docker_smoke.sh, bench_variance.sh) need Docker, a database and datasets; for them the evidence is bash -n plus the construct scan, not a run.
• The mechanism of "locals gone when the trap fires" was reduced to a minimal script and reproduced under bash -c; I did not find the bash 3.2 source line responsible.
• No bash older than 3.2.57 and no macOS ARM-versus-Intel difference was tried.

Decision points for the ADR register

None. The trap fix and the usage error follow ADR-0067 and ADR-0770 unchanged.

Fixes #690

🤖 Generated with Claude Code

https://claude.ai/code/session_01DXMXvHLm94B6pA9FkWWmzu

[cdeust]

ZETETIC-REVIEW: REQUEST_CHANGES
982e6f8

Scope: PR #696 (Fixes #690). Code and tests are correct. Two small text defects inside the diff block approval; neither needs code logic changes.

Blocking (text only)

1. CHANGELOG.md (added entry under "### Fixed"): "Every other shell script was checked for bash 3.2 constructs (none found; benchmarks/energy/run.sh is zsh)." contradicts the diff: the sweep found one real hit and scripts/mutation_check.sh changed behaviour (empty test list: exit 0 on /bin/bash 3.2.57, exit 1 on bash 5.3, now exit 2 with a message on both). That behaviour change has no changelog line. Fix: replace the sentence with the mutation_check.sh usage-error change.
2. benchmarks/lib/bench_regression.sh:81 # shellcheck disable=SC2064 -- early expansion is the point. shellcheck 0.11.0 (installed here) cannot parse it: SC1073/SC1072 errors, plus SC1009 at line 73. The base file only gave an SC2164 info. Repo convention is # shellcheck disable=SC2086 # reason (.github/actions/uv-locked-install/action.yml:39). Fix: # shellcheck disable=SC2064 # early expansion is the point. CI shellcheck runs on workflow run: blocks only, so this does not fail CI; the directive is still wrong.

Q1 root cause and trap (verified)

• /bin/bash is 3.2.57 here; which -a bash: /opt/homebrew/bin/bash (5.3.x) first, then /bin/bash.
• Reproduced on origin/main's script with the head tests: 3 failed, 9 passed, all on [bash=/bin/bash], stderr /bin/bash: wt_dir: unbound variable (the 2 original tests plus the new quoting test). With the fix: 12 passed.
• The mechanism (3.2 drops function locals under bash -c before the subshell EXIT trap fires) is the author's measurement; I reproduced the symptom, not the 3.2 source line. The fix does not depend on the mechanism: the trap body holds no variable reference when it fires.
• printf %q checked through trap-eval on 3.2.57 and 5.3: space, single and double quote, $(touch PWNED), backtick, $x, leading -, unicode, embedded newline, backslashes all round-trip intact; no command ran (no injection). A leading - REPO_ROOT is an argument to git -C; wt_dir is absolute (mktemp under REPO_ROOT).
• Exit paths (reduction with the same trap form, both bashes): normal exit, failing command under set -e, SIGTERM and SIGINT delivered to the subshell all run the trap; SIGKILL cannot (same as before). The subshell is the trap's scope and wt_dir is assigned before the trap is set, so the set-before-assign case does not arise. The parent also runs git worktree remove after the subshell (unchanged, a second path).
• Simpler form: a single-quoted trap with a global wt_dir would also work but leaves the path re-expanded at fire time; build-at-set-time with %q is correct and small. Not a finding.

Q2 sibling search (independently re-run)

• 22 shell files (git ls-files '*.sh' plus shebang files without extension). /bin/bash -n clean on all 22.
• My own regex scan over the 21 bash files (energy/run.sh is zsh): zero bash 4+ constructs. The only regex hits were ${X:-0.005} style defaults (false positives of my negative-offset pattern) and &>/dev/null in setup.sh (valid in 3.2).
• Empty array under set -u: every "${arr[@]}" is either ${a[@]+"${a[@]}"}-guarded (bench_regression.sh, reproduce.sh) or provably non-empty (reproduce.sh entry/mech_args/be_args; trust_factor_sweep.sh GRID; bench_only.sh raw, since empty ONLY returns early and ",," is rejected before the read). mutation_check.sh was the only unguarded case and is now handled. "$@" is exempt in 3.2.
• Not re-verified by me: the 13 plugin.json hook commands, the Codex hooks.json and workflow bash -c snippets (author's claim of no hits only).

Q3 mutation_check.sh

Base measured on this Mac: /bin/bash exit 0 with TEST_ARR[@]: unbound variable (masked by the cleanup trap), bash 5.3 exit 1. Head: exit 2 with the message on both. The check precedes mktemp/cp pyproject, so no state is touched. Whitespace-only lists also land in the empty-array branch. Callers: git grep finds only docs, ADR 0770 and test comments naming the script; no CI or script depends on exit 0. Exit 2 matches the existing usage branch ($# -lt 2). New test fails on base for both bashes (author's table; base behaviour reproduced by me), passes after.

Q4 tests

• Head: test_bench_regression_datasets.py + test_mutation_check_arguments.py: 14 passed (7.9s) with brew bash first; 106 passed over those two plus 8 other script-running test files with /bin/bash first on PATH.
• Mutation by hand: reverting to the base script fails 3 tests on /bin/bash; unquoting the %q (%s) fails the quoting test on both bashes (2 failed, 10 passed). The other three tests do not catch an unquoted path (only the new one does), which is the intended split.
• Assert lines 10 -> 13 in the datasets test, nothing removed or weakened, no sleeps, hermetic (tmp_path git repos, fake runner on PATH).
• uptime was checked (load dropping from 111 to 9); my pytest runs finished; I started no background processes. Processes seen at the end belonged to other sessions and were left alone.

Q5 craftsmanship and PR body

• scripts/check_craftsmanship.py --base origin/main: "Craftsmanship gate: OK" (I read the output). ruff@0.16.6 check and format --check on the two test files: clean. check_no_deps_invariant.py: clean (126 files). datasets test file is 165 lines (<300), bench_regression.sh 147.
• PR body: ledger has 5 rows each mapped to a test; sibling table present; no ADR needed (script fix and test change, no new constant or layer): confirmed. The PR body's "none found" line carries the same inconsistency as the changelog (it lists the mutation_check hit separately, so only the changelog is wrong).
• macOS CI: the two failing tests should pass. The runner's /bin/bash is 3.2.57, so the lead should see in the macOS job log these ids passing: test_missing_dataset_fails_before_runner_and_cleans_worktree[bash=/bin/bash], test_first_runner_failure_is_preserved_and_cleans_worktree[bash=/bin/bash], the new quoting test [bash=/bin/bash] and test_an_empty_test_list_is_a_usage_error[bash=/bin/bash], with 12 plus 2 passing if a second bash is first on PATH (6 plus 1 per bash otherwise). A run with only [bash=/bin/bash] ids and no wt_dir error is the confirmation.

Not verified

The 3.2 source line behind the locals drop; the long Docker/dataset benchmark scripts beyond bash -n and the scan; the plugin.json/hooks.json/workflow snippets; Linux CI behaviour; behaviour under bash 4.x (only 3.2.57 and 5.3 available).

Cleanup: review worktree removed (git worktree list shows none); the disk-hygiene register call refused the detached worktree, so I removed it with git directly; no processes of mine remain.

[cdeust]

ZETETIC-REVIEW: APPROVE
11704b5

Delta review of PR #696 against the previous verdict (982e6f8, two text defects). Both are fixed; nothing new found.

Checks

1. Delta scope: git diff 982e6f8b..11704b53 touches CHANGELOG.md (+5/-2) and benchmarks/lib/bench_regression.sh (+3/-2) only. In the script: the shellcheck directive is split onto its own line with the reason on the line above (# shellcheck disable=SC2064), and cd "$wt_dir" became cd "$wt_dir" || exit 1. No other change. The || exit 1 is a small unrequested addition, harmless: the subshell has set -e and a failed cd would exit anyway, and the trap still fires on exit. Not a finding.
2. CHANGELOG: the "none found" sentence is gone. A new Fixed entry says scripts/mutation_check.sh refuses an empty test list: exit 0 under /bin/bash 3.2.57, exit 1 under bash 5.3, now an error message and exit 2. This matches my earlier measurement and the script (scripts/mutation_check.sh:17-20, exit 2). No other changelog claim contradicts the diff (5 files vs origin/main: CHANGELOG, bench_regression.sh, mutation_check.sh, two tests).
3. shellcheck 0.11.0 (/opt/homebrew/bin/shellcheck), run on base (git show origin/main) and head: base bench_regression.sh gives one SC2164 on cd "$wt_dir" (line 78); base and head mutation_check.sh give nothing; head bench_regression.sh gives nothing, rc 0. SC1072 and SC1073 are gone and there is no new finding.
4. Snippet scan, my own script (python, yaml+json, git ls-files): 111 snippets (plugin.json 13, hooks.json files 12+1+1, workflow and action run: blocks 84), same count as the PR body. Each ${{ }} replaced by a literal, /bin/bash -n (3.2.57) on each: no syntax error. Also the 11 inner bash -c strings of plugin.json parsed separately via shlex: clean. Regex for declare -A, local -n, declare -n, mapfile, readarray, |&, &>>, wait -n, [[ -v, case-conversion and @-transform expansions, coproc, ;;&, globstar, lastpipe: exactly one hit, ci.yml:344 mapfile, in job mcp-host-config which has runs-on: ubuntu-latest (bash 5). This matches the author's table. Hooks run with no macOS job, so no change needed.
5. Tests with /bin/bash first on PATH: test_bench_regression_datasets.py + test_mutation_check_arguments.py: 7 passed (3.7s). Because /bin/bash is first, the PATH-bash parametrization is the same bash, so 7 not 14. check_craftsmanship.py --base origin/main: "Craftsmanship gate: OK" (read). uvx ruff@0.16.6 check and format --check on the two test files: clean. check_no_deps_invariant.py: clean (126 files). /bin/bash -n on the script: ok.

Not verified

CI. Behaviour under brew bash 5.3 in this delta (verified earlier on 982e6f8; the delta is a comment line, || exit 1, and changelog). Full pytest. Linux/Windows runs. The 3.2 source line behind the locals drop.

Housekeeping

Worktree .claude/worktrees/review-696b: disk_hygiene register-worktree refused it (detached head), so I removed it with git worktree remove and prune; no review-696b entry remains. I started no background processes. Load was 4.7 when I ran the tests.