Repository navigation
fix: merge_headers multi-tile merge, strict env-var expansion, stale example run names - #826
Merged
Merged
Conversation
In tile-level mode (EXP_BASE_DIR set), merge_headers_runner only processed input_file_list[0], silently ignoring every other tile's exp_numbers file. This never showed on CANFAR v2.0, where each tile runs in its own job with a single input; in a monolithic run the serial runner receives all tiles' exp_numbers files in one call and only the first tile got a log_exp_headers-<tile>.sqlite. Loop over all inputs, producing one per-tile sqlite each. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
os.path.expandvars silently leaves unset $VAR references as literal strings, turning e.g. an unset $SP_EXP into a relative path and an opaque downstream failure. Expand strictly instead: any environment variable referenced in a config value that is unset now raises a ValueError naming the variable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Several INPUT_DIR / CHECK_EXISTING_DIR entries reference run names that no sibling example config produces, breaking the documented tile and exposure sequences out of the box: - run_sp_Git:get_images_runner -> run_sp_tile_Git (config_tile_Ma_onthefly, config_tile_Sx, config_tile_Sx_nomask); the tile images come from config_tile_Git_vos.ini (RUN_NAME run_sp_tile_Git), while run_sp_Git is the headers-only download run. - run_sp_Ma_tile:mask_runner -> run_sp_tile_Ma (config_tile_Sx); matches config_tile_Ma_onthefly.ini's RUN_NAME. - run_sp_Ma_exp -> run_sp_exp_Ma (config_exp_Ma_onthefly CHECK_EXISTING_DIR, config_exp_psfex commented INPUT_DIR); matches config_exp_Ma_onthefly.ini's RUN_NAME. (run_sp_Ma_tile/run_sp_Ma_exp only exist as outputs of the canfar combine_runs.bash workflow, not in the plain example sequence.) run_sp_combined_flag in config_Rc.ini is left as is: it is a user-created combined directory documented in docs/source/random_cat.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
7 tasks
martinkilbinger
approved these changes
Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three small quality-of-life fixes for failure modes that cost real debugging time during the Nibi test run in #808.
1.
merge_headers: merge all tiles in serial tile-level modeProblem: in serial tile-level runs,
merge_headersonly merged the first tile's header sqlite, silently dropping the rest — downstream modules then failed mysteriously on missing exposures.Fix: loop over all tile inputs. On the Nibi run this takes the merged output from 1 to 10 sqlite files, matching the 10 tiles processed.
2.
config.py: raise a clear error on unset env varsProblem:
getexpandedpassed literal$VARstrings through when the environment variable was unset, producing paths like$SP_RUN/outputthat fail much later with confusing file-not-found errors.Fix: new
_expandvars_strict—getexpandednow raisesValueErrornaming every unset variable at config-read time.Tested in the runtime container: with
SP_TEST_VARset, expansion yields/scratch/foo/output; unset, it raisesEnvironment variable(s) SP_TEST_VAR referenced in config value '$SP_TEST_VAR/output' not set.3.
example/cfis: correct stale run-name referencesProblem: several example configs reference run names that no sibling config produces (
run_sp_Git,run_sp_Ma_tile,run_sp_Ma_exp), so following the examples verbatim fails.Fix: updated to the names actually produced (
run_sp_tile_Git,run_sp_tile_Ma,run_sp_exp_Ma). A grep audit confirms zero remaining run-name references unproduced by siblings (run_sp_combined_flaginconfig_Rc.iniis deliberately kept — it is a user-created directory documented indocs/source/random_cat.md).Found during the Nibi test run in #808.
— Claude (Fable) on behalf of Cail.
🤖 Generated with Claude Code