Repository navigation
config: rewire the MCCD exposure chain after mask removal; static config checks - #918
Merged
Merged
Conversation
commit eb93838 deleted mask_runner but left config_exp_mccd.ini reading its pipeline_flag output and skipping mask_query, so the MCCD exposure chain could not run and never carried MASK_EXT. Mirror the working PSFEx chain: SExtractor reads the split flag image directly, mask_query_runner sits between SExtractor and setools, and setools reads mask_query's sexcat_ext output. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK
Extend the example-config parse smoke test to every workflow/config/cfis config, and add a check that every *_runner token in those configs (in MODULE, INPUT_MODULE, an INPUT_DIR path, or a *_RUNNER option) names a runner that actually exists under src/shapepipe/modules/. This would have caught config_exp_mccd.ini's stale reference to the deleted mask_runner. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK
mccd_plots_runner produces mean-shape and histogram plots only; no rho statistics code exists anywhere in mccd_package. Remove the docstring paragraph advertising a third plot series that was never built. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK
Same defect class as the mask rewiring: config_exp_mccd.ini could not run as written. [MCCD_MERGE_STARCAT_RUNNER] named a section merge_starcat_runner never reads — its config section is its own module name upper-cased, MERGE_STARCAT_RUNNER, and it needs PSF_MODEL, not the MCCD config file's CONFIG_PATH/MODE/VERBOSE trio the wrong section carried. [MCCD_PLOTS_RUNNER] was missing the PSF key that mccd_plots_runner.py reads unconditionally. Both fixed following example/cfis/config_MsPl_mccd.ini, the one place this was already done correctly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK
…itionally Add a static check, per workflow config: for every module in MODULE, resolve its config section the way FileHandler does (module name upper-cased, or <module>_run_<n> upper-cased for a repeated module), then parse that runner's source to find config.get*() calls made unconditionally (not nested in an if/for/while/try/with, and without a fallback= kwarg) and assert each such key is present in the section. Confirmed against the pre-fix config_exp_mccd.ini (git show a1404f6) that this independently catches both the missing MERGE_STARCAT_RUNNER section and the missing PSF key. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK
RUN_DATETIME was left commented out, so a clean run appends a timestamp to RUN_NAME and every hardcoded run_sp_exp_SxSePsf path (mask_query_runner, setools_runner) never resolves. Set it to False, matching config_exp_psfex.ini. Also drop a dated telecon reference and a comment contrasting MERGE_STARCAT_RUNNER against the wrong section name it replaced — both describe how a decision was reached, not what the config does now. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK
The MCCD table omitted mask_query_runner (now in the chain) and expected 80 mccd_preprocessing_runner outputs per exposure. mccd_preprocessing merges the exposure's per-CCD star catalogues into one train and one test catalogue (shapepipe_auxiliary_mccd.py), so the expected count is 2, not 80. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK
Two review fixes to test_workflow_config_module_sections_have_required_keys: - _required_config_keys was a top-level-statement heuristic that false-failed on an early-return has_option guard (a sibling statement, not a wrapping one) and on conditional expressions (ternaries), and never checked that a read's section argument was actually module_config_sec. Replaced with an AST visitor that tracks conditional nesting properly (if/for/while/try/with and ternaries), requires the section argument to be module_config_sec, and drops any key that a has_option check on the same key touches anywhere in the function, not just as a direct wrapper. Documented as heuristic, not exhaustive, in its docstring. - _module_config_sections returned a dict keyed by module name, so a module invoked twice in MODULE (e.g. vignetmaker_runner, as _RUN_1 and _RUN_2) only kept its last section, silently dropping the coverage for the other. Returns a list of (module, section) pairs, one per invocation, instead. Added a regression test that deletes vignetmaker_runner's _RUN_1 section from a real config and asserts the check now catches it. Re-verified against the pre-fix config_exp_mccd.ini (git show a1404f6) that the narrowed heuristic still independently catches both original defects (missing merge section, missing PSF key), and that it stays clean on the fixed file. 73 tests, 0.14s. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK
mccd_preprocessing_runner's expected count is 2 (train + test), not per-CCD, so contrasting mccd_fit_val_runner's count against "the per-CCD preprocessing outputs" no longer describes anything true. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK
cailmdaley
marked this pull request as ready for review
September 26, 2026 12:38
Contributor
Author
|
this seems an improvement over the previous state, and since MCCD will need a lot more work before it's as robust as psfex for the current state of the pipeline, i'm merging this without asking Martin to review. |
cailmdaley
added a commit
that referenced
this pull request
Sep 28, 2026
The merged fixes made six record lints false and broke four value assertions and one contract ref. Re-anchor the MCCD exposure chain to what it now reads (split image/weight/flag, mask_query before setools), the setools FWHM plot to 0.187, and CFIS EXP_PREFIX to a location-only ref (blank). Pin the MCCD completeness counts the rationale now names. Drop the resolved lints from the record, the @sc blocks and CONTRACTS; the IMAFLAGS_ISO export (#912), ngmix's 0.186 pixel scale against star selection's 0.187, and the ngmix noisefill/noise-window doc lints remain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cailmdaley
added a commit
that referenced
this pull request
Sep 28, 2026
Develop carries #894 (squashed, with Martin's run_config fixpoint / ${name} expansion), #918's MCCD exposure chain, and #859's fourth-moment PSF columns. Resolution: - run_config: develop's variable table (every top-level scalar, ${name}, fixpoint, `run` written back) with this branch's required `run:` and recursive unexpanded-$var refusal. - MCCD chain (config_exp_mccd.ini, completeness.py, exposure.smk comment): develop's #918 wiring; psf_model=mccd stays refused at parse time for persistence. - final_cat_merge replaces develop's merge_final_cats rule; config.yaml keeps `run:` unset (required). - #859 composes with the campaign products: MergeStarCatPSFEX's two-pass _COLUMNS and merge_star_cat.py's COLUMNS gain the six M4/RHO4 columns (22 in both), and cfis/final_cat.param lists the per-epoch HSM_M4_1/M4_2/RHO4_PSF_n slots. - make_cat: taken from the updated feat/make-cat-fixed-epoch-slots. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cailmdaley
added a commit
that referenced
this pull request
Sep 28, 2026
…he code by @sc tags (#875) * docs(astra): record the pipeline's scientific decisions in astra.yaml ShapePipe's scientific choices — detection thresholds, masking geometry, star selection, PSF model, ngmix priors and seeding, flag semantics, completeness floors — live in code and committed configs with their reasoning nowhere, or spread across PRs, papers and comments. astra.yaml gathers them: 50 decisions across eight sub-analyses, each with its rationale, the alternatives that were rejected and why, and a greppable anchor back to the code or config that implements it. universes/committed.yaml pins the option this branch selects for every one. The record is ASTRA (astra-tools; `uvx astra-tools@0.2.17 guide`), applied here at codebase level rather than to a single analysis. Conventions are stated in the file's header: anchors as `path::symbol` / `path#SECTION.KEY` and never line numbers, [HARDCODED] for a scientific value with no config exposure, [LINT] for a place where the record and the code — or the code and itself — disagree, [PENDING #NNN] for state not yet on develop. Authoring it surfaced nine such lints, two of which #873 fixes, and mapped ten places where the published Guinot+22 / Farrens+22 descriptions have drifted from the code since publication; 16 decisions carry verbatim paper quotes as prior insights. CLAUDE.md gains the standing instruction: a scientific change is not finished until the record is, amended in the same PR. The membership test is whether a different defensible choice would change which objects enter the shear catalogue, or the numbers attached to them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y2muA2sRojbxRNxU2SKQeP * test(astra): validate decision anchors and universe pins * test(astra): resolve Snakemake rule anchors; JSON report mode Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(astra): rewrite the decision record against develop - masking: describe healsparse queries (mask_query MASK_EXT on exposures, make_cat MASK_<band> on tiles) and the instrument flag image as the only pixel mask, replacing the deleted in-house mask generation - detection: tiles follow the MegaPipe (Gwyn) SExtractor parameters (#896); option ids no longer encode the retired values - shape_measurement: import defect_fill, blend_handling and epoch_masked_fraction_cut from the digital twin with their literature insights; defaults are what the committed code selects - prune to the membership test: drop psf_diagnostics, survey_geometry, the workflow-policy decisions and the root findings; split compound decisions; reserve excluded for considered-and-rejected; strip chronology - re-point anchors to the current configs; the anchor test passes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * docs(claude): point the scientific-decisions section at the anchor test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * docs(astra): correct seven rationale claims against the code - epoch_provenance: names keep their trailing p; EXP_PREFIX is a no-op [LINT] - fit_initialisation: only the PSF guesser takes the catalogue flux; an exception in Ngmix.process drops the object with no row - star_galaxy_classification: thresholds come from SM_STAR_THRESH / SM_GAL_THRESH, which the committed config does not set - psf_train_validation_split: seeded from the unit's file number - stamp_positioning: an out-of-image stamp centre raises - object_position_columns: tile stamps are cut at XWIN_IMAGE (COORD=PIX) - mark the PSFEx built-in SAMPLE_* behaviour and the 33-px trim unverified - record the galaxy prior reused for PSF fits and the silent epoch drops before the 1/3 cut; carry stale completeness, exposure.smk, _mode and pixel-scale comments as [LINT] Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * docs(astra): rephrase unverifiable claims; add Guinot+22 insights; sentinels and completeness precision Follow-up to the correction pass: claims the repo cannot check are stated as what the config assumes; five Guinot+22 prior insights with page-verified quotes replace bare paper citations; failure_sentinels says an NGMIX_N_EPOCH > 0 cut removes failed objects; per_unit_completeness counts only rules that run shapepipe_run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(contracts): validate @sc contracts against the decision record tests/helpers/contracts.py parses @sc/@cc contracts with sc-list's line grammar from Python docstrings, Snakemake comment blocks and CONTRACTS files under src/, workflow/ and scripts/. test_contracts.py fails on malformed lines, missing or duplicate ids, tag lines hidden in .py comments (invisible to sc-list), and decision: metas naming no decision in astra.yaml. A report-only test prints decisions no contract cites and contracts off the record's anchored symbols. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * docs(sc): first scientific contracts at the record's anchors Sixteen @sc contracts in the docstrings of declarations astra.yaml anchors, each citing its decision: star-selection mode and split seeding, SExtractor weight wiring and epoch membership bounds, CCD splitting and WCS source, epoch provenance, stamp rounding, the PSF acceptance gate, catalogue classification scope, never-fit sentinels, mask-column and mask-flag semantics, and per-unit completeness. Where the record carries a [LINT] at the declaration, the contract states the intended behaviour and names the lint. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * docs(astra): record the header saturation level; note PSF_ACCURACY - detection.saturation_level: SATUR_KEY SATURATE with no SATUR_LEVEL sets the FLAGS saturation bit that star selection rejects on; header presence on exposures and tiles is unverified here - psf_model_complexity: PSF_ACCURACY 0.01 with its anchor Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * test(contracts): utilities import boundary src/shapepipe/utilities/CONTRACTS declares utilities-do-not-import-modules (forbid: shapepipe.utilities.* -> shapepipe.modules.*). test_contracts.py reads the forbid rule from that file and resolves every import under src/shapepipe/utilities with ast, relative imports included. It holds today; loom's check-imports agrees. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * docs(astra): central_defect_veto; 4-fold symmetrisation for defect_fill - shape_measurement.central_defect_veto: default disabled (committed develop has no veto); radius_10px, implemented on feat/symmetrized-defect-fill, is the smallest radius with |m| < 1% - defect_fill: the recommended option is the 4-fold OR (symmetrized_4fold_noise); a single rot90 leaves coherent c2 of -0.006 to -0.012 for off-centre columns, 4-fold gives |c| < 2e-4 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * test(contracts): contracts on config keys via governs: Resolve semicolon-separated, directory-relative governed refs with the shared ASTRA anchor resolver. Reject empty refs and repeated metadata keys, and include config contracts in the report-only coverage check. Co-Authored-By: GPT-6 Astra <noreply@openai.com> * test(astra): assert config values in the anchor grammar Check active config values and static Python literals independently of anchor resolution. Normalize numeric and boolean spellings while retaining list shape and SETools comparison operators. Document the grammar and exercise it on PSF_NOISE. Co-Authored-By: GPT-6 Astra <noreply@openai.com> * docs(sc): contracts for the config-anchored decisions Keep canonical contracts beside the CFIS configs, with an inherited workflow contract beside the PSF selector. Cover the 17 previously uncovered decisions and the exposure pixel-scale/diagnostic coupling without changing scientific settings; retain the known config inconsistencies explicitly. Match contract coverage against anchor locators through the shared parser, including the value-assertion grammar added concurrently. Keep the tile-overlap config projection visible as a report-only anchor gap. Co-Authored-By: GPT-6 Astra <noreply@openai.com> * docs(astra): assert committed values on anchors Assert 130 values across 29 decisions without changing defaults or option ids. Cover coupled stamp sizes, detection, star cuts, PSF settings, and literal ngmix priors/metacal settings. Clarify that the CCD's 2048-index span is inclusive, whereas the committed cut excludes both endpoints. Co-Authored-By: GPT-6 Astra <noreply@openai.com> * docs(astra): defect fill, central veto and masked-fraction cut follow the measured design - defect_fill: noise on the unsymmetrized defect set stays default; interpolate (feat/defect-interpolation) describes the bounded-run fill with quarter-turn weight orbit; four-fold symmetrization is excluded on its measured m and c1; the model option is dropped - central_defect_veto: fixed radii (10 px noise, 7 px interpolated) on the defect mask only; size-scaled radius excluded; calibration and known limits stated; default stays disabled - epoch_masked_fraction_cut: the branch counts the raw defect set, with EPOCH_MASKED_FRACTION_CUT configurable; default stays 1/3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * test(astra): assert gate keys and absent keys; ambiguous subscript bindings; INI booleans per getboolean - Assert the booleans that make asserted values live: WEIGHT_IMAGE (weight map), MAKE_POST_PROCESS (CCD_SIZE), vignetmaker MASKING (STAMP_SIZE). - `= absent` asserts a config key has no active line; the record uses it for MASK_EXT in every star-selection mask block and SATUR_LEVEL in both .sex files. New contract psf-stars-vetoed-on-instrument-flags-only. - NAME[...] = / NAME.attr = in the binding's scope makes a value read of NAME ambiguous. - INI booleans follow ConfigParser.getboolean; Y/N only for .sex/.psfex. - A real-record mutation test covers each drift that previously passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * docs(astra): PSFEx compiled SAMPLE_* defaults, verified by psfex -dd psfex_candidate_vetting and psfex-vetting-is-not-fully-disabled state the PSFEx 3.21.1 compiled defaults (psfex -dd in the develop-runtime image) and assert each omitted SAMPLE_* key absent from default.psfex, so pinning one is visible to the record. Which cuts act with SAMPLE_AUTOSELECT N is marked as from the PSFEx source, not re-read here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * docs(astra): PSFEx vetting points at #919; compiled values are not assertable Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * docs(astra): retire lints resolved by #907, #909, #918 The merged fixes made six record lints false and broke four value assertions and one contract ref. Re-anchor the MCCD exposure chain to what it now reads (split image/weight/flag, mask_query before setools), the setools FWHM plot to 0.187, and CFIS EXP_PREFIX to a location-only ref (blank). Pin the MCCD completeness counts the rationale now names. Drop the resolved lints from the record, the @sc blocks and CONTRACTS; the IMAFLAGS_ISO export (#912), ngmix's 0.186 pixel scale against star selection's 0.187, and the ngmix noisefill/noise-window doc lints remain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(astra): tighten the record's prose Rationales no longer restate values their Anchor sentence asserts, and literature comparisons that a cited insight already carries become pointers. The header keeps the anchor grammar and markers; the value grammar's fine print moves to tests/helpers/astra_record.py, beside the parser that enforces it. Anchors, ids, options and evidence are unchanged (206 tests, astra validate). Corrected while tightening: the fit_initialisation default label (the galaxy guess takes its flux from a PSF-flux fit), and the blend_handling `none` description, which now claims only what Jarvis et al. 2016 support. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AkrwnKzU1wfUaHUPt3yYk3 * test: link science guardrails to ASTRA decisions * feat(decisions): add site-tag checker and Values resolver * fix(decisions): tighten tag discovery and section scopes * fix(decisions): resolve multiline INI values within sites * fix(decisions): scope Python selectors to tagged declarations * fix(decisions): ignore values outside Python site grammar * migrate record anchors to site tags * refactor: dissolve config contract sidecars into site tags * docs: document decision tags and Values checks * fix(decisions): tighten absence and duplicate checks * fix(config): narrow decision tag placements * fix(science): refine local decision-site contracts * docs: simplify and wrap Values assertions * docs(astra): record tile/exposure header evidence and the pixel-scale history A sampled tile and exposure carry SATURATE, so the SExtractor fallback level never applies; the exposure's FSCALE matches its PHOTZP against the tiles' zero-point 30. The fit_priors lint now names what #858 settled (the WCS is the source of truth) and what still overrides it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: run committed SExtractor configs with real tools * docs(astra): correct pipeline decision rationale * style(config): remove redundant migration blank lines * fix(decisions): match pipeline config semantics --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: GPT-6 Astra <noreply@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updates the MCCD exposure inputs and corrects its merge/plot configuration:
skipped mask_query entirely. Rewired as the MCCD analogue of the working
PSFEx chain: SExtractor reads the split flag image directly,
mask_query_runner sits between SExtractor and setools, setools reads
mask_query's sexcat_ext output.
paths resolve.
reads (its section is its own module name, MERGE_STARCAT_RUNNER) and
carried the wrong keys. Fixed to [MERGE_STARCAT_RUNNER] with PSF_MODEL.
preprocessing from 80 to 2 files: one training catalogue and one test
catalogue per exposure.
Adds workflow-config parsing and static runner-name/section-key checks.
Against a1404f6, the name check rejects mask_runner; the section/key check
rejects the missing merge section and PSF key. The key check is heuristic,
not exhaustive. Removes the unsupported rho-statistics plot description.
The committed workflow selects PSFEx. Validation was limited to config/unit
tests; MCCD execution and PSF outputs remain unverified.
Closes #917
🤖 Generated with Claude Code