Repository navigation
Tile detection: adopt MegaPipe (Gwyn) SExtractor parameters - #896
Merged
Merged
Conversation
Align default_tile.sex with the CADC megapipe.sex Stephen Gwyn uses for the UNIONS/CFIS tile catalogues, so our tile detection (data and image sims) matches the catalogue we are adopting: DETECT_THRESH/ANALYSIS_THRESH 1.0, DETECT_MINAREA 3, gauss_3.0_7x7 filter, DEBLEND_MINCONT 0.002, AUTO background with BACK_SIZE 512 / FILTERSIZE 9, LOCAL BACKPHOTO. With the previous parameters SExtractor triggered spuriously on about 10% of grid-placed Sersic galaxies in the sims. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…R_NAME) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
martinkilbinger
approved these changes
Sep 22, 2026
cailmdaley
added a commit
that referenced
this pull request
Sep 26, 2026
- masking: describe healsparse queries (mask_query MASK_EXT on exposures, make_cat MASK_<band> on tiles) and the instrument flag image as the only pixel mask, replacing the deleted in-house mask generation - detection: tiles follow the MegaPipe (Gwyn) SExtractor parameters (#896); option ids no longer encode the retired values - shape_measurement: import defect_fill, blend_handling and epoch_masked_fraction_cut from the digital twin with their literature insights; defaults are what the committed code selects - prune to the membership test: drop psf_diagnostics, survey_geometry, the workflow-policy decisions and the root findings; split compound decisions; reserve excluded for considered-and-rejected; strip chronology - re-point anchors to the current configs; the anchor test passes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK
cailmdaley
added a commit
that referenced
this pull request
Sep 28, 2026
config_tile_Sg.ini takes DOT_CONV_FILE from config_tile_Sx.ini (#896), so the uberseg footprints are the ones SExtractor mode would draw. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uemfjv9ybCwtKtprksZbVY
cailmdaley
added a commit
that referenced
this pull request
Sep 28, 2026
…he code by @sc tags (#875) * docs(astra): record the pipeline's scientific decisions in astra.yaml ShapePipe's scientific choices — detection thresholds, masking geometry, star selection, PSF model, ngmix priors and seeding, flag semantics, completeness floors — live in code and committed configs with their reasoning nowhere, or spread across PRs, papers and comments. astra.yaml gathers them: 50 decisions across eight sub-analyses, each with its rationale, the alternatives that were rejected and why, and a greppable anchor back to the code or config that implements it. universes/committed.yaml pins the option this branch selects for every one. The record is ASTRA (astra-tools; `uvx astra-tools@0.2.17 guide`), applied here at codebase level rather than to a single analysis. Conventions are stated in the file's header: anchors as `path::symbol` / `path#SECTION.KEY` and never line numbers, [HARDCODED] for a scientific value with no config exposure, [LINT] for a place where the record and the code — or the code and itself — disagree, [PENDING #NNN] for state not yet on develop. Authoring it surfaced nine such lints, two of which #873 fixes, and mapped ten places where the published Guinot+22 / Farrens+22 descriptions have drifted from the code since publication; 16 decisions carry verbatim paper quotes as prior insights. CLAUDE.md gains the standing instruction: a scientific change is not finished until the record is, amended in the same PR. The membership test is whether a different defensible choice would change which objects enter the shear catalogue, or the numbers attached to them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y2muA2sRojbxRNxU2SKQeP * test(astra): validate decision anchors and universe pins * test(astra): resolve Snakemake rule anchors; JSON report mode Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(astra): rewrite the decision record against develop - masking: describe healsparse queries (mask_query MASK_EXT on exposures, make_cat MASK_<band> on tiles) and the instrument flag image as the only pixel mask, replacing the deleted in-house mask generation - detection: tiles follow the MegaPipe (Gwyn) SExtractor parameters (#896); option ids no longer encode the retired values - shape_measurement: import defect_fill, blend_handling and epoch_masked_fraction_cut from the digital twin with their literature insights; defaults are what the committed code selects - prune to the membership test: drop psf_diagnostics, survey_geometry, the workflow-policy decisions and the root findings; split compound decisions; reserve excluded for considered-and-rejected; strip chronology - re-point anchors to the current configs; the anchor test passes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * docs(claude): point the scientific-decisions section at the anchor test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * docs(astra): correct seven rationale claims against the code - epoch_provenance: names keep their trailing p; EXP_PREFIX is a no-op [LINT] - fit_initialisation: only the PSF guesser takes the catalogue flux; an exception in Ngmix.process drops the object with no row - star_galaxy_classification: thresholds come from SM_STAR_THRESH / SM_GAL_THRESH, which the committed config does not set - psf_train_validation_split: seeded from the unit's file number - stamp_positioning: an out-of-image stamp centre raises - object_position_columns: tile stamps are cut at XWIN_IMAGE (COORD=PIX) - mark the PSFEx built-in SAMPLE_* behaviour and the 33-px trim unverified - record the galaxy prior reused for PSF fits and the silent epoch drops before the 1/3 cut; carry stale completeness, exposure.smk, _mode and pixel-scale comments as [LINT] Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * docs(astra): rephrase unverifiable claims; add Guinot+22 insights; sentinels and completeness precision Follow-up to the correction pass: claims the repo cannot check are stated as what the config assumes; five Guinot+22 prior insights with page-verified quotes replace bare paper citations; failure_sentinels says an NGMIX_N_EPOCH > 0 cut removes failed objects; per_unit_completeness counts only rules that run shapepipe_run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(contracts): validate @sc contracts against the decision record tests/helpers/contracts.py parses @sc/@cc contracts with sc-list's line grammar from Python docstrings, Snakemake comment blocks and CONTRACTS files under src/, workflow/ and scripts/. test_contracts.py fails on malformed lines, missing or duplicate ids, tag lines hidden in .py comments (invisible to sc-list), and decision: metas naming no decision in astra.yaml. A report-only test prints decisions no contract cites and contracts off the record's anchored symbols. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * docs(sc): first scientific contracts at the record's anchors Sixteen @sc contracts in the docstrings of declarations astra.yaml anchors, each citing its decision: star-selection mode and split seeding, SExtractor weight wiring and epoch membership bounds, CCD splitting and WCS source, epoch provenance, stamp rounding, the PSF acceptance gate, catalogue classification scope, never-fit sentinels, mask-column and mask-flag semantics, and per-unit completeness. Where the record carries a [LINT] at the declaration, the contract states the intended behaviour and names the lint. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * docs(astra): record the header saturation level; note PSF_ACCURACY - detection.saturation_level: SATUR_KEY SATURATE with no SATUR_LEVEL sets the FLAGS saturation bit that star selection rejects on; header presence on exposures and tiles is unverified here - psf_model_complexity: PSF_ACCURACY 0.01 with its anchor Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * test(contracts): utilities import boundary src/shapepipe/utilities/CONTRACTS declares utilities-do-not-import-modules (forbid: shapepipe.utilities.* -> shapepipe.modules.*). test_contracts.py reads the forbid rule from that file and resolves every import under src/shapepipe/utilities with ast, relative imports included. It holds today; loom's check-imports agrees. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * docs(astra): central_defect_veto; 4-fold symmetrisation for defect_fill - shape_measurement.central_defect_veto: default disabled (committed develop has no veto); radius_10px, implemented on feat/symmetrized-defect-fill, is the smallest radius with |m| < 1% - defect_fill: the recommended option is the 4-fold OR (symmetrized_4fold_noise); a single rot90 leaves coherent c2 of -0.006 to -0.012 for off-centre columns, 4-fold gives |c| < 2e-4 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * test(contracts): contracts on config keys via governs: Resolve semicolon-separated, directory-relative governed refs with the shared ASTRA anchor resolver. Reject empty refs and repeated metadata keys, and include config contracts in the report-only coverage check. Co-Authored-By: GPT-6 Astra <noreply@openai.com> * test(astra): assert config values in the anchor grammar Check active config values and static Python literals independently of anchor resolution. Normalize numeric and boolean spellings while retaining list shape and SETools comparison operators. Document the grammar and exercise it on PSF_NOISE. Co-Authored-By: GPT-6 Astra <noreply@openai.com> * docs(sc): contracts for the config-anchored decisions Keep canonical contracts beside the CFIS configs, with an inherited workflow contract beside the PSF selector. Cover the 17 previously uncovered decisions and the exposure pixel-scale/diagnostic coupling without changing scientific settings; retain the known config inconsistencies explicitly. Match contract coverage against anchor locators through the shared parser, including the value-assertion grammar added concurrently. Keep the tile-overlap config projection visible as a report-only anchor gap. Co-Authored-By: GPT-6 Astra <noreply@openai.com> * docs(astra): assert committed values on anchors Assert 130 values across 29 decisions without changing defaults or option ids. Cover coupled stamp sizes, detection, star cuts, PSF settings, and literal ngmix priors/metacal settings. Clarify that the CCD's 2048-index span is inclusive, whereas the committed cut excludes both endpoints. Co-Authored-By: GPT-6 Astra <noreply@openai.com> * docs(astra): defect fill, central veto and masked-fraction cut follow the measured design - defect_fill: noise on the unsymmetrized defect set stays default; interpolate (feat/defect-interpolation) describes the bounded-run fill with quarter-turn weight orbit; four-fold symmetrization is excluded on its measured m and c1; the model option is dropped - central_defect_veto: fixed radii (10 px noise, 7 px interpolated) on the defect mask only; size-scaled radius excluded; calibration and known limits stated; default stays disabled - epoch_masked_fraction_cut: the branch counts the raw defect set, with EPOCH_MASKED_FRACTION_CUT configurable; default stays 1/3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * test(astra): assert gate keys and absent keys; ambiguous subscript bindings; INI booleans per getboolean - Assert the booleans that make asserted values live: WEIGHT_IMAGE (weight map), MAKE_POST_PROCESS (CCD_SIZE), vignetmaker MASKING (STAMP_SIZE). - `= absent` asserts a config key has no active line; the record uses it for MASK_EXT in every star-selection mask block and SATUR_LEVEL in both .sex files. New contract psf-stars-vetoed-on-instrument-flags-only. - NAME[...] = / NAME.attr = in the binding's scope makes a value read of NAME ambiguous. - INI booleans follow ConfigParser.getboolean; Y/N only for .sex/.psfex. - A real-record mutation test covers each drift that previously passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * docs(astra): PSFEx compiled SAMPLE_* defaults, verified by psfex -dd psfex_candidate_vetting and psfex-vetting-is-not-fully-disabled state the PSFEx 3.21.1 compiled defaults (psfex -dd in the develop-runtime image) and assert each omitted SAMPLE_* key absent from default.psfex, so pinning one is visible to the record. Which cuts act with SAMPLE_AUTOSELECT N is marked as from the PSFEx source, not re-read here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * docs(astra): PSFEx vetting points at #919; compiled values are not assertable Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bvNTrAmZxcfb1ee83ApPK * docs(astra): retire lints resolved by #907, #909, #918 The merged fixes made six record lints false and broke four value assertions and one contract ref. Re-anchor the MCCD exposure chain to what it now reads (split image/weight/flag, mask_query before setools), the setools FWHM plot to 0.187, and CFIS EXP_PREFIX to a location-only ref (blank). Pin the MCCD completeness counts the rationale now names. Drop the resolved lints from the record, the @sc blocks and CONTRACTS; the IMAFLAGS_ISO export (#912), ngmix's 0.186 pixel scale against star selection's 0.187, and the ngmix noisefill/noise-window doc lints remain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(astra): tighten the record's prose Rationales no longer restate values their Anchor sentence asserts, and literature comparisons that a cited insight already carries become pointers. The header keeps the anchor grammar and markers; the value grammar's fine print moves to tests/helpers/astra_record.py, beside the parser that enforces it. Anchors, ids, options and evidence are unchanged (206 tests, astra validate). Corrected while tightening: the fit_initialisation default label (the galaxy guess takes its flux from a PSF-flux fit), and the blend_handling `none` description, which now claims only what Jarvis et al. 2016 support. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AkrwnKzU1wfUaHUPt3yYk3 * test: link science guardrails to ASTRA decisions * feat(decisions): add site-tag checker and Values resolver * fix(decisions): tighten tag discovery and section scopes * fix(decisions): resolve multiline INI values within sites * fix(decisions): scope Python selectors to tagged declarations * fix(decisions): ignore values outside Python site grammar * migrate record anchors to site tags * refactor: dissolve config contract sidecars into site tags * docs: document decision tags and Values checks * fix(decisions): tighten absence and duplicate checks * fix(config): narrow decision tag placements * fix(science): refine local decision-site contracts * docs: simplify and wrap Values assertions * docs(astra): record tile/exposure header evidence and the pixel-scale history A sampled tile and exposure carry SATURATE, so the SExtractor fallback level never applies; the exposure's FSCALE matches its PHOTZP against the tiles' zero-point 30. The fit_priors lint now names what #858 settled (the WCS is the source of truth) and what still overrides it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: run committed SExtractor configs with real tools * docs(astra): correct pipeline decision rationale * style(config): remove redundant migration blank lines * fix(decisions): match pipeline config semantics --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: GPT-6 Astra <noreply@openai.com>
cailmdaley
added a commit
that referenced
this pull request
Sep 29, 2026
) * Add the UNIONS tile-catalogue inis to the workflow config chain Move the two v2.0-era configs (fetch Steven Gwyn's per-tile .cat, convert it with read_ext_sexcat_runner) into workflow/config/cfis and adapt them to the workflow's conventions: fixed run names, NUMBER_LIST from SP_UNIT_NUM, explicit chained INPUT_DIRs, and the catalogue source and retrieve mode taken from the environment the rules export. The converter writes into run_sp_tile_Sx so that the downstream tile chain reads one path in either detection mode. get_images_runner expands RETRIEVE so it can be set per run. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Add tile_detection: unions_catalogue to the workflow A run config key, validated at parse time, that picks where the tile's galaxy sample comes from. `sextractor` is the existing SExtractor rule, byte for byte. `unions_catalogue` replaces it with two rules: tile_get_catalogue fetches the UNIONS per-tile .cat from inputs.catalogues (a local mirror or a vos: URL), and tile_detect converts it with read_ext_sexcat_runner into run_sp_tile_Sx, linking the module output as sextractor_runner so that tile_vignets, ngmix and make_cat read the one path they always have; the manifest is tile_detect.json in both modes. completeness.py checks tile_detect per mode through SP_TILE_DETECTION, exported on that rule alone so a SExtractor run's prologue is unchanged; run_report lists the fetch stage only for catalogue runs. The catalogue's per-tile unique object ID reaches the final catalogue as TILE_UNIQUE_ID through make_cat's copy of every sexcat column. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Test the UNIONS catalogue conversion and its workflow wiring read_ext_sexcat on a synthetic ASCII catalogue and image: LDAC layout, the tile header in LDAC_IMHEAD, the SExtractor aliases, zero-padded VIGNET stamps cut at the right pixels, TILE_UNIQUE_ID = tile_id * 1e6 + NUMBER, and that column surviving make_cat.save_sextractor_data into the final catalogue. On the workflow side: the two detection inis share the stage's run dir, the fetch ini feeds the converter, the tile_detect count table is selected per mode with sextractor as the unset default, and run_report lists the fetch stage only for catalogue runs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Fix TILE_UNIQUE_ID merge, NUMBER contiguity, and FITS int64 dtype test final_cat.param lists TILE_UNIQUE_ID again; merge_final_cat.py and create_final_cat.py now drop a requested column that is absent from a given catalogue (logging once) instead of failing, so sextractor-mode runs (without the column) and unions_catalogue-mode runs (with it) both merge cleanly. read_ext_sexcat's NUMBER passthrough is not guaranteed contiguous or in order in the external catalogue, which ngmix_range requires; TILE_UNIQUE_ID already preserves the original NUMBER, so the converter now renumbers NUMBER to 1..n_obj in output row order and says so in its docstring. test_read_ext_sexcat's TILE_UNIQUE_ID dtype check compared against np.int64 directly; a FITS 'K' column reads back big-endian, so the check now compares dtype.kind and itemsize instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Build the UberSeg segmentation map for the UNIONS catalogue path UberSeg identifies a stamp's central object BY LABEL: uberseg_weight keeps the pixels whose nearest segmentation footprint carries the object's own catalogue NUMBER. The UNIONS per-tile catalogue carries no segmentation map, and a SExtractor run's labels are its own detection numbers, so the two numberings have to be bridged before ngmix can mask a neighbour. `blend_handling:` (noisefill | uberseg) joins `tile_detection:` in the run config. The pair unions_catalogue + uberseg adds tile_segmentation: one SExtractor run on the tile image whose only product is the SEGMENTATION check image (config_tile_Sg.ini), followed by seg_relabel.py, which gives each catalogue object the footprint its position falls in, relabels it with that object's NUMBER, marks every unclaimed footprint a neighbour, and paints a small disc for an object that falls on sky or on a footprint already claimed. The result lands beside the sexcat, where vignetmaker's segmentation run looks whichever mode wrote the catalogue. Steven's catalogue stays the sample; SExtractor runs once, for the one product the catalogue cannot give. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Default tile_detection to the UNIONS catalogue The per-tile catalogue is the sample; SExtractor detection is the explicit alternative. `inputs.catalogues` travels with it -- the Snakefile refuses unions_catalogue without one -- and is committed as TBD for the run to set. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Build TILE_UNIQUE_ID once, in make_cat, from one helper cfis.get_tile_unique_id defines the survey-wide object ID, tile_id * 10**6 + NUMBER with tile_id = RRR * 1000 + DDD, and raises when NUMBER (or tile_id) falls outside [0, 10**6). cfis.get_tile_id parses the tile, split_tile_unique_id inverts the encoding, and get_tile_number now refuses tile components with more than three digits. make_cat writes TILE_UNIQUE_ID next to TILE_ID for every final catalogue, whatever produced the detection catalogue. read_ext_sexcat is a plain format converter again: it copies NUMBER unchanged and adds no ID. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uemfjv9ybCwtKtprksZbVY * Select ngmix chunks by catalogue row, not NUMBER value ID_OBJ_MIN/MAX are now 1-based closed row ranges (ngmix.chunk_rows), so the partition ngmix_range.py writes covers every object once however NUMBER is ordered or spaced. ngmix_range emits NGMIX_ROW_MIN/MAX, and checks that the EPOCH extensions are row-aligned instead of requiring NUMBER = 1..N. NUMBER stays the object's identity: vignet/PSF store keys, the uberseg label check and position seeding are unchanged. fake_psf keys its PSF store by NUMBER rather than row + 1. For SExtractor catalogues (NUMBER = row) chunk contents are unchanged, but ngmix_range.py's hash changes, so a resume across this commit re-measures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uemfjv9ybCwtKtprksZbVY * Segmentation run uses the tile detection's MegaPipe filter config_tile_Sg.ini takes DOT_CONV_FILE from config_tile_Sx.ini (#896), so the uberseg footprints are the ones SExtractor mode would draw. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uemfjv9ybCwtKtprksZbVY * Move UberSeg segmentation to its own branch The tile_segmentation rule, seg_relabel.py, config_tile_Sg.ini and the blend_handling setting move to feat/uberseg-seg-map, which feeds the consolidated blend/defect PR. This branch keeps the Gwyn catalogue path, TILE_UNIQUE_ID and row-based ngmix chunking; the catalogue path has no segmentation map, so uberseg needs the SExtractor mode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Default tile_detection and psf_model per input type, not per machine Both follow from the kind of input (the UNIONS catalogue and PSFEx for data, SExtractor and the true simulation PSF for image sims), so they move out of the machine entries into a top-level input_types: table. run_config layers config.yaml < input_types[input_type] < machines[machine][input_type] < run config, and expands $variables in every resolved key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record tile_detection in astra.yaml Real data now takes its tile sample from Gwyn's UNIONS catalogue. The detection sub-analysis gains a tile_detection decision (committed: unions_catalogue), pinned to the input_types table, and notes that its SExtractor decisions govern only the sextractor path. The converter's multi-epoch post-processing is tagged under epoch_membership_ccd_bounds, which applies to both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Point inputs.catalogues at vos:cfis/tiles_DR6 on both machines Stephen Gwyn's per-tile catalogues live at vos:cfis/tiles_DR6 (the path develop's config_tile_Git_cat_vos.ini already uses). tile_get_catalogue fetches them with vcp inside the job, using ~/.ssl/cadcproxy.pem; that works from a candide compute node in the runtime container. The unset/placeholder check moves from the Snakefile into run_config.catalogue_source, so the parse-time refusal is unit-tested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Leave the retired standalone merge_final_cat.py to develop The workflow merges final catalogues through final_cat_merge; the standalone script is retired, so this branch no longer patches it or tests filter_available_columns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DksuyF9YrZAwHQeAMXsxp4 * final_cat_merge: request only the columns the tile's detection mode can carry Under tile_detection: unions_catalogue the detection columns are the UNIONS DR6 catalogue's, which lacks nine SExtractor columns final_cat.param names (MAG_WIN, MAGERR_WIN, FLUX_AUTO, FLUXERR_AUTO, FLUX_APER, FLUXERR_APER, SNR_WIN, FWHM_IMAGE, FWHM_WORLD), so every catalogue-mode merge raised. merge_final_cat.py now takes --tile-detection and subtracts exactly SEXTRACTOR_ONLY_COLUMNS in that mode; the merge stays strict on the rest. tests/module/test_final_cat_columns.py runs the converter on the DR6 header and asserts that list is exactly the requested SExtractor columns DR6 lacks. The params pin moves for final_cat_merge only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BeQNTBcjTqu6TPktxysovQ * Mask neighbours in catalogue-mode VIGNETs from the DR6 segmentation map tile_detection: unions_catalogue cut VIGNET straight from the tile image, so ngmix (which masks where the tile VIGNET is -1e30) masked no neighbours: in smk-g10 NGMIX_MAG_NOSHEAR ran >1 mag bright of MAG_AUTO for 6.1% of objects, against 2.7% with SExtractor. tile_get_catalogue now also fetches CFIS.<tile>.r.seg.fits.fz. The converter relabels it to the catalogue's NUMBER by the footprint under each object's centre pixel (relabel_seg: unclaimed footprints -1, a small disc for objects without a footprint), writes it beside the sexcat as seg-<tile>.fits for UberSeg, and sets VIGNET pixels on other objects' footprints to -1e30. Off-image VIGNET pixels are -1e30 too, as SExtractor writes them. Records detection.catalogue_neighbour_marking. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BeQNTBcjTqu6TPktxysovQ * completeness: expect the seg-stamp vignetmaker run under blend_handling: uberseg tile_vignets gains a third vignetmaker run, the segmentation stamps ngmix's SEG_VIGNET_PATH reads, when SP_BLEND_HANDLING=uberseg. Noise-fill runs export nothing and expect exactly what they did before. Landing this here keeps SCRIPT_HASH(completeness.py) at one campaign boundary with #897. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Revert "completeness: expect the seg-stamp vignetmaker run under blend_handling: uberseg" UberSeg's seg stamps will ride the tile catalogue as a SEG_VIGNET column, so there is no extra vignetmaker run to expect. completeness.py is again byte-identical to 8da837b, keeping SCRIPT_HASH at the value smk-g11 runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * config.yaml: plainer input_types comment (review suggestion) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop the relabelled seg-map file output Its only consumer was an UberSeg seg run that no longer exists; the relabelled map is still used in memory for VIGNET neighbour marking. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * completeness: the catalogue converter writes one file The relabelled seg map is no longer written, so expect=2 failed every catalogue-mode tile_detect; a runner-driven test now ties the count to the converter's output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Aligns
workflow/config/cfis/default_tile.sexwith themegapipe.sexStephen Gwyn uses for the CFIS tile catalogues (linked from the MegaPipe processing page), so our tile detection matches the catalogue we are adopting (#832). Data and image sims share this file.config_tile_Sx.inipoints at it, since the runner passes-FILTER_NAME)With the previous values SExtractor triggered spuriously on about 10% of grid-placed Sérsic galaxies in the image sims (Fabian); we expect that rate to drop on
grid_3with these, and data-run detection changes accordingly on the next campaign. Not a gate for this PR — see the rapid-iteration note added to CLAUDE.md.🤖 Generated with Claude Code