Skip to content

geotiff tests: consolidate codec round-trip cluster (jpeg/jpeg2000/lerc/lz4) - #2450

Merged
brendancol merged 4 commits into
mainfrom
issue-2446
May 26, 2026
Merged

brendancol merged 4 commits into
mainfrom
issue-2446

Conversation

@brendancol

Copy link
Copy Markdown
Contributor

Closes #2446. PR A of cluster #2425; part of the long-tail epic #2424.

Summary

Folds 9 top-level codec files into a new unit/test_codec_roundtrip.py.
The sibling unit/test_compression.py keeps its PackBits-kernel-only
scope (its docstring already says round-trips live elsewhere), so the
new file is a sibling rather than an extension.

Files folded:

  • test_jpeg.py
  • test_jpeg2000.py
  • test_lerc.py
  • test_lerc_max_z_error.py
  • test_lerc_valid_mask.py
  • test_lz4.py
  • test_lz4_compression_level_2026_05_11.py
  • test_compression_docstring_1644.py
  • test_compression_level.py

Net file delta: 9 deleted, 1 added => -8 files.

What is preserved

  • 112 tests in scope, all green locally.
  • LERC max_z_error numerical tolerances pinned verbatim
    (budget + 1e-7 slack, bit-exact lossless via assert_array_equal).
  • LERC and JPEG 2000 optional-dep gating via pytest.importorskip /
    skip markers.
  • The to_geotiff(compression='jpeg') rejection contract.
  • The rasterio-only GDAL tiled JPEG read-back tests, still skip-gated.
  • The write_geotiff_gpu docstring drift guard from geotiff: write_geotiff_gpu compression docstring lists 4 codecs but accepts the full 9 #1644.

What changed shape

  • Parametrise dtype / tiled / level / mask matrices with descriptive
    ids (e.g. jpeg2000[uint16-tiled], lerc[float32-stripped],
    lz4-level[16], lerc-mask[uint16-sentinel]).
  • No issue numbers in test or filenames.
  • LERC max_z_error classes flattened into a single class.
  • LERC valid-mask round-trip variants parametrised over
    dtype × nodata × invalid_positions.

Out of scope (PR B, parallel)

PR B owns the photometric cluster and consolidates these into a
different unit/test_photometric.py:

  • test_miniswhite_nodata_1809.py
  • test_miniswhite_writer_roundtrip_1836.py
  • test_signed_miniswhite_rejected_2278.py
  • test_to_geotiff_allow_internal_only_jpeg_parity.py

GPU codec files (test_jpeg_gpu_*, test_lerc_valid_mask_gpu.py,
test_nvcomp_*, test_nvjpeg*) are scoped to the GPU cluster issue
#2438.

Audit trail

xrspatial/geotiff/tests/CLUSTER_AUDIT_CODEC_ROUNDTRIP.md maps every
old file::test to its new file::test_id. It lives on this branch
during review and is deleted in a final pre-merge commit so it does
not land on main.

Test plan

  • pytest xrspatial/geotiff/tests/unit/test_codec_roundtrip.py -v
    (112 passed)
  • pytest xrspatial/geotiff/tests/unit/ -q (247 passed)
  • pytest xrspatial/geotiff/tests/ -x -q (5778 passed, 68 skipped,
    6 xfailed)
  • find xrspatial/geotiff/tests -name 'test_*.py' | wc -l drops by
    8 (259 -> 251)

Fold the codec round-trip files into the new
unit/test_codec_roundtrip.py. unit/test_compression.py keeps its
PackBits-kernel-only scope as its docstring describes.

Files folded:
- test_jpeg.py
- test_jpeg2000.py
- test_lerc.py
- test_lerc_max_z_error.py
- test_lerc_valid_mask.py
- test_lz4.py
- test_lz4_compression_level_2026_05_11.py
- test_compression_docstring_1644.py
- test_compression_level.py

Net file delta: 9 deleted, 1 added => -8 files.

Test count unchanged (112 in scope); descriptive parametrise ids
(e.g. jpeg2000[uint16-tiled], lerc[float32-stripped],
lz4-level[16]) with no issue numbers in test or filenames.

LERC max_z_error numerical tolerances preserved verbatim. JPEG2000
and LERC sections use pytest.importorskip / skip markers for the
optional codec dependencies.

CLUSTER_AUDIT_CODEC_ROUNDTRIP.md maps old file::test to new
file::test_id; it lands on this branch and is deleted in a pre-merge
commit so it does not reach main.

Part of long-tail epic #2424 / cluster #2425 PR A.
@github-actions github-actions Bot added the performance PR touches performance-sensitive code label May 26, 2026

@brendancol brendancol left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review: geotiff tests: consolidate codec round-trip cluster

Tests-only change. No edits to xrspatial/geotiff/ source modules. 9 top-level files folded into a new unit/test_codec_roundtrip.py. 112 tests preserved (same baseline count).

Blockers

None.

Suggestions

  • xrspatial/geotiff/tests/unit/test_codec_roundtrip.py:1308-1315: the comment for _GPU_FALLBACK_CODECS now says "the JPEG round-trip is covered with appropriate uint8 RGB data elsewhere" without naming where. The original file pointed at test_gpu_writer_compression_modes_2026_05_11.py. Since GPU codec consolidation is owned by #2438, the most accurate phrasing here is "covered by the GPU codec cluster (issue #2438)". One-line wording tweak.
  • xrspatial/geotiff/tests/unit/test_codec_roundtrip.py:96-100 and :475-481: there are two helpers named _make_compressible and _make_compressible_large with slightly different shapes ((128,128) vs (512,512)) and noise levels. Keeping them separate is correct (the lz4 size-effect test uses a smaller surface than the zstd/deflate one for runtime reasons), but the naming is easy to misread. Renaming the module-level helper to _make_compressible_smooth_small or inlining it into TestLz4CompressionLevel would remove the ambiguity.
  • xrspatial/geotiff/tests/unit/test_codec_roundtrip.py:617-626: the LERC mask predicate closure captures invalid_positions via the default-arg pattern (def invalid_pred(a, positions=invalid_positions)). Good defensive choice. Worth a one-line comment explaining the bound-default pattern is intentional; without it a future refactor could collapse it to a free closure and reintroduce the late-binding bug.

Nits

  • xrspatial/geotiff/tests/unit/test_codec_roundtrip.py:53: from xrspatial.geotiff._compression import (...) is a long parenthesised import. The leading items COMPRESSION_JPEG, COMPRESSION_JPEG2000, COMPRESSION_LERC, COMPRESSION_LZ4 are constants and the rest are functions; isort already keeps them in one block but a blank-line break after the constants would make the import easier to read.
  • xrspatial/geotiff/tests/unit/test_codec_roundtrip.py:200-204: TestPublicToGeotiffJpegRejected is a class with a single test method. Both the original and the consolidated version use a class; nothing wrong with it, just notable.
  • xrspatial/geotiff/tests/unit/test_codec_roundtrip.py:1098, :1110: path = str(tmp_path / "lz4_bad.tif") is shared across the parametrised cases. pytest's tmp_path is unique per test invocation, so there is no real collision risk, but the project memory note recommends including unique tokens in tmp filenames. Either include the level in the filename (e.g. f"lz4_bad_{level}.tif") or leave a comment saying tmp_path already provides isolation.
  • CLUSTER_AUDIT_CODEC_ROUNDTRIP.md: scheduled for deletion on the pre-merge commit. The file header already says so; flagging here only so the deletion does not get forgotten.

What looks good

  • LERC max_z_error numerical tolerances preserved verbatim (budget + 1e-7 slack, bit-exact lossless via assert_array_equal).
  • pytest.importorskip / module-level skip markers kept for the optional codec libraries (glymur, lerc, lz4, rasterio).
  • Parametrise ids are descriptive and free of issue numbers (lerc[float32-stripped], lz4-level[16], lerc-mask[uint16-sentinel]).
  • The to_geotiff(compression='jpeg') rejection contract is still pinned (TestPublicToGeotiffJpegRejected).
  • The rasterio-only GDAL tiled JPEG class is still skip-gated at the class level, so the non-rasterio environment still exercises the codec-only tests above it.
  • The lerc_writer_with_mask monkeypatch fixture is preserved with the same predicate-callable indirection.
  • Audit file maps every old file::test to the new file::test_id.

Checklist

  • Tests-only; no source changes
  • All implemented backends produce consistent results
  • NaN handling is correct (LERC NaN-nodata case still pinned)
  • Edge cases are covered by tests (boundary levels, empty payloads, single pixel)
  • Optional-dependency skip markers preserved
  • No premature materialization or unnecessary copies
  • No benchmark needed (test consolidation only)
  • No README feature matrix change needed
  • Docstrings present on every consolidated class

…er tmp paths (#2446)

- _GPU_FALLBACK_CODECS comment now names cluster issue #2438 for the
  JPEG GPU coverage referral, replacing the dangling pointer to
  test_gpu_writer_compression_modes_2026_05_11.py.
- Module-level _make_compressible renamed to _make_compressible_smooth_small;
  docstring explains the size split vs TestCompressionLevelEffect._make_compressible_large.
- TestLercTiffRoundTripWithMask predicate closure: comment explaining
  why ``positions=invalid_positions`` is intentional (late-binding guard).
- Out-of-range path filenames now include the codec/level token so a
  future suite-wide xdist run cannot collide on tmp_path names.
- _compression imports grouped with section comments (constants /
  availability flags / functions) so the long import is easier to scan.

All 112 tests still pass; no behaviour change.

@brendancol brendancol left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review after nit pass

Disposition of original findings:

Suggestions

  • GPU codec referral in _GPU_FALLBACK_CODECS comment: fixed in 26657e5. Now names the GPU cluster issue #2438 instead of pointing at a file that the GPU cluster will rename.
  • _make_compressible / _make_compressible_large ambiguity: fixed in 26657e5. Module-level helper renamed to _make_compressible_smooth_small; docstring explains the size split.
  • Default-arg late-binding guard in invalid_pred: fixed in 26657e5. Comment added explaining the bind is intentional.

Nits

  • _compression import grouping: fixed in 26657e5. Section comments separate constants / availability flags / functions.
  • TestPublicToGeotiffJpegRejected single-method class: dismissed. The original module used a class wrapper here too; flipping to a free function is stylistic preference, not a fix.
  • Unique tmp_path filenames for parametrise: fixed in 26657e5. lz4_bad.tif, lz4_dask_bad.tif, and the generic bad.tif now include codec/level tokens.
  • CLUSTER_AUDIT_CODEC_ROUNDTRIP.md deletion reminder: tracked. Will be removed in the pre-merge commit on this branch.

Other notes

  • Merged origin/main (commit fe16afe) without conflicts. The merge only added an unrelated nodata-rejection test file at the top level.
  • 112 tests still pass post-merge and post-fixes.

No new findings. Ready for the pre-merge audit-file deletion commit.

Per the long-tail epic #2424 contract, the per-cluster audit file
lives on the branch during review and is removed in a pre-merge
commit so it does not land on main. Earlier #2390 PRs leaked their
audits; this PR enforces the deletion.
@brendancol
brendancol merged commit bdd2ea1 into main May 26, 2026
6 of 7 checks passed
brendancol added a commit that referenced this pull request May 26, 2026
The audit map served its purpose during review. Per the cluster
convention (see PR #2451 / #2450), the audit file is removed before
merge so it does not land on main.
brendancol added a commit that referenced this pull request May 26, 2026
* geotiff tests: consolidate predictor cluster (predictor 2 / 3)

Fold six top-level predictor test files into a new
`xrspatial/geotiff/tests/unit/test_predictor.py`, parametrised over
predictor in {2, 3} x dtype x endianness x layout (and multi-sample
for the chunky-band paths).

Files folded:
- test_predictor2_big_endian.py
- test_predictor2_int8.py
- test_predictor3_big_endian.py
- test_predictor3_int_dtype_1933.py
- test_predictor_fp_write_1313.py
- test_predictor_multisample.py

Net file delta: 6 deleted, 1 added => -5 files.

Behaviour preserved:
- predictor=2 round-trips across uint8/int8/uint16/int16/uint32/int32,
  big- and little-endian, stripped and tiled.
- predictor=3 round-trips for float32/float64, big- and little-endian,
  stripped and tiled (read + writer end-to-end).
- libtiff/GDAL multi-byte sample-level differencing interop.
- Multi-sample predictor=2 GPU parity and predictor=3 multi-sample
  TN3 decode (issues #1220, #1247, #1479).
- Predictor=3 + integer SampleFormat validator (#1933).
- normalize_predictor bool|int -> int mapping and rejection branches.
- The opt-in perf gate (predictor=3 within 2x of predictor=2).

Boundary: GPU predictor files
(`test_predictor2_big_endian_gpu_1517.py`,
`test_predictor3_int_dtype_gpu_1933.py`) stay in the GPU cluster
issue #2438. GPU regressions co-located with CPU tests in the old
files move with this consolidation so behaviour-grouped coverage
stays together (matches PR #2451).

Audit: `xrspatial/geotiff/tests/CLUSTER_AUDIT_PREDICTOR.md` maps every
old `file::test` (with parametrise expansion) to the new
`file::test_id`. Deleted on the pre-merge commit before this lands on
main.

Closes #2427.

* Address review nits: parametrise IDs + helper comment (#2427)

- Use `le` / `be` labels instead of `<` / `>` for endianness IDs.
- Flatten nested-bracket IDs (`pred2-libtiff[uint16]` ->
  `pred2-libtiff-uint16`, same for the other gpu / writer matrices)
  so pytest collection output reads as a single bracket per id.
- Move `imagecodecs_required` skip marker to the top of the file
  next to `_gpu_only` so all module-level gates live in one place.
- Annotate `_da_xy` about the trailing-axis indexing on 3D inputs
  so a future reader does not "fix" it back to the buggy
  pre-consolidation form.
- Reconcile the audit map and the +4 prose with the renamed IDs.

* Delete CLUSTER_AUDIT_PREDICTOR.md pre-merge (#2427)

The audit map served its purpose during review. Per the cluster
convention (see PR #2451 / #2450), the audit file is removed before
merge so it does not land on main.
@brendancol
brendancol deleted the issue-2446 branch June 25, 2026 13:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance PR touches performance-sensitive code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consolidate codec round-trip tests (epic #2424, cluster #2425 PR A)

1 participant