Skip to content

Fix Fletcher32 for empty buffers - #889

Open
RaphaelFakhri wants to merge 1 commit into
zarr-developers:mainfrom
RaphaelFakhri:fix-fletcher32-empty
Open

RaphaelFakhri wants to merge 1 commit into
zarr-developers:mainfrom
RaphaelFakhri:fix-fletcher32-empty

Conversation

@RaphaelFakhri

Copy link
Copy Markdown

Fletcher32.encode raises IndexError for an empty buffer, and Fletcher32.decode raises IndexError for a buffer that holds only the 4-byte checksum. The codec takes &mv[0] of an empty memoryview, which fails Cython's bounds check.

This change:

  • Returns a checksum of 0 from _fletcher32 for empty input, which matches the HDF5/netCDF reference implementation.
  • Skips the payload memcpy in encode and in the out branch of decode when the payload is empty.
  • Adds test_empty, which round-trips an empty buffer with and without out.

Closes #888

  • Unit tests and/or doctests in docstrings
  • Tests pass locally
  • Docstrings and API docs for any new/modified user-facing classes and functions
  • Changes documented in docs/release.md
  • Docs build locally
  • GitHub Actions CI passes
  • Test coverage to 100% (Codecov passes)

@codecov

codecov Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.71%. Comparing base (4afa628) to head (db932db).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #889   +/-   ##
=======================================
  Coverage   89.71%   89.71%           
=======================================
  Files          27       27           
  Lines         914      914           
=======================================
  Hits          820      820           
  Misses         94       94           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Duo-ee Duo-ee left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The empty-buffer fix looks right, but decode still has no size check on out, and that one is a memory-safety problem rather than a silent repeat:

  • out smaller than b_len - FOOTER_LENGTH → memcpy writes past the destination (heap corruption / crash).
  • out larger → the payload is copied, the tail of the caller's buffer is left untouched, and decode still reports success.
  • ensure_contiguous_ndarray(out) is np.asarray(out) for non-array buffers, so a bytearray out becomes read-only and &out_mv[0] points at read-only memory.

A ValueError before the copy covers all three cases:

payload_len = b_len - FOOTER_LENGTH
out_mv = ensure_contiguous_ndarray(out).view("uint8")
if out_mv.size != payload_len:
    raise ValueError(
        f"Output buffer has {out_mv.size} bytes, but the decoded payload has {payload_len} bytes."
    )
memcpy(&out_mv[0], b_ptr, payload_len)

Worth adding a case with an oversized out and a short one next to test_empty.

@RaphaelFakhri

Copy link
Copy Markdown
Author

@Duo-ee The undersized out overflow is real, but it's on main already and has nothing to do with empty buffers, so I'd rather keep this PR to #888. It fits with your jenkins out size PR, and a matching fix for Fletcher32 (and any other codec that copies into out by hand) can go in as its own PR.

Two notes on the details. A bytearray out is fine: np.asarray(bytearray(...)) is writable, and out_mv is a non-const typed memoryview, so a read-only buffer raises ValueError on assignment instead of being written to. And the snippet above takes &out_mv[0] unconditionally, which brings back the IndexError for an empty payload that this PR fixes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fletcher32 raises IndexError on empty buffers

2 participants