Skip to content

move numcodecs codecs to separate repos #3783

Open

Description

@d-v-b

I propose that we migrate all of the codecs currently defined in the codecs.numcodecs module to stand-alone repositories under the zarr-developers organization. Those repos can themselves depend on numcodecs, or not. We can then decide if these codecs are worth including as optional dependencies for zarr python.

I think we should also look at removing the numcodecs dependency for our core codecs -- gzip is a python built-in, and there are blosc wheels iirc. zstandard is probably available as a backported python library? We should explore this.

Activity

  1. maxrjones commented on Mar 17, 2026

    @maxrjones
    Member

    this module - https://github.com/zarr-developers/zarr-python/blob/main/src/zarr/codecs/numcodecs/_codecs.py? There's only ~300 lines of code there, so I'm not really getting the rationale.

  2. d-v-b commented on Mar 18, 2026

    @d-v-b
    ContributorAuthor

    those codecs are currently thin wrappers over numcodecs, and those wrappers are effectively bugged, because they don't do any input validation. When we do full, proper zarr v3 support for those codecs, like #3774, the code size might grow considerably.

    I'm also motivated to cut down on our numcodecs dependency as much as we can. These both argue for potentially externalizing codecs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    design discussionDiscussion about the design of the library

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions