Repository navigation

Lazy datafusion registration via custom table provider - #100

Merged

alxmrs merged 77 commits into

mainfrom

reader

Feb 1, 2026

Merged

Lazy datafusion registration via custom table provider#100
alxmrs merged 77 commits into
mainfrom
reader

Commits on Jan 15, 2026
Commits on Jan 16, 2026
- Fix DataFusion pushdown filters by using factory pattern for streams
  claude
  committed
  efb23fd
- Simplify factory to return XarrayRecordBatchReader directly
  claude
  committed
  d92c334
Commits on Jan 18, 2026
- Apply pre-commit (pyink).
  alxmrs
  committed
  74518b7
- Pulling rust build items from other branch.
  alxmrs
  committed
  bd2e20a
- rust fmt
  alxmrs
  committed
  2c4d3db
- Add rust fmt to pre-commit.
  alxmrs
  committed
  e6c58e4
- Don't use nightly fmt.
  alxmrs
  committed
  75ca76e
- Updated rust ci.
  alxmrs
  committed
  93dcc51
- rm wrong classifiers.
  alxmrs
  committed
  f44906f
- New release process using uv (recommended by maturin).
  alxmrs
  committed
  707eeeb
- Probably also need to install the rust toolchains to build.
  alxmrs
  committed
  2c288fe
- Updating release process to work with native code on pypi.
  alxmrs
  committed
  560c1d6
- remove unstable features.
  alxmrs
  committed
  449c036
- Better maturin CI
  alxmrs
  committed
  1206eb0
- cross compile with zig.
  alxmrs
  committed
  fe48ecd
- we actually don't need this.
  alxmrs
  committed
  5ff0db4
- Adding CI to test that it builds on other platforms.
  alxmrs
  committed
  cbe2d3e
- reduce combinations.
  alxmrs
  committed
  deecee3
- rm comment.
  alxmrs
  committed
  8a8ae22
- Reduce combinations.
  alxmrs
  committed
  3dfb909
- specify python version.
  alxmrs
  committed
  1e20edd
- Better name
  alxmrs
  committed
  55f182d
- Better python version
  alxmrs
  committed
  9169d3b
- platform target in test build CI name.
  alxmrs
  committed
  93c88ed
- better name.
  alxmrs
  committed
  202afc9
- enabling uv cache in CI.
  alxmrs
  committed
  5bd4dec
- verbose sync for CI
  alxmrs
  committed
  26361bc
- dropping python 3.9 support (RIP).
  alxmrs
  committed
  7584589
- dropping python 3.9 support (RIP).
  alxmrs
  committed
  b24f1ea
- add rust cache to CI
  alxmrs
  committed
  7aa57b7
Commits on Jan 19, 2026
- Need example data to create tests
  alxmrs
  committed
  82f63e7
- Make rust cache consistent.
  alxmrs
  committed
  346297f
- rn int tests from old branch.
  alxmrs
  committed
  493cc28
- sccache enabled for all builds with rust.
  alxmrs
  committed
  45fc728
Commits on Jan 24, 2026
- non-deprecated API.
  alxmrs
  committed
  bfd5713
- Good catch from @maximedion2 in the native sources.
  alxmrs
  committed
  c779d72
- register_table_provider is not deprecated in this version of datafusion. Looks like I need it.
  alxmrs
  committed
  2ba27a3
- upgrading datafusion version.
  alxmrs
  committed
  6e19b40
- Cleaning up the API.
  alxmrs
  committed
  ec76009
- Fixed issues related to datafusion upgrade.
  alxmrs
  committed
  2ea1135
- fix pre-commit for rust. should be a fmt, not a check.
  alxmrs
  committed
  a2fd17a
- rust lint.
  alxmrs
  committed
  e2fa418
- Using newer python api given the datafusion upgrade.
  alxmrs
  committed
  efc1dd2
- pre-commit: filter rust files.
  alxmrs
  committed
  c9a1be2
Commits on Jan 25, 2026
- Not using deprecated API in tests.
  alxmrs
  committed
  2ffcdf9
- Eliminated a test.
  alxmrs
  committed
  7ac7a4d
- Attempt to make the build faster.
  alxmrs
  committed
  12ccd66
- fix build
  alxmrs
  committed
  ddfe41f
- revert install in ci
  alxmrs
  committed
  45acd3f
- we still want verbose
  alxmrs
  committed
  862a512
- trying import hook. thanks
  alxmrs
  committed
  600cbf0
- better system (inspired by datafusion-python).
  alxmrs
  committed
  a4c1230
- new ideas to make the build faster from datafusion-python.
  alxmrs
  committed
  bd06611
- adding cargo cache to ci build.
  alxmrs
  committed
  ac3fcf8
- env aware cargo cache.
  alxmrs
  committed
  723de14
- better build to take advantage of caches.
  alxmrs
  committed
  de549eb
- rm --uv flag for build.
  alxmrs
  committed
  ebb2d38
- more accurate year.
  alxmrs
  committed
  0be5c38
- fix minor typo in readme.
  alxmrs
  committed
  1d0dc2b
- Upgrade `read_xarray` instead of deprecating. Use new reader functionality.
  alxmrs
  committed
  8f8abc2
- error prop
  alxmrs
  committed
  31e5da9
- True streaming implementation.
  alxmrs
  committed
  1dccd88
- attempt to fix rust pre-commit.
  alxmrs
  committed
  87d2597
- rust fmt
  alxmrs
  committed
  5739e1d
- revert
  alxmrs
  committed
  3e5ea89
- experiment: GIL-aware iterator reaching straddling rust and python.
  alxmrs
  committed
  b5cbe40
- Better streaming tests (WIP)
  alxmrs
  committed
  f1d19d7
- Fix: use cstreams via record batches
  alxmrs
  committed
  3dd95af
Commits on Jan 26, 2026
- Fixes to get most tests passing.
  alxmrs
  committed
  41c6138
- Discovered issue with parallel execution. Workaround and filed a TODO.
  alxmrs
  committed
  e26dda5
- try to fix clippy
  alxmrs
  committed
  6886af8
Commits on Feb 1, 2026
- Simplify the code given feedback from Maxime.
  alxmrs
  committed
  2615b6a
- cleaner implementation with `try_stream` macro.
  alxmrs
  committed
  f39f9d8
- fix: typo.
  alxmrs
  committed
  a51bb7f

Commits

Commits on Jan 15, 2026

Commits on Jan 16, 2026

Commits on Jan 18, 2026

Commits on Jan 19, 2026

Commits on Jan 24, 2026

Commits on Jan 25, 2026

Commits on Jan 26, 2026

Commits on Feb 1, 2026