You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If xESMF works for you there is no reason to move over. However;
if you are running into memory issues (the transformation matrix has to fit in memory completely)
are interested in running your code on other platforms (where ESMF is not available, or can be cumbersome for others to install)
or xESMF is a bit too slow
then this package could be for you. Note that your regridding has to be from rectilinear -> rectilinear, and not between different Coordinate Reference Systems.
Thanks so much @BSchilperoort for the prompt response!!!
Your points are indeed pretty compelling. I'm not sure exactly when, but I'll probably give this a try at some point, and I'll make sure to report back. Thanks again!
xesmf always has to generate a large sparse array of weights in serial, which scales like the number of grid points, and is a killer for small jobs. 30s to generate weights on a 1/4deg grid, only to regrid a small array in a few ms is a bummer. Since xarray-regrid limits to rectilinear grids where we can separate each dimension, this step usually feels near instantaneous across the different methods.
Packaging for xesmf has gotten better but is still a hassle due to the ESMF dependency. You need conda
Everything here is built from the modern pangeo stack so is easily modifiable and extensible
Even ignoring the weight generation bottleneck for small data, performance on large and/or chunked datasets ranges from on par to 10x+ faster across different benchmarks after recent enhancements.
The obvious limitation is non-rectilinear grids, where the flexibility of ESMF is hard to beat.
257 ms ± 2.05 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
This is basically an intractable problem for xesmf. I tried using their chunked parallel weight generation scheme and it still ran for 20 minutes then crashed.
ETA 1hr, NCEI server having a bad day I guess. I used xr.ones_like to shortcut.
With -1 chunks, regridding takes about 9s and uses ~15GB of memory. With 1000x1000 chunks, 4s and ~2GB of memory. Pretty good since the data itself is 3.5GB.
Activity
BSchilperoort commentedon Sep 12, 2024
Hi Ben,
Thanks for the feedback!
If xESMF works for you there is no reason to move over. However;
then this package could be for you. Note that your regridding has to be from rectilinear -> rectilinear, and not between different Coordinate Reference Systems.
Notebooks comparing xarray-regrid to CDO and xESMF are available on the docs. For example: https://xarray-regrid.readthedocs.io/en/latest/notebooks/benchmarks/benchmarking_conservative.html
That should show you how the xESMF and xarray-regrid methods differ (they're quite similar).
If you do try xarray-regrid on your workflow it would be great to hear how it runs better/worse than xESMF (CPU time as well as memory use).
maresb commentedon Sep 12, 2024
Thanks so much @BSchilperoort for the prompt response!!!
Your points are indeed pretty compelling. I'm not sure exactly when, but I'll probably give this a try at some point, and I'll make sure to report back. Thanks again!
slevang commentedon Sep 26, 2024
Adding my 2C on advantages this package offers:
xesmfalways has to generate a large sparse array of weights in serial, which scales like the number of grid points, and is a killer for small jobs. 30s to generate weights on a 1/4deg grid, only to regrid a small array in a few ms is a bummer. Sincexarray-regridlimits to rectilinear grids where we can separate each dimension, this step usually feels near instantaneous across the different methods.xesmfhas gotten better but is still a hassle due to the ESMF dependency. You need condaThe obvious limitation is non-rectilinear grids, where the flexibility of ESMF is hard to beat.
slevang commentedon Sep 26, 2024
An nice example for point 1: trying to regrid a large fixed land surface dataset. Here's the 30 arc second ETOPO geoid, which is 21600x43200:
This is basically an intractable problem for xesmf. I tried using their chunked parallel weight generation scheme and it still ran for 20 minutes then crashed.
dcherian commentedon Sep 26, 2024
To be clear, this benchmarks the weight generation and graph creation, correct? Does it compute smoothly too?
slevang commentedon Sep 26, 2024
Then I have to actually download the file 😆 . But yes I'll try that
slevang commentedon Sep 26, 2024
ETA 1hr, NCEI server having a bad day I guess. I used
xr.ones_liketo shortcut.With -1 chunks, regridding takes about 9s and uses ~15GB of memory. With 1000x1000 chunks, 4s and ~2GB of memory. Pretty good since the data itself is 3.5GB.