You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Lazy filter/validate plans contain absolute install paths causing serialized plans to differ between environments #413
My pipelines use LazyFrame.serialize() bytes to determine whether the execution plan has changed.
When using Schema.filter(..., eager=False) (also validate and cast) inside these pipelines, the lazy result contains two things that depend on where dataframely is installed. This causes the same code on the same data to produce different LazyFrame.serialize() bytes when run from a different machine/user/checked out directory, etc. Is there any way to prevent this / fix this in dataframely?
In the plan I can see:
An absolute path to dataframely installation: all_rules_horizontal, all_rules, all_rules_required are registered with use_abs_path=True. The plan stores e.g. /home/david/project/.pixi/envs/default/lib/python3.13/site-packages/dataframely/_native.abi3.so.
A Python function stored in the plan: match_to_schema passes cast_lenient/cast_none/cast_strict to lf.pipe_with_schema(...), so the function is stored in the plan. These are nested functions within match_to_schema, so serializing the plan pickles their full code, including the absolute path of dataframely/_match_to_schema.py.
Example:
importsysimportdataframelyasdyimportpolarsasplclassMySchema(dy.Schema):
a=dy.Int64(nullable=False)
lf=pl.LazyFrame({"a": [1, None]})
valid, failure=MySchema.filter(lf, cast=True, eager=False)
plan=valid.serialize()
print(sys.prefix.encode() inplan) # True: absolute env path is part of the planprint(b"cloudpickle"inplan) # True: a pickled Python function is part of the plan
Ugh, thanks for pointing this out! Specifying the absolute path was necessary in some old version of polars to work around an existing bug. I think we can easily get rid of this today, however.
Mh, even when setting use_abs_path=False, it seems that polars resolves that to an absolute path and puts that into the plan. I'm not sure this will be easy to fix 😕
Activity
borchero commentedon Oct 7, 2026
Ugh, thanks for pointing this out! Specifying the absolute path was necessary in some old version of polars to work around an existing bug. I think we can easily get rid of this today, however.
borchero commentedon Oct 8, 2026
Mh, even when setting
use_abs_path=False, it seems that polars resolves that to an absolute path and puts that into the plan. I'm not sure this will be easy to fix 😕