Skip to content

Lazy filter/validate plans contain absolute install paths causing serialized plans to differ between environments #413

Open

Description

My pipelines use LazyFrame.serialize() bytes to determine whether the execution plan has changed.

When using Schema.filter(..., eager=False) (also validate and cast) inside these pipelines, the lazy result contains two things that depend on where dataframely is installed. This causes the same code on the same data to produce different LazyFrame.serialize() bytes when run from a different machine/user/checked out directory, etc. Is there any way to prevent this / fix this in dataframely?

In the plan I can see:

  1. An absolute path to dataframely installation: all_rules_horizontal, all_rules, all_rules_required are registered with use_abs_path=True. The plan stores e.g. /home/david/project/.pixi/envs/default/lib/python3.13/site-packages/dataframely/_native.abi3.so.

  2. A Python function stored in the plan: match_to_schema passes cast_lenient/cast_none/cast_strict to lf.pipe_with_schema(...), so the function is stored in the plan. These are nested functions within match_to_schema, so serializing the plan pickles their full code, including the absolute path of dataframely/_match_to_schema.py.

Example:

import sys

import dataframely as dy
import polars as pl


class MySchema(dy.Schema):
    a = dy.Int64(nullable=False)


lf = pl.LazyFrame({"a": [1, None]})
valid, failure = MySchema.filter(lf, cast=True, eager=False)
plan = valid.serialize()

print(sys.prefix.encode() in plan)  # True: absolute env path is part of the plan
print(b"cloudpickle" in plan)  # True: a pickled Python function is part of the plan

(dataframely 3.1.2, polars 2.0.0, Python 3.13)

Activity

  1. borchero commented on Oct 7, 2026

    @borchero
    Member

    Ugh, thanks for pointing this out! Specifying the absolute path was necessary in some old version of polars to work around an existing bug. I think we can easily get rid of this today, however.

  2. borchero commented on Oct 8, 2026

    @borchero
    Member

    Mh, even when setting use_abs_path=False, it seems that polars resolves that to an absolute path and puts that into the plan. I'm not sure this will be easy to fix 😕

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions