Repository navigation
Allow to pass through columns #100
Description
I have a dataframe with a lot of columns (>100) and would like to not need to specify all of them.
Would it be possiuble to support a strict = False mode and/or to specify types for multiple columns?
Maybe such an api:
class FeaturesForwardModelSchema(dy.Schema):
claim = dy.String(primary_key=True, nullable=False)
cost = dy.Int64(primary_key=True, nullable=False)
dataframely_options: DataframelyOptions(strict=False)
df = pl.DataFrame({
"claim": ["a"],
"cost": [1],
"extra_col": ["foo"],
})
df_validated = FeaturesForwardModelSchema.validate(df)
df_validated["extra_col"]alternatively, maybe something like this?
FeaturesForwardModelSchema.validate(df, strict=False)Although this could be quite annoying, needing to specify it every time
Activity
pavelzw commentedon Jul 18, 2025
specifying schemas for multiple columns at the same time could be interesting with globs?
DySchema.validate(df, cast=True)should not implicitly drop extra columns #186 as a duplicate of this issueDeflateAwning commentedon Oct 23, 2025
I'd suggest adding an argument to the
.validate()method likeon_extra_columns: Literal["drop", "raise", "keep_at_end"] = "drop"might be an appropriate way to add support for not silently allowing this. Thestrict: boolproposed in this issue does not address the 3+ possible behaviours, and creates the start of a challenging-to-upgrade API.borchero commentedon Oct 24, 2025
I think the core question to answer is whether this option should live in the schema or be an option to
validate(as well asfilterandcast). I.e. is this behavior a property of the schema or should it change for every operation. I haven't fully made up my mind yet but I tend to treat it as a property of the schema as, otherwise, code documentation suffers: when encountering the schema in a function signature, it isn't clear whether it may have additional columns or not.DeflateAwning commentedon Oct 25, 2025
If you're looking for an opinion, I think that these
operationssettings should be able to be defined in the schema, and overridden at the.validate()call via kwargs there.kszlim commentedon Nov 19, 2025
This would be useful, or even an option in
dy.Collectionto return untyped LazyFrames? It's not always possible/desirable to specify the data schemas up front everywhere.mullimanko commentedon Nov 20, 2025
As an aside, I would endorse to define a column constraint in a schema that applies to all columns in the dataframe. For example, I would like to check if all values in all columns of a dataframe are > 10.
brendancooley commentedon Nov 21, 2025
Oliver Borchert (@borchero) I think I'd vote the other way. In the following example
We can think of the data frame
as implementing (and embellishing)
FooSchema. Any expression that you can apply to a frame that strictly implementsFooSchema(contains only "foo") you can also apply todf_with_bar(I think). Not my area but LSP yada yada...I don't think dataframely currently supports this but it would be neat if we could type
df_with_baras followsMaybe some work needed on python first, but this would make it easy to compose subschemas together to validate dataframes that are the result of joins/etc while retaining the ability to validate the components or validate partially when partial validation is sufficient. I think configuring on the
Schemaforecloses these patterns.Stumbled onto this issue also looking to pass extra columns through
validate(..., cast=True), so also support half-measures here. I like DeflateAwning's proposalSome links to current
selectlogicdataframely/dataframely/_match_to_schema.py
Line 13 in 1660d88
dataframely/dataframely/schema.py
Line 689 in 1660d88
mathisdrn commentedon Aug 1, 2026
on_extra_columns: Literal["drop", "raise", "keep_at_end"] = "drop"Personal opinion but I would add
keeporkeep_in_placeas an option too.