Repository navigation
Nullable column missing fails validate(..., cast=True) #253
Description
If a column is nullable, why do we not allow dataframely to create the column if it doesn't exist yet (filled with nulls) when casting? I realise you've probably thought about this but I couldn't find an explicit mention of this in the docs or GitHub issues.
import dataframely as dy
import polars as pl
class TableSchema(dy.Schema):
column_a = dy.String(nullable=False)
column_b = dy.String(nullable=True)
df = pl.DataFrame({"column_a": 0})
TableSchema.validate(df, cast=True)
# Raises SchemaError
# SchemaError: 1 missing columns for schema 'TableSchema': 'column_b'
Activity
borchero commentedon Jan 11, 2026
One key design decision in dataframely today is that is doesn't ever alter inputs to
validate. We are currently re-evaluating this decision though (also in the context of #237) and will revisit this once we've reached a conclusion :)brendancooley commentedon Jan 18, 2026
Is this in tension with the behavior noted by Ion Koutsouris (@ion-elgreco) in #258 (silently dropping extraneous columns)? Should operations that mutate the data frame be handled explicitly by different methods?
filter,derive, etc?borchero commentedon Jan 18, 2026
Yeah, definitely related, there is also #100... all of those are in the same vein and likely warrant an extension to the API in one or multiple ways