You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Nested DATE to numeric casts in structs and maps return the day count or fail #6316
In Legacy mode (spark.sql.ansi.enabled=false, the default on Spark 3.x), Spark returns NULL for
a cast from DATE to any numeric or boolean type, including when the date is a struct field or a
map value. Comet runs those nested casts natively and gets them wrong:
DATE to INT returns the day count since the epoch instead of NULL, silently.
DATE to BIGINT, SMALLINT, TINYINT, BOOLEAN, FLOAT, DOUBLE or DECIMAL fails the
query with an internal error (SQLSTATE XX000), for example Native cast invoked for unsupported cast from Date32 to Boolean.
The same happens one level further down, for example ARRAY<STRUCT<d: DATE>> to ARRAY<STRUCT<d: INT>>. Arrays of dates are not affected.
Steps to reproduce
SET spark.sql.ansi.enabled=false;
CREATE TABLE t(s struct<d: date>, m map<string, date>) USING parquet;
INSERT INTO t VALUES (named_struct('d', date '2024-01-15'), map('k', date '2024-01-15'));
SELECT cast(s AS struct<d: int>), cast(m AS map<string, int>) FROM t;
-- Spark: {null}, {k -> null}
-- Comet: {19737}, {k -> 19737}
SELECT cast(s AS struct<d: boolean>) FROM t;
-- Spark: {null}
-- Comet: task failure, Native cast invoked for unsupported cast from Date32 to Boolean
Reproduced on main at 7649361 with Spark 4.1, and the plan is fully native (CometProject).
Cause
CometCast.convert replaces a top-level cast that is always null (isAlwaysCastToNull) with a
null literal, so isSupported reports DATE to a numeric or boolean type as Compatible. The
struct and map arms of isSupported recurse into that same check, but a nested cast does reach
the native kernel. There, (Date32, Int32) reinterprets the day count, which is the kernel unix_date relies on, and the other targets have no kernel at all. Arrays of dates already have a
guard: ARRAY<DATE> is unsupported to anything other than ARRAY<INT> and ARRAY<STRING>, and
the list kernel returns nulls for DATE to INT.
Activity