You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Cast from boolean to decimal ignores eval mode: throws where Spark returns NULL in legacy/try mode #5068
The native boolean-to-decimal cast ignores eval_mode entirely and throws where Spark returns NULL in legacy and try mode.
CometCast marks boolean-to-decimal as Compatible() in all eval modes (spark/src/main/scala/org/apache/comet/expressions/CometCast.scala:329-333). The native implementation cast_boolean_to_decimal (native/spark-expr/src/conversion_funcs/boolean.rs:31-58) takes no eval_mode argument: it builds 10^scale and unconditionally raises SparkError::NumericValueOutOfRange when the value does not fit the target precision.
Spark 4.1 (Cast.scala): toPrecision(Decimal.ONE, target, ROUND_HALF_UP, nullOnOverflow = !ansiEnabled), so legacy and try mode return NULL, only ANSI throws.
Since ANSI is the default in Spark 4.x, Comet accidentally matches the default, but this is a wrong-exception bug on Spark 3.x and whenever ANSI is off, and try_cast returns an error instead of NULL.
Steps to reproduce
SETspark.sql.ansi.enabled=false;
SELECT CAST(col ASDECIMAL(1,1)) FROM (SELECT true AS col);
Any target where 10^scale exceeds the precision reproduces it, e.g. DECIMAL(1,1), DECIMAL(2,2), DECIMAL(3,3).
Spark (legacy or try mode): NULL.
Comet: throws NUMERIC_VALUE_OUT_OF_RANGE.
Expected behavior
cast_boolean_to_decimal should take the eval mode and return NULL for legacy/try, error only under ANSI.
Additional context
Found by an ANSI-mode audit of all native expressions against Spark 4.1.1. Existing CometCastSuite coverage only uses non-overflowing targets (decimal(10,2), (14,4), (30,0)), which is why this was never caught.
The native boolean-to-decimal cast was removed in #5185 (ce602ac). CometCast.canCastFromBoolean in spark/src/main/scala/org/apache/comet/expressions/CometCast.scala now reports the decimal target as unsupported, and the CodegenDispatchFallback mixin on CometCast routes it through Spark's own generated code inside the Comet pipeline, so Spark's eval-mode semantics apply in legacy, try and ANSI mode. There is no cast_boolean_to_decimal in the native crate any more. Suggest closing.
Activity
docs: document known correctness issues in the compatibility guide (#…
docs: drop compatibility notes for bugs that are now fixed
docs: drop compatibility notes for bugs that are now fixed (#5154)
kazuyukitanimura commentedon Aug 3, 2026
Let's do this in codegen
docs: document known correctness issues in the compatibility guide (a…
docs: drop compatibility notes for bugs that are now fixed (apache#5154)
dwsmith1983 commentedon Sep 11, 2026
The native boolean-to-decimal cast was removed in #5185 (ce602ac).
CometCast.canCastFromBooleanin spark/src/main/scala/org/apache/comet/expressions/CometCast.scala now reports the decimal target as unsupported, and theCodegenDispatchFallbackmixin onCometCastroutes it through Spark's own generated code inside the Comet pipeline, so Spark's eval-mode semantics apply in legacy, try and ANSI mode. There is nocast_boolean_to_decimalin the native crate any more. Suggest closing.andygrove commentedon Sep 11, 2026
Thanks @dwsmith1983. I agree, let's close this one