Skip to content

Cast from boolean to decimal ignores eval mode: throws where Spark returns NULL in legacy/try mode #5068

Closed

Description

@andygrove

Describe the bug

The native boolean-to-decimal cast ignores eval_mode entirely and throws where Spark returns NULL in legacy and try mode.

CometCast marks boolean-to-decimal as Compatible() in all eval modes (spark/src/main/scala/org/apache/comet/expressions/CometCast.scala:329-333). The native implementation cast_boolean_to_decimal (native/spark-expr/src/conversion_funcs/boolean.rs:31-58) takes no eval_mode argument: it builds 10^scale and unconditionally raises SparkError::NumericValueOutOfRange when the value does not fit the target precision.

Spark 4.1 (Cast.scala): toPrecision(Decimal.ONE, target, ROUND_HALF_UP, nullOnOverflow = !ansiEnabled), so legacy and try mode return NULL, only ANSI throws.

Since ANSI is the default in Spark 4.x, Comet accidentally matches the default, but this is a wrong-exception bug on Spark 3.x and whenever ANSI is off, and try_cast returns an error instead of NULL.

Steps to reproduce

SET spark.sql.ansi.enabled=false;
SELECT CAST(col AS DECIMAL(1,1)) FROM (SELECT true AS col);

Any target where 10^scale exceeds the precision reproduces it, e.g. DECIMAL(1,1), DECIMAL(2,2), DECIMAL(3,3).

Spark (legacy or try mode): NULL.
Comet: throws NUMERIC_VALUE_OUT_OF_RANGE.

Expected behavior

cast_boolean_to_decimal should take the eval mode and return NULL for legacy/try, error only under ANSI.

Additional context

Found by an ANSI-mode audit of all native expressions against Spark 4.1.1. Existing CometCastSuite coverage only uses non-overflowing targets (decimal(10,2), (14,4), (30,0)), which is why this was never caught.

Activity

  1. added
    priority:mediumFunctional bugs, performance regressions, broken features
    and removed on Aug 3, 2026
  2. kazuyukitanimura commented on Aug 3, 2026

    @kazuyukitanimura
    Contributor

    Let's do this in codegen

  3. added this to the 1.1.0 milestone on Aug 3, 2026
  4. dwsmith1983 commented on Sep 11, 2026

    @dwsmith1983
    Contributor

    The native boolean-to-decimal cast was removed in #5185 (ce602ac). CometCast.canCastFromBoolean in spark/src/main/scala/org/apache/comet/expressions/CometCast.scala now reports the decimal target as unsupported, and the CodegenDispatchFallback mixin on CometCast routes it through Spark's own generated code inside the Comet pipeline, so Spark's eval-mode semantics apply in legacy, try and ANSI mode. There is no cast_boolean_to_decimal in the native crate any more. Suggest closing.

  5. andygrove commented on Sep 11, 2026

    @andygrove
    MemberAuthor

    Thanks @dwsmith1983. I agree, let's close this one

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:expressionsExpression evaluationbugSomething isn't workingcorrectnesspriority:mediumFunctional bugs, performance regressions, broken features

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions