Skip to content

Non-ANSI divide, div and remainder, and ln, log10 and log2 return wrong results for a nondeterministic operand #7671

Open

Description

@dwsmith1983

Describe the bug

With ANSI off, /, div and % return wrong results when the divisor is a stateful nondeterministic expression, and so do ln, log10 and log2 when their argument is one. It is the same problem as #5781: Comet evaluates the operand twice, once in the guard that turns a zero divisor or a non-positive argument into NULL and once as the value, and the two copies see different rows.

  • / and div: nullIfWhenPrimitive in spark/src/main/scala/org/apache/comet/serde/arithmetic.scala wraps the divisor as If(EqualTo(x, 0), null, x).
  • %: null_if_zero_primitive in native/spark-expr/src/math_funcs/modulo_expr.rs builds the same IfExpr natively.
  • ln, log10, log2: nullIfNegative in spark/src/main/scala/org/apache/comet/serde/math.scala wraps the argument as If(LessThanOrEqual(x, 0), null, x). SQL log(base, x) goes through CometLogarithm and is not affected.

None of these checks whether the operand is deterministic, and all of them run natively.

Steps to reproduce

SET spark.sql.ansi.enabled=false;
CREATE TABLE t (id INT, v INT) USING parquet;
-- one file, so monotonically_increasing_id() is the row index 0..15
INSERT INTO t SELECT CAST(id AS INT), CAST(id + 1 AS INT) FROM range(0, 16, 1, 1);

SELECT id, v / IF(monotonically_increasing_id() % 2 = 0, 2, 0) FROM t;
SELECT id, v div IF(monotonically_increasing_id() % 2 = 0, 2, 0) FROM t;
SELECT id, v % IF(monotonically_increasing_id() % 2 = 0, 3, 0) FROM t;
SELECT id, isnan(ln(IF(monotonically_increasing_id() % 2 = 0, CAST(v AS DOUBLE), CAST(-1 AS DOUBLE)))) FROM t;

On main (ba16e1a), Spark 4.1:

  • /: Comet returns Infinity for ids 2, 6, 10 and 14, where Spark returns 1.5, 3.5, 5.5 and 7.5.
  • div: Comet returns 0 for the same ids, where Spark returns 1, 3, 5 and 7.
  • %: Comet fails with CometNativeException: Divide by zero error, where Spark returns NULL on the odd rows.
  • ln, log10, log2: Comet returns NaN for ids 2, 6, 10 and 14, where Spark returns a number.

Expected behavior

The same results as Spark. As on branch-1.2 in #7670, these could fall back to Spark for a nondeterministic operand, or go through the codegen dispatcher as #5867 does for size, arrays_zip and map_from_arrays.

Additional context

Found while preparing #7670, the branch-1.2 change for #5781.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions