You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Non-ANSI divide, div and remainder, and ln, log10 and log2 return wrong results for a nondeterministic operand #7671
With ANSI off, /, div and % return wrong results when the divisor is a stateful nondeterministic expression, and so do ln, log10 and log2 when their argument is one. It is the same problem as #5781: Comet evaluates the operand twice, once in the guard that turns a zero divisor or a non-positive argument into NULL and once as the value, and the two copies see different rows.
/ and div: nullIfWhenPrimitive in spark/src/main/scala/org/apache/comet/serde/arithmetic.scala wraps the divisor as If(EqualTo(x, 0), null, x).
%: null_if_zero_primitive in native/spark-expr/src/math_funcs/modulo_expr.rs builds the same IfExpr natively.
ln, log10, log2: nullIfNegative in spark/src/main/scala/org/apache/comet/serde/math.scala wraps the argument as If(LessThanOrEqual(x, 0), null, x). SQL log(base, x) goes through CometLogarithm and is not affected.
None of these checks whether the operand is deterministic, and all of them run natively.
Steps to reproduce
SETspark.sql.ansi.enabled=false;
CREATETABLEt (id INT, v INT) USING parquet;
-- one file, so monotonically_increasing_id() is the row index 0..15INSERT INTO t SELECT CAST(id ASINT), CAST(id +1ASINT) FROM range(0, 16, 1, 1);
SELECT id, v / IF(monotonically_increasing_id() % 2=0, 2, 0) FROM t;
SELECT id, v div IF(monotonically_increasing_id() % 2=0, 2, 0) FROM t;
SELECT id, v % IF(monotonically_increasing_id() % 2=0, 3, 0) FROM t;
SELECT id, isnan(ln(IF(monotonically_increasing_id() % 2=0, CAST(v AS DOUBLE), CAST(-1AS DOUBLE)))) FROM t;
/: Comet returns Infinity for ids 2, 6, 10 and 14, where Spark returns 1.5, 3.5, 5.5 and 7.5.
div: Comet returns 0 for the same ids, where Spark returns 1, 3, 5 and 7.
%: Comet fails with CometNativeException: Divide by zero error, where Spark returns NULL on the odd rows.
ln, log10, log2: Comet returns NaN for ids 2, 6, 10 and 14, where Spark returns a number.
Expected behavior
The same results as Spark. As on branch-1.2 in #7670, these could fall back to Spark for a nondeterministic operand, or go through the codegen dispatcher as #5867 does for size, arrays_zip and map_from_arrays.
Additional context
Found while preparing #7670, the branch-1.2 change for #5781.
Activity