Repository navigation
fix: dispatch map lookups with normalized keys and nondeterministic null-guarded children - #5867
dwsmith1983 wants to merge 114 commits into
Conversation
ff812cc to
689c563
Compare
…ull-guarded children Map lookups with float, collated or complex keys were declined by the serde without a dispatch fallback, so the whole projection fell back to Spark. Four array and map serdes reproduce NULL propagation with a guard that serializes the child twice, so a stateful child was evaluated twice natively and returned wrong answers. Mix the codegen dispatch fallback into the map lookups, and decline any nondeterministic child in size, array_append, arrays_zip and map_from_arrays through one shared gate so Spark's generated code evaluates it once. Closes apache#5580 Closes apache#5781
689c563 to
dfbf000
Compare
andygrove
left a comment
There was a problem hiding this comment.
The comment above getSupportLevel says only the array operand sits under the NULL guard, but the item is inside the THEN branch too, and DataFusion's CaseExpr filters the batch before it evaluates THEN. So the item is evaluated only on the rows the guard selects.
That matters for array_append specifically, because it is the one expression in this group whose Spark codegen does not short-circuit. On 3.4 and 3.5 ArrayAppend.doGenCode emits leftGen.code + rightGen.code + ctx.nullSafeExec(left.nullable, leftGen.isNull) { ... }, so rightGen.code runs on every row even when the array is NULL. ElementAt and MapFromArrays go through nullSafeCodeGen and Size has one child, which is why the guard matches Spark for those three.
On your own 16 row table, SELECT _1, array_append(IF(_1 % 2 = 0, array(1), CAST(NULL AS ARRAY<INT>)), monotonically_increasing_id()) FROM test_array_append_nondet gives Spark [1,0], [1,2], [1,4] and so on for the even rows, because the counter advances on all 16. Comet evaluates the native monotonically_increasing_id only over the 8 filtered rows and gives [1,0], [1,1], [1,2] up to [1,7]. The same filtering also swallows an error the item would have raised. Under ANSI, array_append(IF(_1 % 2 = 0, array(1), CAST(NULL AS ARRAY<INT>)), 1 / (_1 - 1)) raises DIVIDE_BY_ZERO in Spark at _1 = 1 and returns cleanly in Comet.
The expect_native case below uses array(1), which is non-nullable, so the guard mask is all true and CaseExpr takes the branch that skips the filter entirely. That case passes whatever the serde does, which is why the fixture does not catch this. Would it make sense for NullGuardSupport to cover both children of ArrayAppend, and to change that fixture to a nullable operand with a stateful item so it fails without the change?
On sequencing, I have commented on #5875 pointing it here, since this PR carries the null-guard correctness work as well as the map lookup dispatch and that makes it the better base. Two things to settle before it lands though. #5875 measured this exact routing change and found lookup-only projections running 8.7 to 11.4 percent slower with the dispatcher on at 65536 rows, with no reproducible benefit at 1024 rows. Do you have numbers for the routing here? And #5854 teaches the native map builders LAST_WIN and null key rejection, which would remove the Incompatible branch this PR routes through the dispatcher for map_from_arrays, so that one needs an order agreed with @peterxcli.
Last thing is coverage for the routes this enables. map_from_arrays_dedup_policy.sql sets spark.comet.exec.scalaUDF.codegen.enabled=false and keeps asserting expect_fallback(mapKeyDedupPolicy), so the new LAST_WIN dispatch route is never exercised. Could those queries assert expect_dispatch(map_from_arrays) instead, with a separate dispatcher-off file if the fallback is still worth pinning? The four new nondeterministic fixtures also build every array inline over an int column, so no dispatched kernel in them reads an Arrow ListVector from a column. #5844 hit a real Spark 3.4.3 ColumnarArray.copy problem in exactly that shape, so an array<int> column in at least the size and map_from_arrays fixtures would be worth having. While you are in expressions.md, cardinality still says Native two rows above size, and Spark registers both names against Size.
…d-nondeterministic-children
…in the dispatched fixtures
…e guard filters its evaluation
Yes to both.
Done:
Added to both: the dispatched kernel now reads its arrays from columns as well as building them inline.
Not yet; I will run the same shape #5875 measured before this lands. On #5854, once native LAST_WIN lands the |
|
Triage note: #5875 also closes #5580, and mixes From reading both, this one looks like the superset: it also closes #5781 for the stateful child being serialized twice under the |
…aches the null guards
The nested null guards serialize each child a second time inside the
`map_from_arrays` call, so a stateful child advances independently in
each copy: the guard's copy sees every row while the constructor's copy
sees only the rows the guard selected. With
map_from_arrays(IF(monotonically_increasing_id() % 2 != 0, array(1), NULL), array(2))
over sixteen rows in one partition, Spark returns eight maps and Comet
returned four (apache#5781). Under LAST_WIN this case used to fall back for
the policy alone, so running the policy natively exposed it there.
Port `NullGuardSupport` from apache#5867 unchanged in name, reason and
position, so that PR rebases by dropping the hunk, and decline a
nondeterministic child in `CometMapFromArrays.getSupportLevel` as
`Unsupported`; the projection falls back to Spark, which evaluates the
child once. apache#5867 still routes the same decline through the JVM codegen
dispatcher and applies it to `size`, `array_append` and `arrays_zip`.
Cover it with the query above as a Scala test on a one-partition table
under LAST_WIN, the same query in `map_from_arrays_dedup_policy.sql`,
and `map_from_arrays_nondeterministic_child.sql` for the default policy,
which mirrors the fixture in apache#5867 with `expect_fallback` in place of
`expect_dispatch`.
Yes, happy for this PR to carry the map-lookup serde. The |
|
Thanks @kazuyukitanimura. Merged it in e3a5cd2. With the item now evaluated natively on NULL-array rows, this PR's array_append item gate is gone. It keeps #7025's rule of declining a nondeterministic operand only for a nullable array, and that case now goes to the JVM codegen dispatcher instead of falling back to Spark. |
…d-nondeterministic-children
…ures Drop comments that restate the code, a query or its expectation, and shorten the rest to the Spark codegen behavior and fixture values they explain.
…d-nondeterministic-children
…d-nondeterministic-children # Conflicts: # spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala # spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleExchangeExec.scala
The serde wraps decimal arithmetic in CheckOverflow before the dispatcher sees it, so the decline only recognizes a shared decimal key when both sides are promoted. Without that, Comet returned rows where Spark raises DIVIDE_BY_ZERO on a NULL map.
…d-nondeterministic-children
…d-nondeterministic-children
…d-nondeterministic-children
…d-nondeterministic-children
…d-nondeterministic-children
…d-nondeterministic-children
andygrove
left a comment
There was a problem hiding this comment.
Thanks for working through all the rounds on this @dwsmith1983. The shared-subexpression check in skipsSharedSubexpression errs toward falling back to Spark, which is the right direction, and running the shuffle-key and top-k checks in the same scope as execution closes the gap @sunchao found. I have two things before this goes in.
The expressions.md rows for arrays_zip, element_at (array and map), size and cardinality now say Hybrid but leave Notes empty. Could each say what goes through the JVM codegen dispatcher, like the neighbouring Hybrid rows do? For element_at and m[key] that's float, double, collated or complex keys, plus a nondeterministic operand under ANSI. For size it's a nondeterministic child. For arrays_zip it's a nondeterministic child or map, interval, TIME or variant elements. Could the array_append note also say it only applies on Spark 3.x, since 4.0 rewrites it to array_insert?
#5781 is a wrong answer in the 1.2 release (size, arrays_zip and map_from_arrays evaluate a nondeterministic child twice natively). Per backporting.md, should this go to branch-1.2? If the dispatcher routing is too much for a backport, a smaller fix there could mark those cases Unsupported so they fall back to Spark. Either way, could the description say which?
I've approved the upstream CI run so the other Spark profiles run on this head.
…d-nondeterministic-children
Done in 46817f9, with the
Yes for #5781. The run you approved hadn't started its build after two hours, so the new head (with main merged) needs approving again. |
Which issue does this PR close?
Closes #5580, closes #5781.
Rationale for this change
Map lookups with float, collated or complex keys fell back to Spark for the whole projection instead of running Spark's generated code inside Comet.
size,arrays_zipandmap_from_arraysevaluate a nondeterministic child twice in their native NULL guard, so they returned silently wrong results.What changes are included in this PR?
map_col[key]andelement_atrun the key types they decline through the codegen dispatcher.size,arrays_zip,map_from_arraysandarray_appendsend a nondeterministic child through the dispatcher, which evaluates it once.For
branch-1.2, #7670 makessize,arrays_zipandmap_from_arraysfall back to Spark for a nondeterministic child instead of backporting this routing, which also carries the #5580 key change.How are these changes tested?
New and updated SQL file tests cover each serde with stateful, deterministic and NULL operands, map lookups over float, struct and collated keys, shared subexpressions under ANSI, and calendar-interval results, all checked against Spark. #5875 measured this routing at 9 to 11 percent slower for lookup-only DOUBLE-key projections and 10 to 13 percent faster for mixed ones.