Skip to content

fix: dispatch map lookups with normalized keys and nondeterministic null-guarded children - #5867

Open
dwsmith1983 wants to merge 114 commits into
apache:mainfrom
dwsmith1983:fix/serde-map-keys-and-nondeterministic-children
Open

dwsmith1983 wants to merge 114 commits into
apache:mainfrom
dwsmith1983:fix/serde-map-keys-and-nondeterministic-children

Conversation

@dwsmith1983

@dwsmith1983 dwsmith1983 commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #5580, closes #5781.

Rationale for this change

Map lookups with float, collated or complex keys fell back to Spark for the whole projection instead of running Spark's generated code inside Comet. size, arrays_zip and map_from_arrays evaluate a nondeterministic child twice in their native NULL guard, so they returned silently wrong results.

What changes are included in this PR?

  • map_col[key] and element_at run the key types they decline through the codegen dispatcher.
  • size, arrays_zip, map_from_arrays and array_append send a nondeterministic child through the dispatcher, which evaluates it once.
  • These fall back to Spark instead when the result holds a calendar interval, or a skippable argument holds a shared subexpression.
  • Shuffle and top-k planning checks use the same subexpression scope as execution.
  • The expressions guide marks the affected functions as Hybrid.

For branch-1.2, #7670 makes size, arrays_zip and map_from_arrays fall back to Spark for a nondeterministic child instead of backporting this routing, which also carries the #5580 key change.

How are these changes tested?

New and updated SQL file tests cover each serde with stateful, deterministic and NULL operands, map lookups over float, struct and collated keys, shared subexpressions under ANSI, and calendar-interval results, all checked against Spark. #5875 measured this routing at 9 to 11 percent slower for lookup-only DOUBLE-key projections and 10 to 13 percent faster for mixed ones.

@github-actions github-actions Bot added bug Something isn't working area:expressions Expression evaluation labels Sep 11, 2026
@dwsmith1983
dwsmith1983 force-pushed the fix/serde-map-keys-and-nondeterministic-children branch 2 times, most recently from ff812cc to 689c563 Compare September 12, 2026 01:48
…ull-guarded children

Map lookups with float, collated or complex keys were declined by the
serde without a dispatch fallback, so the whole projection fell back to
Spark. Four array and map serdes reproduce NULL propagation with a guard
that serializes the child twice, so a stateful child was evaluated twice
natively and returned wrong answers.

Mix the codegen dispatch fallback into the map lookups, and decline any
nondeterministic child in size, array_append, arrays_zip and
map_from_arrays through one shared gate so Spark's generated code
evaluates it once.

Closes apache#5580
Closes apache#5781

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment above getSupportLevel says only the array operand sits under the NULL guard, but the item is inside the THEN branch too, and DataFusion's CaseExpr filters the batch before it evaluates THEN. So the item is evaluated only on the rows the guard selects.

That matters for array_append specifically, because it is the one expression in this group whose Spark codegen does not short-circuit. On 3.4 and 3.5 ArrayAppend.doGenCode emits leftGen.code + rightGen.code + ctx.nullSafeExec(left.nullable, leftGen.isNull) { ... }, so rightGen.code runs on every row even when the array is NULL. ElementAt and MapFromArrays go through nullSafeCodeGen and Size has one child, which is why the guard matches Spark for those three.

On your own 16 row table, SELECT _1, array_append(IF(_1 % 2 = 0, array(1), CAST(NULL AS ARRAY<INT>)), monotonically_increasing_id()) FROM test_array_append_nondet gives Spark [1,0], [1,2], [1,4] and so on for the even rows, because the counter advances on all 16. Comet evaluates the native monotonically_increasing_id only over the 8 filtered rows and gives [1,0], [1,1], [1,2] up to [1,7]. The same filtering also swallows an error the item would have raised. Under ANSI, array_append(IF(_1 % 2 = 0, array(1), CAST(NULL AS ARRAY<INT>)), 1 / (_1 - 1)) raises DIVIDE_BY_ZERO in Spark at _1 = 1 and returns cleanly in Comet.

The expect_native case below uses array(1), which is non-nullable, so the guard mask is all true and CaseExpr takes the branch that skips the filter entirely. That case passes whatever the serde does, which is why the fixture does not catch this. Would it make sense for NullGuardSupport to cover both children of ArrayAppend, and to change that fixture to a nullable operand with a stateful item so it fails without the change?

On sequencing, I have commented on #5875 pointing it here, since this PR carries the null-guard correctness work as well as the map lookup dispatch and that makes it the better base. Two things to settle before it lands though. #5875 measured this exact routing change and found lookup-only projections running 8.7 to 11.4 percent slower with the dispatcher on at 65536 rows, with no reproducible benefit at 1024 rows. Do you have numbers for the routing here? And #5854 teaches the native map builders LAST_WIN and null key rejection, which would remove the Incompatible branch this PR routes through the dispatcher for map_from_arrays, so that one needs an order agreed with @peterxcli.

Last thing is coverage for the routes this enables. map_from_arrays_dedup_policy.sql sets spark.comet.exec.scalaUDF.codegen.enabled=false and keeps asserting expect_fallback(mapKeyDedupPolicy), so the new LAST_WIN dispatch route is never exercised. Could those queries assert expect_dispatch(map_from_arrays) instead, with a separate dispatcher-off file if the fallback is still worth pinning? The four new nondeterministic fixtures also build every array inline over an int column, so no dispatched kernel in them reads an Arrow ListVector from a column. #5844 hit a real Spark 3.4.3 ColumnarArray.copy problem in exactly that shape, so an array<int> column in at least the size and map_from_arrays fixtures would be worth having. While you are in expressions.md, cardinality still says Native two rows above size, and Spark registers both names against Size.

@dwsmith1983

Copy link
Copy Markdown
Contributor Author

Would it make sense for NullGuardSupport to cover both children of ArrayAppend, and to change that fixture to a nullable operand with a stateful item so it fails without the change?

Yes to both. CometArrayAppend now declines when either child is nondeterministic, and the fixture's native case is a nullable array with a stateful item, which fails without the change on exactly the 16-row table you used. The ANSI case where the item raises on a filtered row is the same mechanism but not nondeterminism; I have noted it in the description as the remaining gap for the guard shape rather than widening the decline to every non-literal item here.

Could those queries assert expect_dispatch(map_from_arrays) instead, with a separate dispatcher-off file if the fallback is still worth pinning?

Done: map_from_arrays_dedup_policy.sql asserts the dispatched route with the dispatcher on, and map_from_arrays_dedup_policy_dispatcher_off.sql pins the fallback.

an array<int> column in at least the size and map_from_arrays fixtures would be worth having

Added to both: the dispatched kernel now reads its arrays from columns as well as building them inline. cardinality says Hybrid.

Do you have numbers for the routing here?

Not yet; I will run the same shape #5875 measured before this lands. On #5854, once native LAST_WIN lands the map_from_arrays dispatch branch goes away and only the dispatcher-off fixture needs to follow, so I am fine with #5854 landing first.

@andygrove

Copy link
Copy Markdown
Member

Triage note: #5875 also closes #5580, and mixes CodegenDispatchFallback into the same two serdes (CometMapExtract and CometElementAt in maps.scala) with overlapping element_at / get_map_value fixtures. @LinSimon-901101 flagged the overlap in that description rather than letting us find it in review, and offered to defer.

From reading both, this one looks like the superset: it also closes #5781 for the stateful child being serialized twice under the CASE WHEN child IS NOT NULL guard in size, array_append, arrays_zip and map_from_arrays, which #5875 does not touch. Are you both happy for this PR to carry the map-lookup serde, with any non-overlapping coverage from #5875 rebased on top once it lands?

peterxcli added a commit to peterxcli/datafusion-comet that referenced this pull request Sep 18, 2026
…aches the null guards

The nested null guards serialize each child a second time inside the
`map_from_arrays` call, so a stateful child advances independently in
each copy: the guard's copy sees every row while the constructor's copy
sees only the rows the guard selected. With

    map_from_arrays(IF(monotonically_increasing_id() % 2 != 0, array(1), NULL), array(2))

over sixteen rows in one partition, Spark returns eight maps and Comet
returned four (apache#5781). Under LAST_WIN this case used to fall back for
the policy alone, so running the policy natively exposed it there.

Port `NullGuardSupport` from apache#5867 unchanged in name, reason and
position, so that PR rebases by dropping the hunk, and decline a
nondeterministic child in `CometMapFromArrays.getSupportLevel` as
`Unsupported`; the projection falls back to Spark, which evaluates the
child once. apache#5867 still routes the same decline through the JVM codegen
dispatcher and applies it to `size`, `array_append` and `arrays_zip`.

Cover it with the query above as a Scala test on a one-partition table
under LAST_WIN, the same query in `map_from_arrays_dedup_policy.sql`,
and `map_from_arrays_nondeterministic_child.sql` for the default policy,
which mirrors the fixture in apache#5867 with `expect_fallback` in place of
`expect_dispatch`.
@dwsmith1983

Copy link
Copy Markdown
Contributor Author

Are you both happy for this PR to carry the map-lookup serde, with any non-overlapping coverage from #5875 rebased on top once it lands?

Yes, happy for this PR to carry the map-lookup serde. The size, array_append, arrays_zip and map_from_arrays fix has no counterpart in #5875, so rebasing its extra fixtures and benchmark cases on top once this lands is the smaller move. One sequencing note: if #5854 lands first, its native map_from_arrays replaces the LAST_WIN decline this PR wraps, and the gate here becomes the fallback chain peterxcli described on that PR.

@andygrove andygrove added this to the 1.1.0 milestone Sep 18, 2026
@dwsmith1983

Copy link
Copy Markdown
Contributor Author

Thanks @kazuyukitanimura. Merged it in e3a5cd2. With the item now evaluated natively on NULL-array rows, this PR's array_append item gate is gone. It keeps #7025's rule of declining a nondeterministic operand only for a nullable array, and that case now goes to the JVM codegen dispatcher instead of falling back to Spark.

…ures

Drop comments that restate the code, a query or its expectation, and
shorten the rest to the Spark codegen behavior and fixture values they
explain.
…d-nondeterministic-children

# Conflicts:
#	spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
#	spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleExchangeExec.scala
The serde wraps decimal arithmetic in CheckOverflow before the dispatcher sees it, so the decline only recognizes a shared decimal key when both sides are promoted. Without that, Comet returned rows where Spark raises DIVIDE_BY_ZERO on a NULL map.
@dwsmith1983

Copy link
Copy Markdown
Contributor Author

@comphead could you take another look when you get a chance? Each of your Oct 20 comments has a reply in its thread, and the changes are in e3a5cd2, on top of main with #7025 merged in.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working through all the rounds on this @dwsmith1983. The shared-subexpression check in skipsSharedSubexpression errs toward falling back to Spark, which is the right direction, and running the shuffle-key and top-k checks in the same scope as execution closes the gap @sunchao found. I have two things before this goes in.

The expressions.md rows for arrays_zip, element_at (array and map), size and cardinality now say Hybrid but leave Notes empty. Could each say what goes through the JVM codegen dispatcher, like the neighbouring Hybrid rows do? For element_at and m[key] that's float, double, collated or complex keys, plus a nondeterministic operand under ANSI. For size it's a nondeterministic child. For arrays_zip it's a nondeterministic child or map, interval, TIME or variant elements. Could the array_append note also say it only applies on Spark 3.x, since 4.0 rewrites it to array_insert?

#5781 is a wrong answer in the 1.2 release (size, arrays_zip and map_from_arrays evaluate a nondeterministic child twice natively). Per backporting.md, should this go to branch-1.2? If the dispatcher routing is too much for a backport, a smaller fix there could mark those cases Unsupported so they fall back to Spark. Either way, could the description say which?

I've approved the upstream CI run so the other Spark profiles run on this head.

@dwsmith1983

Copy link
Copy Markdown
Contributor Author

Could each say what goes through the JVM codegen dispatcher, like the neighbouring Hybrid rows do?

Done in 46817f9, with the array_append note limited to Spark 3.x. I checked each against its serde. element_at on an array dispatches under ANSI only when the array itself is nullable and nondeterministic. m[key] has only the key-type check, so its note covers just the key types. Variant elements in arrays_zip fall back to Spark rather than dispatch, since the dispatcher has no variant support.

Per backporting.md, should this go to branch-1.2?

Yes for #5781. size, arrays_zip and map_from_arrays evaluate a nondeterministic child twice on 1.2 too, while array_append and ANSI element_at already decline there. I wouldn't backport this PR as is, though. It also moves map lookups with float, collated and complex keys to the dispatcher (#5580). That's an enhancement, and #5875 measured it 9 to 11 percent slower for lookup-only DOUBLE-key projections. #7670 makes those three fall back to Spark on branch-1.2, with this PR's SQL tests switched to expect_fallback.

The run you approved hadn't started its build after two hours, so the new head (with main merged) needs approving again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:expressions Expression evaluation area:shuffle Shuffle (JVM and native) bug Something isn't working run-all-spark-profiles Run the Comet test suites against every Spark profile on this pull request, ahead of the merge queue

Projects

None yet

5 participants