Repository navigation
feat: Support Union type in approx_distinct - #23714
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #23714 +/- ##
=======================================
Coverage 80.69% 80.69%
=======================================
Files 1088 1088
Lines 367766 367813 +47
Branches 367766 367813 +47
=======================================
+ Hits 296781 296820 +39
- Misses 53276 53279 +3
- Partials 17709 17714 +5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| } | ||
|
|
||
| #[test] | ||
| fn union_support_hll_acc_and_group_acc() { |
There was a problem hiding this comment.
not possible to do via SLT?
There was a problem hiding this comment.
It's tricky via SLT. There is no SQL union type but only union operations in datafusion.
There was a problem hiding this comment.
what if we add more rows here:
datafusion/datafusion/sqllogictest/src/test_context.rs
Lines 560 to 594 in 3a2bc0a
then can use in slts like here
datafusion/datafusion/sqllogictest/test_files/union_function.slt
Lines 18 to 30 in 3a2bc0a
9096c52 to
addd02c
Compare
| ctx.register_batch("union_table", batch).unwrap(); | ||
| } | ||
|
|
||
| fn register_approx_distinct_union_table(ctx: &SessionContext) { |
There was a problem hiding this comment.
I added a separate batch which fits better for testing.
|
thanks @mkleen |
|
@Jefffrey thanks for the review! |
## Which issue does this PR close? - Relates to apache#22989 but does not close it. More types are coming. ## Rationale for this change - Support the Arrow type `Union` for `approx_distinct` ## What changes are included in this PR? - Enable `HLLAccumulator` and `HllGroupsAccumulator` to support `Union` - Tests ## Are these changes tested? Yes ## Are there any user-facing changes? Yes, `approx_distinct` supports now `Union` but no breaking changes.
Which issue does this PR close?
approx_distinctfunction #22989 but does not close it. More types are coming.Rationale for this change
Unionforapprox_distinctWhat changes are included in this PR?
HLLAccumulatorandHllGroupsAccumulatorto supportUnionAre these changes tested?
Yes
Are there any user-facing changes?
Yes,
approx_distinctsupports nowUnionbut no breaking changes.