Repository navigation
Union types not handled when building AnalysisContext::try_from_column #8499
Description
Describe the bug
AnalysisContext will try and create a ScalarValue when there are no statistics for a given column. Since there is no ScalarValue representation for union types this fails.
The constructor is fallible so this isn't a bug per se, but in other contexts this value is unwrapped and can cause panics during query planning
To Reproduce
Minimized example of what we have observed causing panics during query planning:
#[test]
fn test_equivalence_properties_union_type() -> Result<()> {
let union_type = DataType::Union(
UnionFields::new(
vec![0,1],
vec![
Field::new("f1", DataType::Int32, true),
Field::new("f2", DataType::Utf8, true),
],
),
UnionMode::Sparse,
);
let schema = Arc::new(Schema::new(vec![
Field::new("c1", DataType::Int32, true),
Field::new("c2", union_type, true),
]));
let exec = FilterExec::try_new(
binary(
binary(col("c1", &schema)?, Operator::GtEq, lit(1u32), &schema)?,
Operator::And,
binary(col("c1", &schema)?, Operator::LtEq, lit(4u32), &schema)?,
&schema,
)?,
Arc::new(EmptyExec::new(true, schema.clone()))
)?;
exec.statistics().unwrap();
Ok(())
}
Since FilterExec::equivalence_properties unwraps the result of FilterExec::statistics this just panics.
Expected behavior
FilterExec::equuivalence_properties is infallible so it shouldn't panic :)
Additional context
There is probably a way to patch this but seems like we should just add a ScalarValue::Union variant.
Activity
thinkharderdev commentedon Dec 11, 2023
cc @Dandandan
alamb commentedon Dec 12, 2023
There is a similar report in #8262 (related to
Maptype)alamb commentedon Dec 12, 2023
cc @mustafasrepo
berkaysynnada commentedon Dec 13, 2023
With the new design of statistics underway, this problem may be resolved by simply using an absent interval. Unless it's an emergency, I don't think we should apply a temporary patch to the analysis context. It would be better to add missing scalar value variants.
thinkharderdev commentedon Dec 13, 2023
Yeah, I agree. I think adding the missing scalar value variants is probably the better approach either way as there are probably other contexts where it would be useful to have them