Repository navigation
substrait: add an extension point for per-field Arrow metadata on user-defined types - #176
Closed
moraistejerina wants to merge 1 commit into
Closed
moraistejerina wants to merge 1 commit into
moraistejerina wants to merge 1 commit into
Conversation
…strait types Arrow cannot represent some Substrait types natively. Our "json", "group", and "inet" user-defined types are all mapped to plain Utf8, which is then indistinguishable from a real string column once it reaches a consumer that only sees the Arrow schema (e.g. over Arrow Flight, with no access to the original plan). Retriever's streaming shadow variant was comparing against control_group results without this distinction, producing false-positive column_types_differ diffs. Add SubstraitConsumer::consume_type_metadata, an additive extension point defaulting to Ok(None), that lets a consumer attach Arrow field metadata for a Substrait type it recognizes. It takes the whole Type rather than just a UserDefined variant, so a consumer can also key off a type nested inside a list, map, or struct. Apply it at a single point, substrait_type_to_field, that every Arrow Field built from a Substrait type now goes through: the two public field_from_substrait_type* entry points, the list element field, the map key/value fields, the struct child fields, and the EmptyMap literal. This keeps metadata attached for a type nested inside a list, map, or struct without every call site having to ask for it, and avoids a signature change to from_substrait_type, which keeps returning a bare DataType. A Dictionary-variation map is the one case that still can't carry metadata, since DataType::Dictionary has no field to hold it. Existing consumers are unaffected: the default keeps today's behavior exactly, and no public signature changes. Submitted upstream as apache#25015; this cherry-picks onto branch-54 ahead of that landing so dd-datafusion can consume it now.
moraistejerina
force-pushed
the
jorge.morais/json-udt-field-metadata
branch
from
September 7, 2026 12:52
8a4fd34 to
a1d1741
Compare
Author
|
Superseded by #178, a proper cherry-pick (with -x) of the now-merged apache#25015, per the confluence "upstream first" process. This branch predated the merge and manually recreated the diff. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
consume_user_defined_typemaps a Substrait UDT straight to a bareDataType, so a consumer has no way to mark the resultingFieldfor a downstream reader to recover semantic type lost in that mapping (e.g. thejsonUDT becomes plainUtf8, indistinguishable from a real string column).consume_user_defined_type_metadata, a second, additive extension point onSubstraitConsumerdefaulting toOk(None), and applies its result to theFieldbuilt infrom_substrait_struct_type.Motivation
In dd-source, retriever streams query results over Arrow Flight through DataFusion/Substrait. JSON-typed columns come back as plain
stringinstead ofjson, because Arrow has no native JSON type and nothing marks the column as semantically JSON. The synchronous (non-streaming) path avoids this by reconciling against the query planner's own richer type, but that reconciliation isn't available to a pure Arrow Flight consumer. This extension point lets a consumer (in a follow-up dd-source change) attach metadata analogous to what the existing Trino downstream already does for its own Arrow output, so a Flight client can recover the original type.