Repository navigation
Provide valid extensionUris and extensionUriReferences in Substrait #11545
Description
Describe the bug
The Substrait produced by DF contains SimpleExtensionsDeclarations. See https://substrait.io/extensions/#simple-extensions for definition and https://substrait.io/tutorial/sql_to_substrait/#plans for an example (the example only has scalar functions but idea is the same for types and type variations). However, it does not contain proper SimpleExtensionUris, but rather uses a hardcoded extensionUriReference of max i32.
We should switch to providing proper SimpleExtensionUris with correct references.
To Reproduce
No response
Expected behavior
DF-produced Substrait plans contain valid extensionUris like shown here: https://substrait.io/tutorial/sql_to_substrait/#plans
Additional context
Technically, the SimpleExtensions should be defined in YAML files, like the ones here: https://github.com/substrait-io/substrait/blob/a68c1ac62f92d703da624cb8ac0cef854dd2b35f/extensions. That however makes some things more strict than they are now, and there are open questions in Substrait around how to evolve those files etc. So as the first alternative, we could consider just providing a reference to an imaginary uri rather than pointing to a proper YAML.
Activity
benbellick commentedon Oct 1, 2025
FYI, due to this change in upstream repo (substrait-io/substrait#856), it is extensionUrns, and extensionUrnReferences that need to be added (once substrait is updated).
test: run a DataFusion plan through substrait-java and Spark
test: run a DataFusion plan through substrait-java and Spark
test: run a DataFusion plan through substrait-java and Spark
alexandrefimov commentedon Oct 8, 2026
Confirmed on main
102a16285(55.1.0), withi BIGINT NOT NULL: SQL-producedi + 1andabs(i)round-trip in DataFusion, but Java 0.103.0 rejects them because no URN is registered at anchor -1. A read-only plan loads.The plans contain no URNs and reference anchor
4294967295. Adding the arithmetic URN exposes a second problem: bare namesadd/abs. Usingadd:i64_i64/abs:i64as well lets Java load both, without changing the relation tree.fix: Emit LIKE the way the Substrait extension defines it (apache#25442)
fix: Emit LIKE the way the Substrait extension defines it (apache#25442)
fix: write the grouping set index map as a Substrait switch
test: run a DataFusion plan through substrait-java and Spark