Skip to content

GH-1302: Add support for the arrow.json extension type - #1303

Open
ddupg wants to merge 3 commits into
apache:mainfrom
ddupg:feat/arrow-json
Open

ddupg wants to merge 3 commits into
apache:mainfrom
ddupg:feat/arrow-json

Conversation

@ddupg

@ddupg ddupg commented Sep 20, 2026 •

Copy link
Copy Markdown
Member

Adds JsonType and JsonVector for the canonical arrow.json extension, backed by Utf8, LargeUtf8, or Utf8View.

Includes metadata handling, registration, and vector transfers that preserve the extension type. IPC loading and unloading now inspect the underlying vector when counting view buffers, so JSON backed by Utf8View can round-trip correctly.

Closes #1302.

Change-Id: I94ff9448ea6a2ac4089489c49d2bc61f1312a16f
@github-actions

This comment has been minimized.

Change-Id: I2f961fbc983bd29bffd2b4280021f7b696e3ad50
Change-Id: If49bdef2c665414b786ea0f2b60d0336b3839b9b
@lidavidm lidavidm added the enhancement PRs that add or improve features. label Sep 20, 2026
@ddupg
ddupg marked this pull request as ready for review September 21, 2026 07:06
@github-actions github-actions Bot added this to the 20.0.0 milestone Sep 21, 2026
// Only view storage has variadic buffers.
long variadicBufferLayoutCount = 0;
if (vector instanceof BaseVariableWidthViewVector) {
if (storageVector instanceof BaseVariableWidthViewVector) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This now expects a variadicBufferCounts entry for extension vectors whose storage is a view vector, but c/StructVectorUnloader (which feeds this loader in Data.importIntoVectorSchemaRoot) still checks vector instanceof BaseVariableWidthViewVector on the wrapper, so it never emits one.

Importing a C Data batch with an arrow.json or OpaqueType column on Utf8View/BinaryView storage now fails with IlllegalStateException: No variadicBufferCounts available for BaseVariableWidthViewVector when all values are inlined (<= 12 bytes) or the batch is empty. The same OpaqueVector(Utf8View) case loads on main, so this is a regression for existing extension types, not only for the new one. It also affects ArrowArrayStreamReader.loadNextBatch and the dataset NativeScanner.

Could you apply the same unwrapping in StructVectorUnloader, and add a C Data round-trip test for an extension type on view storage? RoundtripTest.testExtensionTypeVector only covers UuidType.

long variadicBufferLayoutCount = 0;
if (vector instanceof BaseVariableWidthViewVector) {
if (storageVector instanceof BaseVariableWidthViewVector) {
if (variadicBufferCounts.hasNext()) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same problem in the export direction: this now adds a variadic count for extension vectors on view storage, but c/StructVectorLoader.loadBuffers still checks the wrapper and never consumes it.

Data.exportVectorSchemaRoot (and ArrayStreamExporter) on a root with a JsonVector(Utf8View) column then throws IllegalArgumentException: not all nodes, buffers and variadicBufferCounts were consumed. It fails even with zero variadic buffers, a case that succeeds on main.

StructVectorLoader needs the same change as VectorLoader. Since all four loader/unloader classes have to agree on which vectors carry a variadic count, a single shared helper for the unwrap (for example a static storage-vector accessor on ExtensionTypeVector) would keep them in sync.

* instance. Write UTF-8 JSON through {@link #getUnderlyingVector()}; values are not parsed or
* validated.
*/
public class JsonVector extends ExtensionTypeVector<FieldVector>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ExtensionTypeVector does not delegate exportCDataBuffers / getExportedCDataBufferCount to the storage vector, so JsonVector inherits the FieldVector defaults. For Utf8View storage that means Data.exportVector exports 3 buffers (validity, views, data) where ViewVarCharVector exports 4, i.e. the trailing variadic-sizes buffer is missing.

On the import side, BufferImportTypeVisitor.visitVariableWidthView takes the last buffer as the sizes buffer and computes the data buffer count as buffers.length - 3, so a consumer would misread the data buffer as sizes.

Delegating both methods to getUnderlyingVector() would fix it, ideally in ExtensionTypeVector so OpaqueVector on view storage gets it too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement PRs that add or improve features.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for the arrow.json extension type

3 participants