This repository was archived by the owner on Mar 6, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Mar 6, 2026. It is now read-only.
feat: ensure Arrow type metadata is consistent between REST and BQ Storage APIs #894
Description
The BQ Storage API annotates certain types with additional information so that they can be disambiguated. For example, to tell GEOGRAPHY from STRING columns.
We don't currently annotate the types we create when data is initially downloaded from the REST API.
If we make these consistent, the user could register Arrow extension types, which can hook into the default pyarrow -> pandas conversions. See discussion on: #848 (comment)
Activity
jimfulton commentedon Sep 1, 2021
I have questions.
Confirming, this needs to be done on the client, right?
Do we know what extension metadata are added by the BigQuery storage API?
Is this documented somewhere? Or do we need to determine it empirically?
jimfulton commentedon Sep 1, 2021
It would be cool if you could attach application-specific metadata to BigQuery columns. :) You could, for example mark a struct as defining a
timedelta. :)jimfulton commentedon Sep 3, 2021
I know folks are busy, so I'm going to assume:
yes.
empirical.
tswast commentedon Sep 3, 2021
There are some docs here: https://cloud.google.com/bigquery/docs/reference/storage#arrow_schema_details
But I think empirical will best help us make sure the specific values.
I consider the metadata from the BQ Storage API to be the canonical version. We should update the "arrow from REST API" logic to match.