Skip to content

[GH-3298] Bump GraphFrames from 0.11.0 to 0.12.2 - #3299

Merged
jiayuasu merged 1 commit into
apache:masterfrom
james-willis:jw/graphframes-0.12.2
Aug 29, 2026
Merged

jiayuasu merged 1 commit into
apache:masterfrom
james-willis:jw/graphframes-0.12.2

Conversation

@james-willis

@james-willis james-willis commented Aug 28, 2026 •

Copy link
Copy Markdown
Collaborator

Did you read the Contributor Guide?

Is this PR related to a ticket?

What changes were proposed in this PR?

Bumps <graphframes.version> in the root pom.xml from 0.11.0 to 0.12.2.

Sedona's only GraphFrames usage is GraphFrame(...).connectedComponents in DBSCAN.scala. The one breaking change in the 0.12.x line is the low-level Pregel edge-attribute contract (edge columns must now be requested via requiredEdgeColumns), which Sedona does not use. Artifact coordinates are unchanged and 0.12.2 is published for all three combinations Sedona builds against: graphframes-spark3_2.12, graphframes-spark3_2.13, graphframes-spark4_2.13.

Scope reduced

This PR originally also switched DBSCAN's connected components to randomized_contraction, as proposed in #3298. That part has been dropped, because the algorithm cannot run on Sedona's DBSCAN at all.

RandomizedContraction maps results back with coalesce(col(COMPONENT), col(ID)), where COMPONENT is a Long and ID is the original vertex id. DBSCAN.dbscan always sets id to sha2(to_json(struct("*")), 256) — a hex String — even when the input already has an id column (which is renamed to __id). Spark coerces the two branches to a common type, casts the hash to BIGINT, and fails:

org.apache.spark.SparkNumberFormatException: [CAST_INVALID_INPUT] The value
'36e53dd410ea44fd0f989eac55c1d9f90118dbf637374aa4e8733be40f3128f5' of the type
"STRING" cannot be cast to "BIGINT" because it is malformed. SQLSTATE: 22018

Every DBSCAN case in GeoStatsSuite failed on Spark 4.0.2 with this. Upstream, randomized_contraction is only exercised by TestLDBCCases, whose graphs are explicitly LongType, so non-integral vertex IDs are untested there. Reported upstream; the algorithm switch can be revisited once that is fixed and released.

I have left #3298 open to track the switch. Happy to close it instead if you would rather track only the bump.

How was this patch tested?

Full CI matrix. No source changes — dependency version only.

Did this PR include necessary documentation updates?

  • No, this PR does not affect any public API so no need to change the documentation.

@james-willis
james-willis force-pushed the jw/graphframes-0.12.2 branch from 9693ab8 to 878c432 Compare August 28, 2026 19:35
@james-willis
james-willis marked this pull request as ready for review August 28, 2026 19:44
@james-willis
james-willis requested a review from jiayuasu as a code owner August 28, 2026 19:44
@james-willis
james-willis force-pushed the jw/graphframes-0.12.2 branch 3 times, most recently from b328b4f to 380941e Compare August 28, 2026 20:13
@james-willis james-willis changed the title [GH-3298] Bump GraphFrames to 0.12.2 and use randomized contraction for DBSCAN [GH-3298] Bump GraphFrames from 0.11.0 to 0.12.2 Aug 28, 2026
@james-willis
james-willis requested a review from prantogg August 28, 2026 23:44
@jiayuasu jiayuasu added this to the sedona-2.0.0 milestone Aug 29, 2026
@jiayuasu
jiayuasu merged commit ea6b4dd into apache:master Aug 29, 2026
45 checks passed
@james-willis
james-willis deleted the jw/graphframes-0.12.2 branch October 1, 2026 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bump GraphFrames from 0.11.0 to 0.12.2 and use randomized contraction for DBSCAN

2 participants