Repository navigation
[GH-3298] Bump GraphFrames from 0.11.0 to 0.12.2 - #3299
Merged
Merged
Conversation
james-willis
force-pushed
the
jw/graphframes-0.12.2
branch
from
August 28, 2026 19:35
9693ab8 to
878c432
Compare
james-willis
marked this pull request as ready for review
August 28, 2026 19:44
james-willis
force-pushed
the
jw/graphframes-0.12.2
branch
3 times, most recently
from
August 28, 2026 20:13
b328b4f to
380941e
Compare
jiayuasu
approved these changes
Aug 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Did you read the Contributor Guide?
Is this PR related to a ticket?
[GH-XXX] my subject. Closes Bump GraphFrames from 0.11.0 to 0.12.2 and use randomized contraction for DBSCAN #3298What changes were proposed in this PR?
Bumps
<graphframes.version>in the rootpom.xmlfrom 0.11.0 to 0.12.2.Sedona's only GraphFrames usage is
GraphFrame(...).connectedComponentsinDBSCAN.scala. The one breaking change in the 0.12.x line is the low-level Pregel edge-attribute contract (edge columns must now be requested viarequiredEdgeColumns), which Sedona does not use. Artifact coordinates are unchanged and 0.12.2 is published for all three combinations Sedona builds against:graphframes-spark3_2.12,graphframes-spark3_2.13,graphframes-spark4_2.13.Scope reduced
This PR originally also switched DBSCAN's connected components to
randomized_contraction, as proposed in #3298. That part has been dropped, because the algorithm cannot run on Sedona's DBSCAN at all.RandomizedContractionmaps results back withcoalesce(col(COMPONENT), col(ID)), whereCOMPONENTis aLongandIDis the original vertex id.DBSCAN.dbscanalways setsidtosha2(to_json(struct("*")), 256)— a hexString— even when the input already has anidcolumn (which is renamed to__id). Spark coerces the two branches to a common type, casts the hash toBIGINT, and fails:Every DBSCAN case in
GeoStatsSuitefailed on Spark 4.0.2 with this. Upstream,randomized_contractionis only exercised byTestLDBCCases, whose graphs are explicitlyLongType, so non-integral vertex IDs are untested there. Reported upstream; the algorithm switch can be revisited once that is fixed and released.I have left #3298 open to track the switch. Happy to close it instead if you would rather track only the bump.
How was this patch tested?
Full CI matrix. No source changes — dependency version only.
Did this PR include necessary documentation updates?