Repository navigation
Conversation
Spark RoundCeil takes a decimal and a constant Int32 scale. Implement the two-arg form on SparkCeil with result types from RoundBase::dataType, and error on overflow instead of wrapping (matches Spark NUMERIC_VALUE_OUT_OF_RANGE). Enable the previously commented PySpark cases in ceil.slt. Positive-scale `ceil(...)` still hits the SQL planner's CeilFloorKind::Scale path; use the `ceiling` alias for that case (documented in the .slt). Closes apache#21560.
Member
|
Closing due to
It's impossible to contribute responsibly in such PR volume, please don't do automated contribution in this project. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Spark
ceil/ceilingaccepts a second scale argument (RoundCeil). Only the one-arg form was implemented indatafusion-spark, and the matching cases inceil.sltwere commented out with a TODO pointing at this issue.Earlier attempts (#21562, #21710, #24643) did not land. This PR focuses on the decimal path Spark declares for
RoundCeil, and treats overflow as an error instead of wrapping (Spark raisesNUMERIC_VALUE_OUT_OF_RANGEfor extreme negative scales).What changes are included in this PR?
SparkCeilacceptsceil(decimal, Int32 scale)with a constant scale.RoundBase::dataTypeviareturn_field_from_args.ceil.slt. The positive-scale case uses theceilingalias because sqlparser still routesceil(x, 3)throughCeilFloorKind::Scale, which the planner rejects before UDF resolution (same as today for FLOOR).Non-decimal inputs with a scale are left for a follow-up (Spark casts them to decimal first).
What is the testing strategy for this PR?
ceil.rsfor the PySpark cases, rounding toward +∞, no-op scales, carry, precision bounds, and overflow errors.cargo test -p datafusion-spark --lib function::math::ceilcargo test -p datafusion-sqllogictest --test sqllogictests -- spark/math/ceilAre there any user-facing changes?
Spark sessions gain working
ceil/ceilingwith a decimal scale argument. No change to DataFusion's built-in one-argceil.AI assistance
Assisted by AI tooling for drafting and iterating; I reviewed the Spark
RoundCeil/RoundBasesemantics, the overflow cases called out on #24643, and the test results before opening this PR.