Repository navigation
Conversation
…spile tests with branch-4.3 The SPARK-58626 cherry-pick brought in three tests that depend on SPARK-27052 (Python UDFs inside higher-order function lambdas), which is master-only. On this branch CheckAnalysis unconditionally rejects a resolved lambda containing a Python UDF, so the tests fail at analysis whenever they actually run; the intermittent CI green was dev/run-tests skipping unchanged modules on some pushes. There is no user-facing regression: the transpiler leaves lambda calls to the Python UDF, so such a query fails at analysis identically with transpilation off. Update the tests to assert the analysis error, which pins "not lowered" on this branch -- a lowered call would analyze and run. Co-authored-by: Holden Karau <holden@pigscanfly.ca> Co-authored-by: Holden Karau <holden.karau@snowflake.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…n tests pin CheckAnalysis runs before the optimizer's ConvertToCatalyst rule, so the analysis rejection these tests assert would fire even with the rule's lambda guard broken: the rejection pins the user-facing behavior, not "not lowered". Say so in the comments and point at the ConvertToCatalystSuite rule-level test of the same name for the guard. (PR review feedback.) Co-authored-by: Holden Karau <holden@pigscanfly.ca> Co-authored-by: Holden Karau <holden.karau@snowflake.com> Co-authored-by: Cursor <cursoragent@cursor.com>
HyukjinKwon
approved these changes
Sep 29, 2026
holdenk
marked this pull request as ready for review
September 29, 2026 17:54
asf-gitbox-commits
pushed a commit
that referenced
this pull request
Sep 29, 2026
…n transpile tests with branch-4.3 Adjusts the three tests SPARK-58626 added that assert master's behavior for a transpiled UDF call inside a higher-order function's lambda, so they assert branch-4.3's behavior instead: analysis rejects the query with `UNSUPPORTED_FEATURE.LAMBDA_FUNCTION_WITH_PYTHON_UDF` — the same error the query gets with transpilation off. - `test_udf_transpile_declines_inside_a_higher_order_function` and the lambda half of `test_udf_transpile_position_can_rule_out_sharing` (`test_udf_transpile_unit.py`) - `TranspiledUDFParameterSuite.keeps the Python UDF inside a higher-order function's lambda` Expecting the rejection is itself the assertion that matters on this branch: a call lowered inside the lambda would analyze and run, so pinning the error pins "not lowered". The SPARK-58626 cherry-pick (#58189) broke the branch-4.3 build. The three tests expect the query to *run* — which it does on master because SPARK-27052 ("Support Python UDFs inside higher-order function lambdas") rewrites the lambda to apply the UDF over the whole array. SPARK-27052 is master-only; on branch-4.3 `CheckAnalysis` rejects any Python UDF inside a higher-order function's lambda, transpiled or not, deterministically. The branch build looked intermittent only because `dev/run-tests` skips unchanged modules on some pushes ("There are no modules to test, exiting without testing"), so those pushes went green without running these tests. The alternative fix is backporting SPARK-27052 to branch-4.3. That is a feature backport (new config, analyzer/optimizer rules) and a release-management call, not something to fold into a build fix. Two adjacent notes, not addressed here: - branch-4.x is NOT affected: SPARK-27052 and its follow-ups (SPARK-58695, SPARK-58728) are all on branch-4.x, and the two test files plus `CheckAnalysis.scala` are byte-identical to master there. Only branch-4.3 lacks the lambda support. - The branch-4.3 build has a second, unrelated red job: `UdfSerializationSuite` (Decimal serialVersionUID audit, SPARK-59708 area). - Pre-existing stale comments reference the master-only `ExtractPythonUDFFromLambda` rule on this branch (`Optimizer.scala` and `ConvertToCatalystSuite.scala`); left alone here as out of scope. No this is test-only (and transpilation has not yet been released either way). User-facing behavior on branch-4.3 is unchanged for this shape: the query fails with the same error whether transpilation is on or off. On branch-4.3 (Java 21): - `python -u -m unittest pyspark.sql.tests.test_udf_transpile_unit` — 82 tests, OK - `build/sbt -Phive 'sql/testOnly *TranspiledUDFParameterSuite'` — 10 tests, OK - `RUN_HYPOTHESIS=1 RUN_HYPOTHESIS_MAX_EXAMPLES=200 python -u -m unittest pyspark.sql.tests.test_udf_transpile_hypothesis` — OK Generated-by: Cursor (kimi-k3) Closes #59128 from holdenk/SPARK-58626-branch-4.3-investigate. Authored-by: Holden Karau <holden.karau@snowflake.com> Co-Authored-by: Holden Karau <holden@pigscanfly.ca> Signed-off-by: Holden Karau <holden.karau@snowflake.com>
Contributor
Author
|
Merge Summary:
Posted by |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Adjusts the three tests SPARK-58626 added that assert master's behavior for a transpiled UDF call inside a higher-order function's lambda, so they assert branch-4.3's behavior instead: analysis rejects the query with
UNSUPPORTED_FEATURE.LAMBDA_FUNCTION_WITH_PYTHON_UDF— the same error the query gets with transpilation off.test_udf_transpile_declines_inside_a_higher_order_functionand the lambda half oftest_udf_transpile_position_can_rule_out_sharing(test_udf_transpile_unit.py)TranspiledUDFParameterSuite.keeps the Python UDF inside a higher-order function's lambdaExpecting the rejection is itself the assertion that matters on this branch: a call lowered inside the lambda would analyze and run, so pinning the error pins "not lowered".
Why are the changes needed?
The SPARK-58626 cherry-pick (#58189) broke the branch-4.3 build. The three tests expect the query to run — which it does on master because SPARK-27052 ("Support Python UDFs inside higher-order function lambdas") rewrites the lambda to apply the UDF over the whole array. SPARK-27052 is master-only; on branch-4.3
CheckAnalysisrejects any Python UDF inside a higher-order function's lambda, transpiled or not, deterministically.The branch build looked intermittent only because
dev/run-testsskips unchanged modules on some pushes ("There are no modules to test, exiting without testing"), so those pushes went green without running these tests.The alternative fix is backporting SPARK-27052 to branch-4.3. That is a feature backport (new config, analyzer/optimizer rules) and a release-management call, not something to fold into a build fix.
Two adjacent notes, not addressed here:
CheckAnalysis.scalaare byte-identical to master there. Only branch-4.3 lacks the lambda support.UdfSerializationSuite(Decimal serialVersionUID audit, SPARK-59708 area).ExtractPythonUDFFromLambdarule on this branch (Optimizer.scalaandConvertToCatalystSuite.scala); left alone here as out of scope.Does this PR introduce any user-facing change?
No this is test-only (and transpilation has not yet been released either way). User-facing behavior on branch-4.3 is unchanged for this shape: the query fails with the same error whether transpilation is on or off.
How was this patch tested?
On branch-4.3 (Java 21):
python -u -m unittest pyspark.sql.tests.test_udf_transpile_unit— 82 tests, OKbuild/sbt -Phive 'sql/testOnly *TranspiledUDFParameterSuite'— 10 tests, OKRUN_HYPOTHESIS=1 RUN_HYPOTHESIS_MAX_EXAMPLES=200 python -u -m unittest pyspark.sql.tests.test_udf_transpile_hypothesis— OKWas this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (kimi-k3)