Skip to content

Deprecate support for Flink 1.19, Spark 3.3, 3.4 - #7285

Closed
pan3793 wants to merge 2 commits into
apache:masterfrom
pan3793:deprecate
Closed

Deprecate support for Flink 1.19, Spark 3.3, 3.4#7285
pan3793 wants to merge 2 commits into
apache:masterfrom
pan3793:deprecate

Conversation

@pan3793

@pan3793 pan3793 commented Dec 25, 2025

Copy link
Copy Markdown
Member

Why are the changes needed?

Flink 1.19 and Spark 3.3, 3.4 are EOLed for a long time, and Kyuubi are likely to drop the support for those versions in the future, this PR adds a deprecated warning message before dropping.

How was this patch tested?

Review.

Was this patch authored or co-authored using generative AI tooling?

No.

@codecov-commenter

codecov-commenter commented Dec 25, 2025 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (acdb6a3) to head (569dca4).
⚠️ Report is 36 commits behind head on master.

Files with missing lines Patch % Lines
...rg/apache/kyuubi/engine/spark/SparkSQLEngine.scala 0.00% 2 Missing ⚠️
Additional details and impacted files 
@@          Coverage Diff           @@
##           master   #7285   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         698     698           
  Lines       43634   43636    +2     
  Branches     5892    5893    +1     
======================================
- Misses      43634   43636    +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

aajisaka
aajisaka approved these changes Dec 25, 2025
View reviewed changes

@aajisaka aajisaka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Thank you

@pan3793 pan3793 self-assigned this Dec 25, 2025
@pan3793 pan3793 added this to the v1.11.0 milestone Dec 25, 2025
@pan3793 pan3793 closed this in 675d3e7 Dec 25, 2025
@pan3793

pan3793 commented Dec 25, 2025

Copy link
Copy Markdown
Member Author

thanks all, merged to master/1.11

@maomaodev maomaodev mentioned this pull request May 12, 2026
Closed
1 of 2 tasks
pan3793 added a commit to pan3793/kyuubi that referenced this pull request Aug 13, 2026
@pan3793
[KYUUBI apache#7285] Drop support for Spark 3.3 and 3.4
6b0978e 
### Why are the changes needed?

Spark 3.3 and 3.4 are EOL and were deprecated in KYUUBI apache#7285. Kyuubi 1.13 drops
support for them and the minimum supported Spark version becomes 3.5.

### What changes are proposed?

- Remove the spark-3.3/spark-3.4 Maven profiles and the
  kyuubi-extension-spark-3-3/3-4 modules
- Drop Spark 3.3/3.4 from CI matrices, release scripts, and codecov
- Clean up compatibility shims that only existed for Spark < 3.5
- Update docs to record the removal

### How was this patch tested?

- build/mvn -Pfast,spark-3.5 clean install -DskipTests -pl externals/kyuubi-spark-sql-engine,extensions/spark/kyuubi-spark-connector-hive -am
- build/mvn -Pfast,spark-4.2,scala-2.13 clean install -DskipTests -pl externals/kyuubi-spark-sql-engine,extensions/spark/kyuubi-spark-connector-hive -am
- build/mvn -Pfast,spark-3.5 clean install -DskipTests -pl extensions/spark/kyuubi-spark-authz,extensions/spark/kyuubi-spark-lineage -am
- build/mvn -Pfast,spark-4.2,scala-2.13 clean install -DskipTests -pl extensions/spark/kyuubi-spark-authz,extensions/spark/kyuubi-spark-lineage -am
- Spotless, Scalastyle, and dev/reformat pass

### Was this patch authored or co-authored using generative AI tooling?

Assisted-by: DeepSeek V4 Pro
pan3793 added a commit to pan3793/kyuubi that referenced this pull request Aug 13, 2026
@pan3793
Drop support for Spark 3.3 and 3.4
57589d3 
### Why are the changes needed?

Spark 3.3 and 3.4 are EOL and were deprecated in KYUUBI apache#7285. Kyuubi 1.13 drops
support for them and the minimum supported Spark version becomes 3.5.

### What changes are proposed?

- Remove the spark-3.3/spark-3.4 Maven profiles and the
  kyuubi-extension-spark-3-3/3-4 modules
- Drop Spark 3.3/3.4 from CI matrices, release scripts, and codecov
- Clean up compatibility shims that only existed for Spark < 3.5
- Update docs to record the removal

### How was this patch tested?

- build/mvn -Pfast,spark-3.5 clean install -DskipTests -pl externals/kyuubi-spark-sql-engine,extensions/spark/kyuubi-spark-connector-hive -am
- build/mvn -Pfast,spark-4.2,scala-2.13 clean install -DskipTests -pl externals/kyuubi-spark-sql-engine,extensions/spark/kyuubi-spark-connector-hive -am
- build/mvn -Pfast,spark-3.5 clean install -DskipTests -pl extensions/spark/kyuubi-spark-authz,extensions/spark/kyuubi-spark-lineage -am
- build/mvn -Pfast,spark-4.2,scala-2.13 clean install -DskipTests -pl extensions/spark/kyuubi-spark-authz,extensions/spark/kyuubi-spark-lineage -am
- Spotless, Scalastyle, and dev/reformat pass

### Was this patch authored or co-authored using generative AI tooling?

Assisted-by: DeepSeek V4 Pro
pan3793 added a commit to pan3793/kyuubi that referenced this pull request Aug 13, 2026
@pan3793
Drop support for Spark 3.3 and 3.4
3e1cfd6 
### Why are the changes needed?

Spark 3.3 and 3.4 are EOL and were deprecated in KYUUBI apache#7285. Kyuubi 1.13 drops
support for them, and Spark 3.5 becomes the minimum supported version.

### What changes are proposed?

- Remove the spark-3.3/spark-3.4 Maven profiles and the
  kyuubi-extension-spark-3-3/kyuubi-extension-spark-3-4 modules
- Drop Spark 3.3/3.4 from CI matrices, snapshot publishing, release scripts, and codecov
- Remove compatibility shims that only existed for Spark < 3.5, including reflections,
  version gates, and Python 2/pre-3.8 code
- Update docs to record the removal

### How was this patch tested?

The patch also updates CI itself; after the change, GHA covers:

- default job: Spark 3.5 (Scala 2.12, JDK 17) and Spark 4.0/4.1/4.2 (Scala 2.13, JDK 21)
- binary verification: the engine built with Spark 3.5 runs against Spark 4.0.3/4.1.2/4.2.0
  distributions
- connector cross-version tests: connectors compiled with Spark 3.5 run against Spark
  4.0/4.1/4.2 runtimes
- scala-test: Spark 3.5 on JDK 8 with Scala 2.13

### Was this patch authored or co-authored using generative AI tooling?

Assisted-by: DeepSeek V4 Pro
@pan3793 pan3793 mentioned this pull request Aug 13, 2026
Closed
pan3793 added a commit to pan3793/kyuubi that referenced this pull request Aug 13, 2026
@pan3793
Drop support for Spark 3.3 and 3.4
d64e861 
### Why are the changes needed?

Spark 3.3 and 3.4 are EOL and were deprecated in KYUUBI apache#7285. Kyuubi 1.13 drops
support for them, and Spark 3.5 becomes the minimum supported version.

### What changes are proposed?

- Remove the spark-3.3/spark-3.4 Maven profiles and the
  kyuubi-extension-spark-3-3/kyuubi-extension-spark-3-4 modules
- Drop Spark 3.3/3.4 from CI matrices, snapshot publishing, release scripts, and codecov
- Remove compatibility shims that only existed for Spark < 3.5, including reflections,
  version gates, and Python 2/pre-3.8 code
- Update docs to record the removal

### How was this patch tested?

The patch also updates CI itself; after the change, GHA covers:

- default job: Spark 3.5 (Scala 2.12, JDK 17) and Spark 4.0/4.1/4.2 (Scala 2.13, JDK 21)
- binary verification: the engine built with Spark 3.5 runs against Spark 4.0.3/4.1.2/4.2.0
  distributions
- connector cross-version tests: connectors compiled with Spark 3.5 run against Spark
  4.0/4.1/4.2 runtimes
- scala-test: Spark 3.5 on JDK 8 with Scala 2.13

### Was this patch authored or co-authored using generative AI tooling?

Assisted-by: DeepSeek V4 Pro
pan3793 added a commit to pan3793/kyuubi that referenced this pull request Aug 13, 2026
@pan3793
Drop support for Spark 3.3 and 3.4
13a4769 
Spark 3.3 and 3.4 are EOL and were deprecated in KYUUBI apache#7285. Kyuubi 1.13 drops
support for them, and Spark 3.5 becomes the minimum supported version.

- Remove the spark-3.3/spark-3.4 Maven profiles and the
  kyuubi-extension-spark-3-3/kyuubi-extension-spark-3-4 modules
- Drop Spark 3.3/3.4 from CI matrices, snapshot publishing, release scripts, and codecov
- Remove compatibility shims that only existed for Spark < 3.5, including reflections,
  version gates, and Python 2/pre-3.8 code
- Update docs to record the removal

The patch also updates CI itself; after the change, GHA covers:

- default job: Spark 3.5 (Scala 2.12, JDK 17) and Spark 4.0/4.1/4.2 (Scala 2.13, JDK 21)
- binary verification: the engine built with Spark 3.5 runs against Spark 4.0.3/4.1.2/4.2.0
  distributions
- connector cross-version tests: connectors compiled with Spark 3.5 run against Spark
  4.0/4.1/4.2 runtimes
- scala-test: Spark 3.5 on JDK 8 with Scala 2.13

Assisted-by: DeepSeek V4 Pro
pan3793 added a commit that referenced this pull request Aug 14, 2026
@pan3793
[KYUUBI #7631] Drop support for Spark 3.3 and 3.4
ad817af 
### Why are the changes needed?

Spark 3.3 and 3.4 are EOL and were deprecated in KYUUBI #7285. Kyuubi 1.13 drops support for them, and Spark 3.5 becomes the minimum supported version.

### What changes are proposed?

- Remove the spark-3.3/spark-3.4 Maven profiles and the kyuubi-extension-spark-3-3/kyuubi-extension-spark-3-4 modules
- Drop Spark 3.3/3.4 from CI matrices, snapshot publishing, release scripts, and codecov
- Remove compatibility shims that only existed for Spark < 3.5, including reflections, version gates, and Python 2/pre-3.8 code
- Update docs to record the removal

### How was this patch tested?

The patch also updates CI itself; after the change, GHA covers:

- default job: Spark 3.5 (Scala 2.12, JDK 17) and Spark 4.0/4.1/4.2 (Scala 2.13, JDK 21)
- binary verification: the engine built with Spark 3.5 runs against Spark 4.0.3/4.1.2/4.2.0 distributions
- connector cross-version tests: connectors compiled with Spark 3.5 run against Spark 4.0/4.1/4.2 runtimes
- scala-test: Spark 3.5 on JDK 8 with Scala 2.13

### Was this patch authored or co-authored using generative AI tooling?

Assisted-by: DeepSeek V4 Pro

Closes #7631 from pan3793/kyuubi-drop-spark-3-3-3-4.

Closes #7631

ec36d52 [Cheng Pan] Update snapshot publish matrix to Spark 3.5 and 4.x
f2e6f01 [Cheng Pan] Drop Python 3.11 from Python Client CI matrix
13a4769 [Cheng Pan] Drop support for Spark 3.3 and 3.4

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
@pan3793 pan3793 mentioned this pull request Aug 14, 2026
Closed
pan3793 added a commit that referenced this pull request Aug 17, 2026
@pan3793
[KYUUBI #7633] Drop support for Flink 1.17, 1.18 and 1.19
ddcc31c 
### Why are the changes needed?

Flink 1.17, 1.18 and 1.19 are EOL and were deprecated earlier (KYUUBI #7199, KYUUBI #7285),
with their building support already removed. Kyuubi 1.13 drops the remaining support, making
Flink 1.20 the minimum. Flink 2.0 is EOL, so it is marked deprecated.

### What changes are proposed?

- Remove the flink-1.17/flink-1.18/flink-1.19 Maven profiles and their deprecation warning
- Drop Flink 1.17/1.18/1.19 from the CI matrix
- Clean up stale adaptation code that only existed for Flink < 1.20, including the
  FlinkResultSet reflection shim, the 4-arg DefaultContext.load branch, and >= "1.17" test guards
- Remove the obsolete kyuubi.engine.flink.doAs.generateTokenFile workaround, superseded by
  FLINK-35525 (1.20.0)
- Deprecate Flink 2.0 at engine startup
- Update docs to record the removal and deprecation

### How was this patch tested?

Cross-version tested: engine compiled with Flink 1.20, run against a Flink 2.3.0 distribution;
FlinkOperationSuite and FlinkOperationSuiteOnYarn passed 12/12.

CI coverage: the flink-it matrix keeps Flink 1.20 (JDK 8) plus Flink 2.0/2.1/2.2/2.3
binary verification (JDK 17). Spotless and Scalastyle pass.

### Was this patch authored or co-authored using generative AI tooling?

Assisted-by: DeepSeek V4 Pro

Closes #7633 from pan3793/kyuubi-drop-flink-1-17-1-18-1-19.

Closes #7633

76c03d9 [Cheng Pan] Remove obsolete kyuubi.engine.flink.doAs.generateTokenFile
a33970f [Cheng Pan] Drop support for Flink 1.17, 1.18 and 1.19

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@aajisaka aajisaka aajisaka approved these changes
@cxzl25 cxzl25 cxzl25 approved these changes
@turboFei turboFei turboFei approved these changes
@cfmcgrady cfmcgrady cfmcgrady approved these changes
@SteNicholas SteNicholas SteNicholas approved these changes

Assignees

@pan3793 pan3793

Labels

kind:documentation Documentation is a feature! module:flink module:spark

Projects

None yet

Milestone

v1.11.0

Development

Successfully merging this pull request may close these issues.

7 participants

@pan3793 @codecov-commenter @aajisaka @cxzl25 @turboFei @cfmcgrady @SteNicholas