Skip to content

[AMORO-3909][CI] Split workflows by module and compatibility matrix - #4391

Draft
xxubai wants to merge 5 commits into
masterfrom
split-ci-matrix
Draft

xxubai wants to merge 5 commits into
masterfrom
split-ci-matrix

Conversation

@xxubai

@xxubai xxubai commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Why

The current Hadoop 2 and Hadoop 3 workflows build most modules together across Spark and JDK combinations. This duplicates work, makes failures difficult to attribute, and encourages packaging commands to exclude unrelated modules when the full reactor becomes too expensive.

What changed

  • Replace the broad core workflows with dedicated checks for Maven validation, formats, Flink/Spark/Trino mixed-format connectors, AMS, Optimizer, Dist, frontend, and site.
  • Use representative compatibility combinations for pull requests, pushes, and manually dispatched runs; remove scheduled full-matrix builds.
  • Use JDK 11 by default and keep JDK 17 compatibility coverage in AMS. Trino remains on JDK 17 because the module compiles with Java 17 source and target levels.
  • Keep Hadoop 2/3 compatibility coverage on connectors and formats while keeping AMS, Optimizer, and Dist on Hadoop 3.
  • Use positive Maven module selection with -pl ... -am, plus stable gate jobs when path filters skip a build.
  • Always build and validate the assembled distribution, including AMS, optimizer jobs, dashboard assets, shell syntax, and the absence of the Hadoop 2 shaded runtime.
  • Preserve the existing Deps, OpenAPI, PR labeler, MCP, Helm, site deployment, Docker, take, and stale workflows.

Validation

  • actionlint for all new and modified workflows
  • YAML parsing, shell syntax checks, and git diff --check
  • Maven project and profile validation
  • Flink and Spark package verification
  • Clean Dist package and archive verification with JDK 11
  • Frontend production build
  • Amoro site and documentation Hugo builds

Frontend lint and type checking remain non-blocking because the current master baseline already contains lint and TypeScript errors; the production build remains required.

Closes #3909

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement]: Split GitHub CI workflows by module and compatibility matrix

1 participant