CI: Select PR matrices incrementally

[ajantha-bhat]

Mailing list discussion: https://lists.apache.org/thread/36vxlql61gojbg639c86mnz78n57kvgm

Summary

• add a shared PR CI planner that uses changed files, full-ci labels, and global build/workflow changes to select PR matrices
• keep full Java/Spark/Flink/Hive/Kafka/Delta/CVE coverage for main, release branches, tags, full-ci, global build/workflow changes, and unknown changed code paths
• use Java 17 as the primary JVM for ordinary PRs, while full CI paths keep Java 17 and Java 21
• preserve the existing Gradle cache policy from Build: Designate a single Gradle cache writer across CI workflows #16356: only java-ci.yml build-checks (17) writes cache on main; all other Gradle jobs are read-only

Selective PR behavior

• Spark-only changes run Spark CI, not Flink/Hive/Kafka CI
• spark/v<known-version>/** selects only that Spark version; unknown Spark version paths force full CI
• flink/v<known-version>/** selects only that Flink version; unknown Flink version paths force full CI
• API/Core/Data/file-format changes run Java checks plus latest Spark and latest Flink canaries
• Hive, Kafka Connect, and Delta jobs run only when their owned paths change or when full CI is selected

CVE behavior

• CVE artifact-level selection is intentionally not included in this PR
• CVE-relevant PR changes still run the CVE workflow, but the workflow uses the full CVE distribution matrix when it runs
• docs-only PRs skip CVE, and global/unknown/full-ci paths force CVE along with the rest of the full matrix

Maintainability guardrails

• the planner fails closed: unknown non-doc/code paths force full CI instead of silently skipping coverage
• Spark/Flink versions are read from gradle.properties; versioned paths are parsed generically rather than listed one by one
• the Java CI plan job runs .github/scripts/plan-pr-ci-test.sh to validate important routing cases
• the planner writes a GitHub step summary showing whether full CI was selected and which workflows will run

Validation

• YAML parse for edited workflows
• git diff --check origin/main...HEAD
• bash -n .github/scripts/plan-pr-ci.sh
• bash -n .github/scripts/plan-pr-ci-test.sh
• bash .github/scripts/plan-pr-ci-test.sh
• synthetic planner checks for Spark 4.1, Flink 2.0, core/file-format canaries, global build changes, docs-only changes, unknown paths, and old Spark version paths
• ./gradlew -DallModules build -x test -x javadoc -x integrationTest

Rebased on latest origin/main and pushed as single commit ba49482fac.

[kevinjqliu]

thanks for the PR @ajantha-bhat

i've done some work with gradle cache recently (#16356) and made it so that there's only 1 canonical writer. Before this change, I saw that we were constantly getting thrashed by multiple cache writers and cache utilization was really low.

Theres also a security component to this, we should only write to cache on push to main branch. We should not allow PRs to write to the shared cache since that's a cache poisoning vulnerability.

I'm curious how this change effects when to save to cache and how its reused

[ajantha-bhat]

Thanks @kevinjqliu, good catch.

I updated the PR to preserve #16356's cache model. The selective PR matrix planner now stands on its own, and this PR no longer adds custom cache artifact merge/store actions or additional Gradle cache writers.

Behavior after the update:

• PRs: Gradle cache restore only, no writes
• main: only java-ci.yml build-checks (17) writes, same as Build: Designate a single Gradle cache writer across CI workflows #16356
• release branches/tags: read-only cache, full coverage
• ordinary PRs: selected matrices with Java 17
• full-ci/global changes: full matrices with Java 17 and 21

So the runner reduction now comes from selective matrices, while the shared Gradle cache writer remains unchanged.

[ajantha-bhat]

@kevinjqliu, @nastra, @stevenzwu : Can we please review this?
IMO we need to reduce the GH runner usage as much as possible.

[ajantha-bhat]

ping (maybe @kevinjqliu)

[kevinjqliu]

Thanks for working on this. I agree with the goal of reducing GH Actions usage.

My main concern is maintainability. This adds a fairly large hand-maintained CI planner with path regexes, workflow routing, version selection, and CVE artifact mapping all in Bash. As the repo changes, it seems easy to miss updating this and accidentally skip important CI coverage.

So +1 on the direction, but I’m not sure this implementation is safe to maintain as-is. I think we should first agree on how we want to keep this selection logic understandable and hard to accidentally under-run.

[ajantha-bhat]

Thanks @kevinjqliu , I agree with the maintainability concern. But the win is still worth pursuing given the GH Actions pressure and PR wall-time cost, but only if the planner is conservative and easy to reason about.

I revised the PR with that in mind:

• the planner now fails closed: unknown non-doc/code paths, unknown Spark/Flink version paths, missing changed-file detection, and global build/workflow changes force full CI
• Spark/Flink version selection is now generic from spark/v<version>/ and flink/v<version>/, checked against knownSparkVersions / knownFlinkVersions in gradle.properties
• I removed fine-grained CVE artifact selection from this PR; CVE-relevant changes now run the full CVE matrix
• added planner tests for the important routing cases, including docs-only, core/file-format canaries, Spark/Flink version paths, global changes, unknown paths, and old Spark version paths
• the planner writes a CI summary so the selected plan is visible in each run

The intended invariant is: stale planner logic may run more CI, but should not silently under-run coverage.

Open questions:

• Is there a specific remaining part of the planner that you think is still too much duplicated build knowledge?
• Do you have another shape in mind that would give us this level of PR matrix reduction while keeping the logic closer to Gradle?
• Would you prefer splitting this into smaller PRs? My slight preference is to keep it together because the pieces are interdependent and the total change would be roughly the same, but I’m happy to split if that makes review easier.

[Baunsgaard]

To post here as well, I made a smaller PR that simplifies the maintainability and reduce CI jobs #16951 .

[ajantha-bhat]

keeping this PR as draft as I have created the smaller PRs from this

#16945 CI: Use one Java version for PR checks
#16946 CI: Select Spark PR matrix by changed version
#16947 CI: Select Flink PR matrix by changed version