This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 91d81c833f66 [SPARK-57075][INFRA] Share precompile Coursier cache with
host-runner SBT jobs
91d81c833f66 is described below
commit 91d81c833f66a4f382a870159a5b5b6a2a4ebfa5
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Fri May 29 09:32:47 2026 +0800
[SPARK-57075][INFRA] Share precompile Coursier cache with host-runner SBT
jobs
### What changes were proposed in this pull request?
Add `precompile-coursier-<hash>` and `precompile-coursier-` as restore-key
fallbacks on `Cache Coursier local repository` for the four host-runner SBT
jobs: `build` (Scala tests), `tpcds-1g`, `docker-integration-tests`,
`k8s-integration-tests`. The primary key and existing prefix fallback are
unchanged — the new entries are pure fallback.
### Why are the changes needed?
The `precompile` job already resolves all dependencies and writes them to
`~/.cache/coursier`, but it saves under the key prefix `precompile-coursier-`,
while the downstream test jobs read from
`<matrix.java>-<matrix.hadoop>-coursier-`, `tpcds-coursier-`, etc. So on cold
caches (new branch, modified `pom.xml` / `plugins.sbt`), the downstream jobs
re-download dependencies that the precompile job already resolved minutes
earlier in the same workflow.
### Does this PR introduce _any_ user-facing change?
No. CI-only.
### How was this patch tested?
Verified on https://github.com/zhengruifeng/spark/actions/runs/26564295518
(commit `919909f5cb9`): 12/12 host-runner SBT jobs restored a Coursier cache
without re-downloading from Maven Central. 9 of the 12 fell back to the new
`precompile-coursier-<hash>` cache; the other 3 found their own per-job cache
from a prior run.
Container jobs (`pyspark`, `sparkr`, `lint`, `docs`) are excluded — their
Coursier cache step is a no-op due to a host↔container `$HOME` path mismatch
(host writes `/home/runner/.cache/coursier`, container looks at
`/github/home/.cache/coursier`). Independent of this PR; can be addressed in a
follow-up.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.7)
Closes #56118 from zhengruifeng/share-precompile-coursier-cache-dev6.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 299e51784b97c1ca99d2af6373c5c50e5bd4b183)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
.github/workflows/build_and_test.yml | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index d848aed4c55a..fa18ff93b71c 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -382,6 +382,8 @@ jobs:
key: ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-${{
hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
${{ matrix.java }}-${{ matrix.hadoop }}-coursier-
+ precompile-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ precompile-coursier-
- name: Free up disk space
run: |
if [ -f ./dev/free_disk_space ]; then
@@ -1401,6 +1403,8 @@ jobs:
key: tpcds-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
tpcds-coursier-
+ precompile-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ precompile-coursier-
- name: Install Java ${{ inputs.java }}
uses: actions/setup-java@v5
with:
@@ -1519,6 +1523,8 @@ jobs:
key: docker-integration-coursier-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
restore-keys: |
docker-integration-coursier-
+ precompile-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ precompile-coursier-
- name: Install Java ${{ inputs.java }}
uses: actions/setup-java@v5
with:
@@ -1605,6 +1611,8 @@ jobs:
key: k8s-integration-coursier-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
restore-keys: |
k8s-integration-coursier-
+ precompile-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt')
}}
+ precompile-coursier-
- name: Free up disk space
run: |
if [ -f ./dev/free_disk_space ]; then
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
