ErikBPF opened a new issue, #6102:
URL: https://github.com/apache/datafusion-comet/issues/6102
## Problem
Both TPC correctness jobs key their dataset cache on the workflow file:
```
key: tpch-${{ hashFiles('.github/workflows/pr_build_linux.yml') }} # :705
key: tpcds-${{ hashFiles('.github/workflows/pr_build_linux.yml') }} # :779
```
Any edit to `pr_build_linux.yml` rotates the key and regenerates the SF=1
dataset; that file changed **9× in 7 days**. Today `gh api actions/caches`
shows **10 TPC entries / ~2.6 GiB**, including entries written from throwaway
PR refs that later runs cannot restore — they only evict the shared LRU budget
that protects the ~4.1 GiB `Linux-cargo-debug` cache, whose eviction costs a
**~26-minute cold build**.
## Change
**4a — pin the generators first.** The key cannot be stable while the inputs
are not: `GenTPCHData.scala:125` clones `databricks/tpch-dbgen` HEAD (only one
file is SHA-pinned), and the `tpcds-kit` checkout has no `ref:`
(`pr_build_linux.yml:785-790`).
**4b — stable key + restore-keys + main-only save:**
- key on the pinned generator revisions + Spark version, not the workflow
file;
- `restore-keys: tpch-sf1-` / `tpcds-sf1-`;
- save only from `main`, consistent with the existing large-cache policy.
## Expected gain
~3.5 min per avoided regeneration, and it removes ~2.6 GiB of cache churn.
Direct ≈10 min/72 h; the real value is avoiding the cold builds the churn
causes.
## Verification
- `gh api actions/caches`: TPC entries fall from 10 to ~2; no `Cache not
found` for TPC across consecutive non-database runs.
- A deliberate generator-revision bump still rotates the key.
- TPC jobs still green on a PR and on the queue.
## Risk / rollback
Stale dataset reuse if the generators are unpinned — hence 4a first. Revert
the keys; one regeneration restores the old behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
