andygrove opened a new issue, #4102: URL: https://github.com/apache/datafusion-comet/issues/4102
## Describe the proposed change The macOS CI workflow (`.github/workflows/pr_build_macos.yml`) currently runs: - 3 profiles (Spark 3.4 / JDK 11 / Scala 2.12, Spark 3.5 / JDK 17 / Scala 2.13, Spark 4.0 / JDK 17 / Scala 2.13) - 7 suites (fuzz, shuffle, parquet, csv, exec, expressions, sql) That is 21 jobs per PR. macOS runners on GitHub Actions cost roughly 2x Linux minutes, so this is the most expensive workflow per job. ## Rationale The macOS workflow exists primarily to catch platform-specific issues on Apple Silicon: native library loading, FFI, threading, and shuffle. Those concerns are largely independent of Spark/Scala/JDK version. The full Spark/Java/Scala matrix coverage already happens on Linux (`pr_build_linux.yml`). Running 3 Spark profiles × 7 suites on macOS is duplicative for everything that is not platform-sensitive. ## Proposed change Two options, in order of aggressiveness: **Option A (recommended): one profile, all suites — 7 jobs** Keep only the newest supported profile (Spark 4.0 / JDK 17 / Scala 2.13). Run all 7 suites. Saves 14 jobs. **Option B: keep all profiles, reduce to platform-sensitive suites — 9 jobs** Drop suites that are not platform-sensitive (`csv`, `expressions`, `sql`, `exec`). Keep `shuffle`, `parquet`, `fuzz` across all 3 profiles. Saves 12 jobs. I lean toward Option A: platform issues rarely depend on Spark version, and we can always add a second profile back if a regression slips through. ## Additional context Part of a broader CI cleanup ahead of adding Spark 4.0.1 and 4.1.1 to the test matrix. Companion issue: #4101 (Iceberg JDK halving). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
