kevinjqliu opened a new issue, #16397: URL: https://github.com/apache/iceberg/issues/16397
### Context `spark-ci` dominates the overall time CI takes to run. Based on [most recent stats](https://github.com/apache/iceberg/actions/metrics/performance), it averages 1h5m while the next slowest ci job (`flink-ci`) averages around only 21m. This means PRs must wait at least 1h for `spark-ci` to complete before merging. A couple of things we can do to improve the overall time it takes to run `spark-ci` 1. fix the gradle cache. reduce time to download dependencies. fixed in https://github.com/apache/iceberg/pull/16356 2. parallelize execution. enabling `-DtestParallelism=auto` for all tests, following `flink-ci`. And split `spark-ci` into 2 concurrent jobs. Fixed in https://github.com/apache/iceberg/pull/16357. This reduced Spark CI walltime from ~84 min to ~58 min (−31%) ### Ask One more potential fix is to reduce the runtime of the tests inside `spark-ci`. I've pulled data on how long each test class takes to run. See https://gist.github.com/kevinjqliu/86827bedf9d02adad36881f476484208#top-25--core-spark for more details. Some of the tests are inherently slow, but exacerbated by parameterized matrix that dramatically multiples the number of runs. Is it worth to dig into these tests and see if there are any optimizations? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
