viirya commented on PR #6883:
URL:
https://github.com/apache/datafusion-comet/pull/6883#issuecomment-6272507688
Thanks @andygrove.
1. No, it isn't meant to become its own module. The sources are now in
`spark/src/main/scala/org/apache/comet/local` and the suite in
`spark/src/test/scala/org/apache/comet/local` (b82f38783), which drops the
build-helper source roots, the extra scalastyle execution, the spotless include
and the `compute-changes.py` and `check-ci-config.py` entries.
2. I measured `collect()` of an 8-slice range with a wide projection at
`local[8]` on an M4 Max, each mode in its own JVM, median of 5 runs after 2
warmups, in forward and reverse mode order (ms):
| Query | Spark | Comet | Comet local |
|---|---:|---:|---:|
| 5M rows × 8 longs | 718 / 675 | 660 / 721 | 441 / 379 |
| 1M rows × 32 longs | 429 / 434 | 455 / 425 | 296 / 289 |
Local execution is about 35–45% faster than ordinary Comet here, which is
the encoding, compression and decoding of task results that the handoff skips.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
