andygrove commented on issue #2162: URL: https://github.com/apache/datafusion-ballista/issues/2162#issuecomment-5073283881
**Status update: the gap that prompted this is resolved** The benchmark gap that led me to file this (SF1000 TPC-H, Ballista slower than Spark 3.4) is now closed. On current `main`, the comparable 19-query subset runs in about **721s on Ballista vs about 700s on Spark 3.4** on the same cluster shape, down from 1353 vs 845. See the refreshed `docs/source/contributors-guide/benchmarking.md`. What actually closed it was shuffle-write I/O and memory, not the S3 read path: - A dedicated `gp3` EBS PVC for the executor shuffle work-dir. On EBS-only `r6i` nodes the shared root volume bandwidth was the binding constraint under sustained shuffle write (#2171). - A bounded, auto-sized memory pool via `--memory-pool-size` (#2160). - Keeping executors heartbeating while all task slots are busy (#2159). So the specific slowdown that made me wonder about S3 prefetching turned out to be shuffle storage and memory, not Parquet reads from S3. **On the actual question: can Ballista prefetch from S3 like Spark?** Ballista does not have an I/O layer of its own to prefetch in. All Parquet and S3 reads are delegated to DataFusion (`DataSourceExec` / `ParquetSource`) and, below that, to the `object_store` crate's S3 client. Ballista only registers and configures the object store. So "prefetching like Spark" is really a DataFusion-stack question, plus whatever knobs Ballista chooses to expose. For reference, Spark's edge here comes from the S3A connector: fadvise (it starts sequential and switches to random when a reader seeks backwards, which is the Parquet access pattern), `fs.s3a.readahead.range`, and the newer block prefetch in Hadoop 3.4+ (8 MB blocks, background thread pool) that overlaps network I/O with decode. Where the DataFusion stack stands today: - `object_store` already coalesces adjacent column-chunk byte ranges within a row group, so we are not blindly issuing one tiny request per column. - The real prefetch win is upstream: apache/datafusion#18470 proposes overlapping the next row group's I/O with decoding the current one, which is exactly the prefetch Spark benefits from. It is open with a promising POC. When it lands and we bump DataFusion, Ballista gets it for free. DataFusion 54 (what we pin) does not have it yet. Levers available today with no code changes: - `SET datafusion.execution.parquet.metadata_size_hint` to cut footer round trips. - Read concurrency via `datafusion.execution.target_partitions` and file-group assignment. - DataFusion's metadata / statistics / list-file caches, which exist but are not currently wired into Ballista's `RuntimeProducer`. **Recommendation** Since the gap that motivated this is closed, I do not think a Ballista-specific prefetch layer is worth building right now. Better to: 1. Track apache/datafusion#18470 and pick it up on the next DataFusion bump. That is the highest-leverage, lowest-cost path. 2. Revisit a Ballista-side readahead wrapper only if a future workload shows S3 read stalls (idle CPU waiting on I/O) are actually a bottleneck. Measure first. Leaving this open as a low-priority tracking issue for the upstream prefetch work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
