andygrove commented on issue #2168: URL: https://github.com/apache/datafusion-ballista/issues/2168#issuecomment-5071623033
Some data from a run on our SF1000 K8s cluster (AWS EKS, 32 executors × 8 vCPU × 64 GiB memory, `--concurrent-tasks=8`, `--memory-pool-size=48GB`, AQE on, Parquet on S3): - **Q8 standalone**: 47.4 s - **Q8 running after Q1–Q7 (in a full-suite run)**: 228.4 s — **4.8× slower** Same commit (`main` @ `696ca29b`, Ballista 54.0.0, DataFusion 54.1.0), same physical K8s nodes (5 of 6 nodes shared between the two runs with **identical executor placement per node** — ruling out node placement / co-tenant differences at the K8s level). ## The slowdown is concentrated in one stage Comparing per-stage `EXPLAIN ANALYZE` aggregates (all times summed across the 256 tasks in the stage): | stage (op) | metric | standalone | in-suite | Δ | |---|---|--:|--:|--:| | **1 (lineitem scan → hash shuffle write)** | **write_time** | **1694 s** | **8392 s** | **+396 %** | | 2 (part⋈lineitem hash join) | fetch_time | 1263 s | 239 s | −81 % | | 6 | fetch_time | 7235 s | 1281 s | −82 % | | 6 | permit_wait_time | 3874 s | 1446 s | −63 % | | 8 | fetch_time | 509 s | 123 s | −76 % | | 8 | permit_wait_time | 24 179 s | 6 656 s | −72 % | Every stage *except* stage 1 is **faster** in-suite (warm shuffle-client connections after Q1–Q7). Stage 1's inflation swamps those wins. ## Stage 1: same bytes, no spilling, 5× slower writes | stage 1 metric | standalone | in-suite | |---|--:|--:| | output_rows | 6.00 B | 6.00 B | | output_bytes | **313.2 GB** | **313.2 GB** | | bytes_scanned | 85.81 GB | 85.81 GB | | output_batches | 733.9 K | 733.9 K | | repart_time | 61 s | 64 s | | spill_bytes | **0** | **0** | | spill_count | **0** | **0** | | **write_time** | **1694 s** | **8392 s** | Per-task effective throughput: standalone ≈ **7.1 GB/s**, in-suite ≈ **1.4 GB/s**. Same 313 GB of shuffle output, no extra spilling — just 5× slower per byte. ## What's actually going on: kernel page cache vs bandwidth-limited disk 7.1 GB/s per task is far above any local-disk sustained write rate — meaning **the standalone writes never actually reach the disk synchronously**; they're absorbed by the kernel's page cache and `write()` returns immediately (buffered writes on the container overlay FS). Our K8s worker nodes are on AWS Nitro instances with EBS-backed root filesystems (`/dev/nvme0n1p1`). Ballista's executor working dir (`Executor working directory: /tmp/.tmpXXX` from the executor log) is on `/tmp`, which is on the container overlay FS, which is on the root EBS volume. EBS has strict per-instance bandwidth caps (~600 MB/s on typical m5/r5 sizes, shared across all pods on the instance). **In-suite hypothesis:** Q1–Q7 dirty hundreds of GB of pages that the kernel is still flushing to EBS when Q8's stage 1 begins. With page cache already loaded, Q8's stage-1 writes force synchronous flushes at EBS bandwidth (~1.4 GB/s per task = ~few hundred MB/s per pod = matches a per-instance EBS cap split across 4–7 pods per node). Standalone Q8 hits a clean cache and never pays that. ## Ruling out other explanations - **Node placement**: identical, 5/6 nodes shared with same executor distribution. - **Additional spilling**: `spill_bytes = 0` and `spill_count = 0` on every operator in both runs. - **Memory pool exhaustion**: per-task pool peaks (measured via a small `TrackedMemoryPool` wrapper I wrote in #2163) top out at ~1.7 GB / 5.6 GiB limit on Q8. Nowhere near ceiling. Zero residual reservations at pool drop. ## Suggested next diagnostic Run `q8, q1, q2, ..., q7, q8` back-to-back on the same cluster. If the second `q8` is much slower than the first, it confirms the within-run effect on the same executor processes on the same nodes — separating it from any external variability. The broader implication is a benchmarking-setup lesson too: **on EBS-backed nodes without dedicated instance-store or a provisioned-throughput data volume, sustained shuffle-heavy workloads will hit page-cache saturation and drop to EBS bandwidth**. Might be worth documenting the recommended node-storage config for `benchmarks/`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
