andygrove opened a new pull request, #7066: URL: https://github.com/apache/datafusion-comet/pull/7066
## Which issue does this PR close? Closes #7060. ## Rationale for this change Every native plan built its own `RuntimeEnv`, and with it a `DiskManager` in `DiskManagerMode::Directories` over the executor's local disks. DataFusion 55.2 creates that mode's `datafusion-*` directory in every configured directory when the `DiskManager` is built and removes it recursively when it is dropped. Each plan therefore paid a `stat` and a `mkdir` per local disk at creation and a recursive delete at release, whether or not it ever spilled. #7060 measured these syscalls at 4.7% of task and Tokio thread samples for a small-task workload (470 map tasks) on macOS with one local directory. The number of calls grows with the number of local disks. ## What changes are included in this PR? - `shared_disk_manager()` in `jni_api.rs` keeps one `DiskManager` per `(local_dirs, max_temp_directory_size)` for the process. `prepare_datafusion_session_context` passes it to the `RuntimeEnvBuilder` instead of a `DiskManagerBuilder`, so the directories are created once per executor. Spill files are still deleted when their last reference is dropped. - `spark.comet.maxTempDirectorySize` is now one limit shared by every native plan in the executor, not one per plan. This is a behavior change: the limit is more constraining for an executor that spills heavily from several concurrent plans. The config doc and the "Limiting Spill Disk Usage" section of the memory tuning guide say so. ## How are these changes tested? New unit test `plans_share_a_disk_manager_per_set_of_local_dirs` checks that the same directories and limit return the same `DiskManager`, that the temp directory is created once, and that a different limit gets a different manager. `execute_plan_drains_a_plan` still passes, and `cargo clippy -D warnings` and prettier are clean. I have not re-run the profile from the issue, and I have not run the Spark SQL suite. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
