andygrove opened a new issue, #7091: URL: https://github.com/apache/datafusion-comet/issues/7091
### What is the problem the feature request solves? When a native plan's spill files reach `spark.comet.maxTempDirectorySize`, the task fails with DataFusion's message: ``` The used disk space during the spilling process has exceeded the allowable limit of 100.0 GB. Please try increasing the config: `datafusion.runtime.max_temp_directory_size`. ``` A Comet user can't act on that advice. The limit Comet applies is `spark.comet.maxTempDirectorySize`, to each native plan separately. `spark.comet.datafusion.*` keys only reach DataFusion when the testing-only `spark.comet.exec.respectDataFusionConfigs` is on, and even then they are applied as session options, while DataFusion 55.2's `ConfigOptions` has no `runtime` namespace. Raised by @viirya in https://github.com/apache/datafusion-comet/pull/7066#discussion_r4300374988. ### Describe the potential solution When a native error carries this message, add a hint that names `spark.comet.maxTempDirectorySize` and says the limit applies to each native plan, for example in `From<DataFusionError> for CometError` (`native/jni-bridge/src/errors.rs`). DataFusion raises it as a plain `std::io::Error` from `FileSpillWriter::write`, so matching the message text may be the only hook. ### Additional context The message comes from `FileSpillWriter::write` in `datafusion-execution` 55.2 (`disk_manager.rs`). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
