zhuqi-lucas commented on code in PR #21426:
URL: https://github.com/apache/datafusion/pull/21426#discussion_r3045414206
##########
datafusion/common/src/config.rs:
##########
@@ -557,6 +557,19 @@ config_namespace! {
/// batches and merged.
pub sort_in_place_threshold_bytes: usize, default = 1024 * 1024
+ /// Maximum buffer capacity (in bytes) per partition for BufferExec
+ /// inserted during sort pushdown optimization.
+ ///
+ /// When PushdownSort eliminates a SortExec under
SortPreservingMergeExec,
+ /// a BufferExec is inserted to replace SortExec's buffering role. This
+ /// prevents I/O stalls by allowing the scan to run ahead of the merge.
+ ///
+ /// This uses strictly less memory than the SortExec it replaces (which
+ /// buffers the entire partition). The buffer respects the global
memory
+ /// pool limit. Setting this to a large value is safe — actual memory
+ /// usage is bounded by partition size and global memory limits.
+ pub sort_pushdown_buffer_capacity: usize, default = 1024 * 1024 * 1024
Review Comment:
Good point! A dynamic approach with both a row limit and a memory cap
(whichever is reached first) would adapt better to different row widths.
Created #21440 to track this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
