ariel-miculas opened a new issue, #23559: URL: https://github.com/apache/datafusion/issues/23559
### Describe the bug Basically in multi level merge: https://github.com/apache/datafusion/blob/b7e7b514e9f93c0a013286511a36e86ffcf2d662/datafusion/physical-plan/src/sorts/multi_level_merge.rs#L456-L465 the reservation is multiplied by buffer_len. This buffer_size is later on used to read the spill files as a stream: https://github.com/apache/datafusion/blob/b7e7b514e9f93c0a013286511a36e86ffcf2d662/datafusion/physical-plan/src/sorts/multi_level_merge.rs#L329-L339 read_spill_as_stream calls spawn_buffered with self.batch_read_buffer_capacity: https://github.com/apache/datafusion/blob/b7e7b514e9f93c0a013286511a36e86ffcf2d662/datafusion/physical-plan/src/spill/spill_manager.rs#L194 The maximum amount of RecordBatches produced by this spawn_buffered call: https://github.com/apache/datafusion/blob/739e1471b262e9e3b2e26089d4678ad4c054c399/datafusion/physical-plan/src/common.rs#L191-L208 is actually `buffer + 1`. Example with buffer = 1: RecordBatch is produced, channel capacity is 1, sender.send succeeds, 1 RecordBatch in memory RecordBatch is produced, channel is full, sender.send blocks, but the RecordBatch has already been produced => 2 RecordBatches in memory This is an issue especially with large Record batches ### To Reproduce _No response_ ### Expected behavior Since buffer needs to be at least one, get a permit before producing the RecordBatch: https://docs.rs/tokio/latest/tokio/sync/mpsc/struct.Permit.html Update all the callers to use buffer + 1 in order to preserve behavior ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
