ranflarion commented on PR #25763: URL: https://github.com/apache/datafusion/pull/25763#issuecomment-6373403520
> `MemoryBufferedStream::new` in `datafusion/physical-plan/src/buffer.rs` (behind `BufferExec`) still drives its input from `SpawnedTask::spawn` Switched it to `spawn_reclaimable`. The new `memory_gets_released_if_stream_drops_while_input_is_pending` buffers four messages, leaves the producer parked on an input that never ends, drops the stream and checks the pool is back to 0 right away, and it fails with `spawn`. The existing `memory_gets_released_if_stream_drops` couldn't catch this, because its input finishes and the task has already exited when the stream drops. `MaterializedCteExec` (#25580) arrived with the merge of main and has the same shape, each buffer task owning a body partition's stream and a spillable reservation, so it's switched too, with a `test_drop_cancel`. So is `collect_partitioned`, whose tasks each own a partition's stream, with `test_collect_partitioned_drop_cancel`. That leaves one plain spawn that owns a plan stream, the write path's demux task, which is dropped only together with the whole query, and I haven't checked its destructors for code that has to run inside its task, so I'd keep it for a follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
