pepijnve commented on PR #23522: URL: https://github.com/apache/datafusion/pull/23522#issuecomment-4982566176
> I'm worried about dropping the FIFO semantics (even if they were broken). Consider a plan like `Projection(inner=Repartition(inner=Sort, preserve_order=true)))`: if `RepartitionExec` does _not_ actually preserve order under spilling then the sort order will be lost and the results will be wrong. I'm probably missing a bit too much context. Could you help me define the exact order guarantee that's being promised by the spill pool? Single writer remains FIFO, but in the presence of multiple concurrent writers, what order can you actually guarantee for the reader? The best you can do I think is guarantee that the relative order of the batches per writer is retained. Is that sufficient? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
