pepijnve commented on PR #23522: URL: https://github.com/apache/datafusion/pull/23522#issuecomment-4982191269
I've updated the documentation and code comments for correctness. I've deemphasised FIFO semantics quite a bit. The existing documentation stated that this was always guaranteed, but that was not actually the case. Prior to this PR the following chain of events could happen: - Writer A starts a batch push of `B1`, takes `current_write_file` `F1` - Writer B starts a batch push `B2`, `current_write_file` is `None` so it starts a new file `F2` - Writer B finishes its batch push, and sets `current_write_file` to `F2` - Writer A finished its batch push, and sets `current_write_file` to `F1` - Writer B pushes another batch `B3` which is written `current_write_file` which is `F1` The reader will observe batches in the order `B1, B3, B2`. This MR assumes that we can take this even further, since the FIFO guarantee already doesn't hold when multiple writers are present. Would be good to get some feedback from @adriangb indeed since the non-FIFO behaviour was already present in the first commit of this code unless I'm completely misreading it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
