unikdahal commented on PR #6773: URL: https://github.com/apache/datafusion-comet/pull/6773#issuecomment-6114804344
Thanks @andygrove , this is a really nice piece of work. I spent a while on the close/reopen path and it holds up well: held rows get flushed, the writer properties carry over when a partition reopens, file names stay unique through the shared counter, cleanup still sees the reopened files, and the output order stays deterministic. There's one thing I think needs fixing before merge. `reserve()` now treats any `try_resize()` error as the pool saying no, so a real exception coming back from Spark can get swallowed. I have posted the details inline above. The rest of my comments are small and non-blocking. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
