andygrove commented on PR #6940: URL: https://github.com/apache/datafusion-comet/pull/6940#issuecomment-6268007130
Thanks @viirya. The `memory_management.md` bullet now mentions the 1 MiB output-stream buffer in the reservation, and says S3A's own upload buffers (`fs.s3a.fast.upload.buffer`) sit outside Comet's pool. (65e2bfdfe1) On validation: I have no Docker on my machine, so I cannot run `ParquetWriteWithMagicCommitterSuite` against MinIO here. I am running it as committed on Spark 3.5 and 4.1 on a Docker host, and dispatching the manual `Spark SQL Writer Tests` workflow for 3.5 and 4.1, and I will post the results here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
