kaveti commented on PR #17547: URL: https://github.com/apache/iceberg/pull/17547#issuecomment-5241572581
One thing worth flagging: before this change writes went straight into a WriteChannel, so nothing was held in memory. Now anything under the threshold gets fully buffered in an on-heap ByteArrayOutputStream until close(). S3OutputStream avoids this by spilling to a local temp file instead of holding bytes on heap — might be worth doing something similar here so raising the threshold or having lots of concurrent writers doesn't blow up heap usage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
