stevenzwu commented on PR #11781: URL: https://github.com/apache/iceberg/pull/11781#issuecomment-2578314363
BTW, I like the new direction that @RussellSpitzer outlined. using byte size (instead of number of elements) is more intuitive and easier to calculate a good default to cap memory foot print. E.g., the max read bytes could be set to 1/10 of JVM heap size. 1/10 is just an example. We can estimate a proper default value based on typical file size/memory foot print ratio. there could be an edge condition that a single manifest file is larger than the default value. In this case and if the buffer is empty or less than half full, task should be allowed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
