rdblue commented on PR #7844: URL: https://github.com/apache/iceberg/pull/7844#issuecomment-1861228366
This class cannot use a blocking queue with the worker pool, so I'm -1 on this change. The problem is that planning uses a shared threadpool. Using a blocking queue would cause tasks to stall, which would then tie up the threads in the shared pool and cause all planning to halt. If you want to limit memory consumption here, then you need to do the following: 1. Add a new `BlockingParallelIterable` 2. Use a blocking queue in `BlockingParallelIterable` 3. Use an isolated threadpool in `BlockingParallelIterable` that is not shared with other planning tasks Once that new variant is in, we can look at how to use it from the scan API. Feel free to contact me to review this, since this is a part of the code where bad changes can cause a lot of trouble! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
