aokolnychyi commented on code in PR #8755:
URL: https://github.com/apache/iceberg/pull/8755#discussion_r1350591568
##########
core/src/main/java/org/apache/iceberg/SystemConfigs.java:
##########
@@ -42,6 +42,13 @@ private SystemConfigs() {}
Math.max(2, Runtime.getRuntime().availableProcessors()),
Integer::parseUnsignedInt);
+ public static final ConfigEntry<Integer> DELETE_WORKER_THREAD_POOL_SIZE =
+ new ConfigEntry<>(
+ "iceberg.worker.delete-num-threads",
+ "ICEBERG_WORKER_DELETE_NUM_THREADS",
+ 4 * Runtime.getRuntime().availableProcessors(),
Review Comment:
This value may sound ridiculous but here is my thought process: there is one
such thread pool per JVM, each core in an executor can get a data task that may
need to load 1 to many delete files, these tasks are I/O intensive. This value
essentially means we can try to load 4 delete files concurrently per each data
task. The cache is also blocking to prevent reading the same files twice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
