aokolnychyi commented on code in PR #8755:
URL: https://github.com/apache/iceberg/pull/8755#discussion_r1350591568
##########
core/src/main/java/org/apache/iceberg/SystemConfigs.java:
##########
@@ -42,6 +42,13 @@ private SystemConfigs() {}
Math.max(2, Runtime.getRuntime().availableProcessors()),
Integer::parseUnsignedInt);
+ public static final ConfigEntry<Integer> DELETE_WORKER_THREAD_POOL_SIZE =
+ new ConfigEntry<>(
+ "iceberg.worker.delete-num-threads",
+ "ICEBERG_WORKER_DELETE_NUM_THREADS",
+ 4 * Runtime.getRuntime().availableProcessors(),
Review Comment:
This value may sound ridiculous but here is my thought process: there is one
such thread pool per JVM, each core in an executor can get a data task that may
need to load 1 to many delete files, these tasks are I/O intensive. This value
essentially means we can try to load 4 delete files concurrently per each data
task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
