lurnagao-dahua commented on code in PR #10661:
URL: https://github.com/apache/iceberg/pull/10661#discussion_r1671665042
##########
mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -144,21 +147,32 @@ public List<InputSplit> getSplits(JobContext context) {
InputFormatConfig.InMemoryDataModel model =
conf.getEnum(
InputFormatConfig.IN_MEMORY_DATA_MODEL,
InputFormatConfig.InMemoryDataModel.GENERIC);
- try (CloseableIterable<CombinedScanTask> tasksIterable = scan.planTasks())
{
- Table serializableTable = SerializableTable.copyOf(table);
- tasksIterable.forEach(
- task -> {
- if (applyResidual
- && (model == InputFormatConfig.InMemoryDataModel.HIVE
- || model == InputFormatConfig.InMemoryDataModel.PIG)) {
- // TODO: We do not support residual evaluation for HIVE and PIG
in memory data model
- // yet
- checkResiduals(task);
- }
- splits.add(new IcebergSplit(serializableTable, conf, task));
- });
- } catch (IOException e) {
- throw new UncheckedIOException(String.format("Failed to close table
scan: %s", scan), e);
+ final ExecutorService workerPool =
+ ThreadPools.newWorkerPool(
+ "iceberg-plan-worker-pool",
+ conf.getInt(
+ SystemConfigs.WORKER_THREAD_POOL_SIZE.propertyKey(),
+ ThreadPools.WORKER_THREAD_POOL_SIZE));
+ try {
+ scan = scan.planWith(workerPool);
Review Comment:
> This test can also run in non-kerberos env, so i think you can use
`org.apache.hadoop.security.UserGroupInformation` to login or get ugi info,
then you don't need to launch the MiniKdc service.
In the local file system, process users are used to read and write files,
not ugi users. File permissions are determined by the permissions of the Linux
system, so how to verify this issue?
Moreover, work_pool will shut down after execution and cannot obtain the
actual ugi in the thread,and the scope of the `planInputSplits `method is
private.
Can you provide further guidance on how to write this unit test case?
Looking forward to your reply very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
