manuzhang opened a new issue, #14165: URL: https://github.com/apache/iceberg/issues/14165
### Apache Iceberg version main (development) ### Query engine Flink ### Please describe the bug 🐞 ``` 2025-09-22T15:52:48.5606763Z TestFlinkTableSinkExtended > testHashDistributeMode() > isStreamingJob=false, useV2Sink=true FAILED 2025-09-22T15:52:48.5607878Z java.lang.RuntimeException: Failed to collect table result 2025-09-22T15:52:48.5608737Z at org.apache.iceberg.flink.SqlBase.sql(SqlBase.java:48) 2025-09-22T15:52:48.5616266Z at org.apache.iceberg.flink.TestFlinkTableSinkExtended.testHashDistributeMode(TestFlinkTableSinkExtended.java:266) 2025-09-22T15:52:48.5618326Z 2025-09-22T15:52:48.5618790Z Caused by: 2025-09-22T15:52:48.5619579Z org.apache.flink.table.api.TableException: Failed to wait job finish 2025-09-22T15:52:48.5621331Z at app//org.apache.flink.table.api.internal.InsertResultProvider.hasNext(InsertResultProvider.java:85) 2025-09-22T15:52:48.5627627Z at app//org.apache.flink.table.api.internal.InsertResultProvider$Iterator.hasNext(InsertResultProvider.java:112) 2025-09-22T15:52:48.5628955Z at app//org.apache.iceberg.relocated.com.google.common.collect.Iterators.addAll(Iterators.java:369) 2025-09-22T15:52:48.5630093Z at app//org.apache.iceberg.relocated.com.google.common.collect.Lists.newArrayList(Lists.java:154) 2025-09-22T15:52:48.5630985Z at app//org.apache.iceberg.flink.SqlBase.sql(SqlBase.java:46) 2025-09-22T15:52:48.5631478Z ... 1 more 2025-09-22T15:52:48.5631674Z 2025-09-22T15:52:48.5631786Z Caused by: 2025-09-22T15:52:48.5632579Z java.util.concurrent.ExecutionException: org.apache.flink.runtime.client.JobExecutionException: Job execution failed. 2025-09-22T15:52:48.5633813Z at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) 2025-09-22T15:52:48.5634781Z at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2005) 2025-09-22T15:52:48.5636078Z at org.apache.flink.table.api.internal.InsertResultProvider.hasNext(InsertResultProvider.java:83) 2025-09-22T15:52:48.5636817Z ... 5 more 2025-09-22T15:52:48.5637020Z 2025-09-22T15:52:48.5637146Z Caused by: 2025-09-22T15:52:48.5637698Z org.apache.flink.runtime.client.JobExecutionException: Job execution failed. 2025-09-22T15:52:48.5638953Z at app//org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144) 2025-09-22T15:52:48.5640249Z at app//org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:140) 2025-09-22T15:52:48.5641552Z at [email protected]/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642) 2025-09-22T15:52:48.5642644Z at [email protected]/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) 2025-09-22T15:52:48.5643917Z at [email protected]/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2079) 2025-09-22T15:52:48.5645718Z at org.apache.flink.runtime.rpc.pekko.PekkoInvocationHandler.lambda$invokeRpc$1(PekkoInvocationHandler.java:268) 2025-09-22T15:52:48.5647142Z at [email protected]/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) 2025-09-22T15:52:48.5648493Z at [email protected]/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) 2025-09-22T15:52:48.5655671Z at [email protected]/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) 2025-09-22T15:52:48.5657100Z at [email protected]/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2079) 2025-09-22T15:52:48.5658222Z at app//org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1317) 2025-09-22T15:52:48.5659765Z at app//org.apache.flink.runtime.concurrent.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$1(ClassLoadingUtils.java:93) 2025-09-22T15:52:48.5661581Z at app//org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68) 2025-09-22T15:52:48.5663326Z at app//org.apache.flink.runtime.concurrent.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92) 2025-09-22T15:52:48.5665260Z at [email protected]/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) 2025-09-22T15:52:48.5666613Z at [email protected]/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) 2025-09-22T15:52:48.5667921Z at [email protected]/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) 2025-09-22T15:52:48.5669176Z at [email protected]/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2079) 2025-09-22T15:52:48.5681304Z at org.apache.flink.runtime.concurrent.pekko.ScalaFutureUtils$1.onComplete(ScalaFutureUtils.java:47) 2025-09-22T15:52:48.5682061Z at org.apache.pekko.dispatch.OnComplete.internal(Future.scala:310) 2025-09-22T15:52:48.5682576Z at org.apache.pekko.dispatch.OnComplete.internal(Future.scala:307) 2025-09-22T15:52:48.5683029Z at org.apache.pekko.dispatch.japi$CallbackBridge.apply(Future.scala:234) 2025-09-22T15:52:48.5683488Z at org.apache.pekko.dispatch.japi$CallbackBridge.apply(Future.scala:231) 2025-09-22T15:52:48.5683931Z at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74) 2025-09-22T15:52:48.5684567Z at org.apache.flink.runtime.concurrent.pekko.ScalaFutureUtils$DirectExecutionContext.execute(ScalaFutureUtils.java:65) 2025-09-22T15:52:48.5685490Z at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:82) 2025-09-22T15:52:48.5686050Z at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:298) 2025-09-22T15:52:48.5686858Z at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:298) 2025-09-22T15:52:48.5687730Z at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:298) 2025-09-22T15:52:48.5688246Z at org.apache.pekko.pattern.PromiseActorRef.$bang(AskSupport.scala:624) 2025-09-22T15:52:48.5688854Z at org.apache.pekko.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:33) 2025-09-22T15:52:48.5689425Z at scala.concurrent.Future.$anonfun$andThen$1(Future.scala:536) 2025-09-22T15:52:48.5689850Z at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:42) 2025-09-22T15:52:48.5690486Z at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74) 2025-09-22T15:52:48.5691037Z at org.apache.pekko.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:73) 2025-09-22T15:52:48.5691695Z at org.apache.pekko.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:110) 2025-09-22T15:52:48.5692269Z at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) 2025-09-22T15:52:48.5692868Z at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85) 2025-09-22T15:52:48.5693530Z at org.apache.pekko.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:110) 2025-09-22T15:52:48.5694089Z at org.apache.pekko.dispatch.TaskInvocation.run(AbstractDispatcher.scala:59) 2025-09-22T15:52:48.5694773Z at org.apache.pekko.dispatch.ForkJoinExecutorConfigurator$PekkoForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:61) 2025-09-22T15:52:48.5695766Z at [email protected]/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) 2025-09-22T15:52:48.5696342Z at [email protected]/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) 2025-09-22T15:52:48.5696917Z at [email protected]/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) 2025-09-22T15:52:48.5697441Z at [email protected]/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) 2025-09-22T15:52:48.5698152Z at [email protected]/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) 2025-09-22T15:52:48.5698839Z 2025-09-22T15:52:48.5698980Z Caused by: 2025-09-22T15:52:48.5699634Z org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy 2025-09-22T15:52:48.5701005Z at app//org.apache.flink.runtime.executiongraph.failover.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:219) 2025-09-22T15:52:48.5702085Z at app//org.apache.flink.runtime.executiongraph.failover.ExecutionFailureHandler.handleFailureAndReport(ExecutionFailureHandler.java:166) 2025-09-22T15:52:48.5703147Z at app//org.apache.flink.runtime.executiongraph.failover.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:121) 2025-09-22T15:52:48.5704049Z at app//org.apache.flink.runtime.scheduler.DefaultScheduler.recordTaskFailure(DefaultScheduler.java:294) 2025-09-22T15:52:48.5704785Z at app//org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:285) 2025-09-22T15:52:48.5706095Z at app//org.apache.flink.runtime.scheduler.adaptivebatch.AdaptiveBatchScheduler.handleTaskFailure(AdaptiveBatchScheduler.java:461) 2025-09-22T15:52:48.5706926Z at app//org.apache.flink.runtime.scheduler.DefaultScheduler.onTaskFailed(DefaultScheduler.java:278) 2025-09-22T15:52:48.5707740Z at app//org.apache.flink.runtime.scheduler.adaptivebatch.AdaptiveBatchScheduler.onTaskFailed(AdaptiveBatchScheduler.java:453) 2025-09-22T15:52:48.5708771Z at app//org.apache.flink.runtime.scheduler.SchedulerBase.onTaskExecutionStateUpdate(SchedulerBase.java:814) 2025-09-22T15:52:48.5709534Z at app//org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:791) 2025-09-22T15:52:48.5710249Z at app//org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:83) 2025-09-22T15:52:48.5711238Z at app//org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:516) 2025-09-22T15:52:48.5712632Z at [email protected]/java.lang.reflect.Method.invoke(Method.java:566) 2025-09-22T15:52:48.5713652Z at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.lambda$handleRpcInvocation$1(PekkoRpcActor.java:318) 2025-09-22T15:52:48.5714679Z at app//org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83) 2025-09-22T15:52:48.5715871Z at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcInvocation(PekkoRpcActor.java:316) 2025-09-22T15:52:48.5716539Z at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcMessage(PekkoRpcActor.java:229) 2025-09-22T15:52:48.5717241Z at org.apache.flink.runtime.rpc.pekko.FencedPekkoRpcActor.handleRpcMessage(FencedPekkoRpcActor.java:88) 2025-09-22T15:52:48.5717925Z at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleMessage(PekkoRpcActor.java:174) 2025-09-22T15:52:48.5718525Z at org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:33) 2025-09-22T15:52:48.5719043Z at org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:29) 2025-09-22T15:52:48.5719511Z at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) 2025-09-22T15:52:48.5720073Z at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126) 2025-09-22T15:52:48.5720955Z at org.apache.pekko.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:29) 2025-09-22T15:52:48.5721840Z at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175) 2025-09-22T15:52:48.5722317Z at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) 2025-09-22T15:52:48.5722760Z at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) 2025-09-22T15:52:48.5723198Z at org.apache.pekko.actor.Actor.aroundReceive(Actor.scala:547) 2025-09-22T15:52:48.5723609Z at org.apache.pekko.actor.Actor.aroundReceive$(Actor.scala:545) 2025-09-22T15:52:48.5724084Z at org.apache.pekko.actor.AbstractActor.aroundReceive(AbstractActor.scala:229) 2025-09-22T15:52:48.5724589Z at org.apache.pekko.actor.ActorCell.receiveMessage(ActorCell.scala:590) 2025-09-22T15:52:48.5725317Z at org.apache.pekko.actor.ActorCell.invoke(ActorCell.scala:557) 2025-09-22T15:52:48.5725782Z at org.apache.pekko.dispatch.Mailbox.processMailbox(Mailbox.scala:272) 2025-09-22T15:52:48.5726216Z at org.apache.pekko.dispatch.Mailbox.run(Mailbox.scala:233) 2025-09-22T15:52:48.5726627Z at org.apache.pekko.dispatch.Mailbox.exec(Mailbox.scala:245) 2025-09-22T15:52:48.5726945Z ... 5 more 2025-09-22T15:52:48.5727097Z 2025-09-22T15:52:48.5727185Z Caused by: 2025-09-22T15:52:48.5729081Z java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@716e6a55[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@9dd4642[Wrapped task = org.apache.iceberg.util.LockManagers$InMemoryLockManager$$Lambda$1303/0x0000000100d52040@28156521]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@5b60484e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] 2025-09-22T15:52:48.5731369Z at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055) 2025-09-22T15:52:48.5732044Z at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825) 2025-09-22T15:52:48.5732860Z at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340) 2025-09-22T15:52:48.5733704Z at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:632) 2025-09-22T15:52:48.5734539Z at java.base/java.util.concurrent.Executors$DelegatedScheduledExecutorService.scheduleAtFixedRate(Executors.java:785) 2025-09-22T15:52:48.5735471Z at org.apache.iceberg.util.LockManagers$InMemoryLockManager.acquireOnce(LockManagers.java:220) 2025-09-22T15:52:48.5736107Z at org.apache.iceberg.util.LockManagers$InMemoryLockManager.lambda$acquire$1(LockManagers.java:249) 2025-09-22T15:52:48.5736668Z at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413) 2025-09-22T15:52:48.5737142Z at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219) 2025-09-22T15:52:48.5737579Z at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203) 2025-09-22T15:52:48.5737961Z at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196) 2025-09-22T15:52:48.5738445Z at org.apache.iceberg.util.LockManagers$InMemoryLockManager.acquire(LockManagers.java:249) 2025-09-22T15:52:48.5739155Z at org.apache.iceberg.hadoop.HadoopTableOperations.renameToFinal(HadoopTableOperations.java:363) 2025-09-22T15:52:48.5740284Z at org.apache.iceberg.hadoop.HadoopTableOperations.commit(HadoopTableOperations.java:162) 2025-09-22T15:52:48.5741345Z at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:467) 2025-09-22T15:52:48.5741946Z at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413) 2025-09-22T15:52:48.5742427Z at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219) 2025-09-22T15:52:48.5742893Z at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203) 2025-09-22T15:52:48.5743291Z at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196) 2025-09-22T15:52:48.5743746Z at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:439) 2025-09-22T15:52:48.5744317Z at org.apache.iceberg.flink.sink.IcebergCommitter.commitOperation(IcebergCommitter.java:298) 2025-09-22T15:52:48.5745241Z at org.apache.iceberg.flink.sink.IcebergCommitter.commitDeltaTxn(IcebergCommitter.java:253) 2025-09-22T15:52:48.5745913Z at org.apache.iceberg.flink.sink.IcebergCommitter.commitPendingResult(IcebergCommitter.java:205) 2025-09-22T15:52:48.5746678Z at org.apache.iceberg.flink.sink.IcebergCommitter.commitPendingRequests(IcebergCommitter.java:175) 2025-09-22T15:52:48.5747649Z at org.apache.iceberg.flink.sink.IcebergCommitter.commit(IcebergCommitter.java:136) 2025-09-22T15:52:48.5749080Z at org.apache.flink.streaming.runtime.operators.sink.committables.CheckpointCommittableManagerImpl.commit(CheckpointCommittableManagerImpl.java:150) 2025-09-22T15:52:48.5751069Z at org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.commitAndEmit(CommitterOperator.java:181) 2025-09-22T15:52:48.5752619Z at org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.commitAndEmitCheckpoints(CommitterOperator.java:174) 2025-09-22T15:52:48.5754114Z at org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.endInput(CommitterOperator.java:157) 2025-09-22T15:52:48.5755754Z at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.endOperatorInput(StreamOperatorWrapper.java:96) 2025-09-22T15:52:48.5757487Z at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.endInput(RegularOperatorChain.java:97) 2025-09-22T15:52:48.5758311Z at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:68) 2025-09-22T15:52:48.5759035Z at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:637) 2025-09-22T15:52:48.5759737Z at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231) 2025-09-22T15:52:48.5760444Z at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:972) 2025-09-22T15:52:48.5761038Z at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:916) 2025-09-22T15:52:48.5761626Z at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:963) 2025-09-22T15:52:48.5762198Z at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:942) 2025-09-22T15:52:48.5762676Z at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:756) 2025-09-22T15:52:48.5763115Z at org.apache.flink.runtime.taskmanager.Task.run(Task.java:568) 2025-09-22T15:52:48.5763508Z at java.base/java.lang.Thread.run(Thread.java:829) ``` ### Willingness to contribute - [ ] I can contribute a fix for this bug independently - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
