wenzhenghu opened a new issue, #61099: URL: https://github.com/apache/doris/issues/61099
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version >= 3.1 ### What's Wrong? Found one Java-level deadlock: ============================= "catalog-refresh-timer-pool-0": waiting to lock monitor 0x00007eee184120f0 (object 0x00007ef82c3ab300, a java.util.concurrent.ConcurrentHashMap$Node), which is held by "thrift-server-pool-6557" "thrift-server-pool-6557": waiting to lock monitor 0x00007ef1cc01d500 (object 0x00007ef8212ea650, a org.apache.doris.datasource.paimon.PaimonHMSExternalCatalog), which is held by "catalog-refresh-timer-pool-0" Java stack information for the threads listed above: =================================================== "catalog-refresh-timer-pool-0": ``` at java.util.concurrent.ConcurrentHashMap.computeIfPresent([email protected]/ConcurrentHashMap.java:1819) - waiting to lock <0x00007ef82c3ab300> (a java.util.concurrent.ConcurrentHashMap$Node) at com.github.benmanes.caffeine.cache.BoundedLocalCache.remove(BoundedLocalCache.java:2164) at com.github.benmanes.caffeine.cache.LocalManualCache.invalidate(LocalManualCache.java:140) at org.apache.doris.datasource.paimon.PaimonMetadataCache$$Lambda$338/0x0000000801476250.accept(Unknown Source) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept([email protected]/ForEachOps.java:183) at java.util.stream.ReferencePipeline$2$1.accept([email protected]/ReferencePipeline.java:179) at com.github.benmanes.caffeine.cache.BoundedLocalCache$KeySpliterator.lambda$forEachRemaining$0(BoundedLocalCache.java:2920) at com.github.benmanes.caffeine.cache.BoundedLocalCache$KeySpliterator$$Lambda$329/0x0000000801472648.accept(Unknown Source) at java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining([email protected]/ConcurrentHashMap.java:3612) at com.github.benmanes.caffeine.cache.BoundedLocalCache$KeySpliterator.forEachRemaining(BoundedLocalCache.java:2923) at java.util.stream.AbstractPipeline.copyInto([email protected]/AbstractPipeline.java:509) at java.util.stream.AbstractPipeline.wrapAndCopyInto([email protected]/AbstractPipeline.java:499) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential([email protected]/ForEachOps.java:150) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential([email protected]/ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate([email protected]/AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach([email protected]/ReferencePipeline.java:596) at org.apache.doris.datasource.paimon.PaimonMetadataCache.invalidateCatalogCache(PaimonMetadataCache.java:135) at org.apache.doris.datasource.paimon.PaimonMetadataCacheMgr.invalidateCatalogCache(PaimonMetadataCacheMgr.java:41) at org.apache.doris.datasource.ExternalMetaCacheMgr.invalidateCatalogCache(ExternalMetaCacheMgr.java:290) at org.apache.doris.datasource.ExternalCatalog.refreshOnlyCatalogCache(ExternalCatalog.java:605) at org.apache.doris.datasource.ExternalCatalog.resetToUninitialized(ExternalCatalog.java:585) - locked <0x00007ef8212ea650> (a org.apache.doris.datasource.paimon.PaimonHMSExternalCatalog) at org.apache.doris.catalog.RefreshManager.refreshCatalogInternal(RefreshManager.java:75) at org.apache.doris.catalog.RefreshManager.handleRefreshCatalog(RefreshManager.java:58) at org.apache.doris.catalog.RefreshManager$RefreshTask.run(RefreshManager.java:291) at java.util.concurrent.Executors$RunnableAdapter.call([email protected]/Executors.java:539) at java.util.concurrent.FutureTask.runAndReset([email protected]/FutureTask.java:305) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run([email protected]/ScheduledThreadPoolExecutor.java:305) at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1136) at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635) at java.lang.Thread.run([email protected]/Thread.java:833) ``` "thrift-server-pool-6557": ``` at org.apache.doris.datasource.ExternalCatalog.makeSureInitialized(ExternalCatalog.java:307) - waiting to lock <0x00007ef8212ea650> (a org.apache.doris.datasource.paimon.PaimonHMSExternalCatalog) at org.apache.doris.datasource.ExternalCatalog.getSchema(ExternalCatalog.java:610) at org.apache.doris.datasource.ExternalSchemaCache.loadSchema(ExternalSchemaCache.java:78) at org.apache.doris.datasource.ExternalSchemaCache$$Lambda$3306/0x000000080235dc30.load(Unknown Source) at com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$2(LocalLoadingCache.java:145) at com.github.benmanes.caffeine.cache.LocalLoadingCache$$Lambda$205/0x00000008010641c0.apply(Unknown Source) at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2406) at com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$2972/0x0000000802288a50.apply(Unknown Source) at java.util.concurrent.ConcurrentHashMap.compute([email protected]/ConcurrentHashMap.java:1916) - locked <0x00007ef82c3be920> (a java.util.concurrent.ConcurrentHashMap$ReservationNode) at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2404) at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2387) at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108) at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:56) at org.apache.doris.datasource.ExternalSchemaCache.getSchemaValue(ExternalSchemaCache.java:89) at org.apache.doris.datasource.paimon.PaimonMetadataCache.getPaimonSchemaCacheValue(PaimonMetadataCache.java:90) at org.apache.doris.datasource.paimon.PaimonMetadataCache.loadSnapshot(PaimonMetadataCache.java:71) at org.apache.doris.datasource.paimon.PaimonMetadataCache.lambda$new$0(PaimonMetadataCache.java:63) at org.apache.doris.datasource.paimon.PaimonMetadataCache$$Lambda$210/0x000000080106c638.load(Unknown Source) at com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$2(LocalLoadingCache.java:145) at com.github.benmanes.caffeine.cache.LocalLoadingCache$$Lambda$205/0x00000008010641c0.apply(Unknown Source) at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139) at com.github.benmanes.caffeine.cache.LocalCache$$Lambda$3268/0x00000008022fc208.apply(Unknown Source) at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2406) at com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$2972/0x0000000802288a50.apply(Unknown Source) at java.util.concurrent.ConcurrentHashMap.compute([email protected]/ConcurrentHashMap.java:1955) - locked <0x00007ef82c3ab300> (a java.util.concurrent.ConcurrentHashMap$Node) at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2404) at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2387) at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108) at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:56) at org.apache.doris.datasource.paimon.PaimonMetadataCache.getPaimonSnapshot(PaimonMetadataCache.java:155) at org.apache.doris.datasource.paimon.PaimonExternalTable.getPaimonSnapshotCacheValue(PaimonExternalTable.java:147) at org.apache.doris.datasource.paimon.PaimonExternalTable.loadSnapshot(PaimonExternalTable.java:263) at org.apache.doris.nereids.StatementContext.loadSnapshots(StatementContext.java:698) at org.apache.doris.nereids.rules.analysis.BindRelation.getLogicalPlan(BindRelation.java:390) at org.apache.doris.nereids.rules.analysis.BindRelation.bind(BindRelation.java:186) at org.apache.doris.nereids.rules.analysis.BindRelation.doBindRelation(BindRelation.java:145) at org.apache.doris.nereids.rules.analysis.BindRelation.lambda$build$0(BindRelation.java:121) at org.apache.doris.nereids.rules.analysis.BindRelation$$Lambda$1000/0x00000008018860d0.apply(Unknown Source) at org.apache.doris.nereids.pattern.PatternMatcher$1.transform(PatternMatcher.java:92) at org.apache.doris.nereids.jobs.rewrite.PlanTreeRewriteJob.rewrite(PlanTreeRewriteJob.java:57) at org.apache.doris.nereids.jobs.rewrite.PlanTreeRewriteBottomUpJob.rewriteThis(PlanTreeRewriteBottomUpJob.java:91) at org.apache.doris.nereids.jobs.rewrite.PlanTreeRewriteBottomUpJob.execute(PlanTreeRewriteBottomUpJob.java:75) at org.apache.doris.nereids.jobs.scheduler.SimpleJobScheduler.executeJobPool(SimpleJobScheduler.java:44) at org.apache.doris.nereids.jobs.rewrite.RootPlanTreeRewriteJob.execute(RootPlanTreeRewriteJob.java:66) at org.apache.doris.nereids.jobs.executor.AbstractBatchJobExecutor.execute(AbstractBatchJobExecutor.java:139) at org.apache.doris.nereids.jobs.executor.Analyzer.analyze(Analyzer.java:84) at org.apache.doris.nereids.NereidsPlanner.lambda$analyze$4(NereidsPlanner.java:377) at org.apache.doris.nereids.NereidsPlanner$$Lambda$983/0x0000000801870678.run(Unknown Source) at org.apache.doris.nereids.NereidsPlanner.keepOrShowPlanProcess(NereidsPlanner.java:920) at org.apache.doris.nereids.NereidsPlanner.analyze(NereidsPlanner.java:377) at org.apache.doris.nereids.trees.plans.commands.insert.FastInsertIntoValuesPlanner.analyze(FastInsertIntoValuesPlanner.java:63) at org.apache.doris.nereids.NereidsPlanner.planWithoutLock(NereidsPlanner.java:253) at org.apache.doris.nereids.NereidsPlanner.planWithLock(NereidsPlanner.java:227) at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:149) at org.apache.doris.nereids.trees.plans.commands.insert.InsertIntoTableCommand.planInsertExecutor(InsertIntoTableCommand.java:404) at org.apache.doris.nereids.trees.plans.commands.insert.InsertIntoTableCommand.initPlanOnce(InsertIntoTableCommand.java:267) at org.apache.doris.nereids.trees.plans.commands.insert.InsertIntoTableCommand.initPlan(InsertIntoTableCommand.java:185) at org.apache.doris.nereids.trees.plans.commands.insert.InsertIntoTableCommand.initPlan(InsertIntoTableCommand.java:154) at org.apache.doris.nereids.trees.plans.commands.insert.InsertIntoTableCommand.runInternal(InsertIntoTableCommand.java:409) at org.apache.doris.nereids.trees.plans.commands.insert.InsertIntoTableCommand.run(InsertIntoTableCommand.java:144) at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:790) at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:631) at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:593) .... ``` Found 1 deadlock. ### What You Expected? fix it ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
