[doris] branch branch-1.2-lts updated: [branch1.2][fix](catalog) disable hdfs fs cache to avoid to many fs cache in memory (#21276)

[yiguolei]

This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
     new 7a808893ea [branch1.2][fix](catalog) disable hdfs fs cache to avoid to 
many fs cache in memory (#21276)
7a808893ea is described below

commit 7a808893eadf0e0725bf545af07a7d0e5b2f9953
Author: Mingyu Chen <morning...@163.com>
AuthorDate: Wed Jun 28 15:03:21 2023 +0800

    [branch1.2][fix](catalog) disable hdfs fs cache to avoid to many fs cache 
in memory (#21276)
---
 .../main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java | 3 +++
 1 file changed, 3 insertions(+)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java
 
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java
index 8c266168a0..8b7d7f83c6 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java
@@ -256,6 +256,9 @@ public class HiveMetaStoreCache {
             // Otherwise, getSplits() may throw exception: "Not a file xxx"
             // 
https://blog.actorsfit.com/a?ID=00550-ce56ec63-1bff-4b0c-a6f7-447b93efaa31
             jobConf.set("mapreduce.input.fileinputformat.input.dir.recursive", 
"true");
+            // FileInputFormat.setInputPaths() will call FileSystem.get(), 
which will create new FileSystem
+            // and save it in FileSystem.Cache. We don't need this cache.
+            jobConf.set("fs.hdfs.impl.disable.cache", "true");
             FileInputFormat.setInputPaths(jobConf, finalLocation);
             try {
                 InputFormat<?, ?> inputFormat = 
HiveUtil.getInputFormat(jobConf, key.inputFormat, false);


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org