This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 3f06b50a65086f7cd035a3d83099eb57a4eea564
Author: Mingyu Chen <morning...@163.com>
AuthorDate: Tue Sep 5 11:54:19 2023 +0800
[improvemnt](script) support custom lib dir to save custom libs (#23887)
Sometimes, user need to add some custom libs to the cluster, such lzo.jar,
orai18n.jar, etc.
In previous, these lib files are places in fe/lib or be/lib.
But when upgrading cluster, the lib dir will be replaced by new lib dir, so
that all custom libs are lost.
In this PR, I add new dir custom_lib for FE and BE, and user can place
custom lib files in it.
---
bin/start_be.sh | 7 +++++++
bin/start_fe.sh | 7 +++++++
docs/en/docs/lakehouse/multi-catalog/faq.md | 4 ++++
docs/en/docs/lakehouse/multi-catalog/jdbc.md | 4 +++-
docs/en/docs/lakehouse/multi-catalog/paimon.md | 5 ++++-
docs/zh-CN/docs/lakehouse/multi-catalog/faq.md | 4 ++++
docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md | 2 ++
docs/zh-CN/docs/lakehouse/multi-catalog/paimon.md | 7 +++++--
8 files changed, 36 insertions(+), 4 deletions(-)
diff --git a/bin/start_be.sh b/bin/start_be.sh
index 0af43edfa9..514da5e02a 100755
--- a/bin/start_be.sh
+++ b/bin/start_be.sh
@@ -112,6 +112,13 @@ if [[ -d "${DORIS_HOME}/lib/hadoop_hdfs/" ]]; then
done
fi
+# add custome_libs to CLASSPATH
+if [[ -d "${DORIS_HOME}/custom_lib" ]]; then
+ for f in "${DORIS_HOME}/custom_lib"/*.jar; do
+ DORIS_CLASSPATH="${DORIS_CLASSPATH}:${f}"
+ done
+fi
+
if [[ -n "${HADOOP_CONF_DIR}" ]]; then
export DORIS_CLASSPATH="${HADOOP_CONF_DIR}:${DORIS_CLASSPATH}"
fi
diff --git a/bin/start_fe.sh b/bin/start_fe.sh
index 3cdebe8f4b..82d0e78501 100755
--- a/bin/start_fe.sh
+++ b/bin/start_fe.sh
@@ -176,6 +176,13 @@ for f in "${DORIS_HOME}/lib"/*.jar; do
CLASSPATH="${f}:${CLASSPATH}"
done
+# add custome_libs to CLASSPATH
+if [[ -d "${DORIS_HOME}/custom_lib" ]]; then
+ for f in "${DORIS_HOME}/custom_lib"/*.jar; do
+ CLASSPATH="${f}:${CLASSPATH}"
+ done
+fi
+
# make sure the doris-fe.jar is at first order, so that some classed
# with same qualified name can be loaded priority from doris-fe.jar
CLASSPATH="${DORIS_FE_JAR}:${CLASSPATH}"
diff --git a/docs/en/docs/lakehouse/multi-catalog/faq.md
b/docs/en/docs/lakehouse/multi-catalog/faq.md
index 9a756bf830..a32e33887c 100644
--- a/docs/en/docs/lakehouse/multi-catalog/faq.md
+++ b/docs/en/docs/lakehouse/multi-catalog/faq.md
@@ -150,6 +150,10 @@ under the License.
14. When using JDBC Catalog to synchronize MySQL data to Doris, the date data
synchronization error occurs. It is necessary to check whether the MySQL
version corresponds to the MySQL driver package. For example, the driver
com.mysql.cj.jdbc.Driver is required for MySQL8 and above.
+ Starting from version 2.0.2, this file can be placed in BE's `custom_lib/`
directory (if it does not exist, just create it manually) to prevent the file
from being lost due to the replacement of the lib directory when upgrading the
cluster.
+
+## HDFS
+
15. If an error is reported while configuring Kerberos in the catalog: `SIMPLE
authentication is not enabled. Available:[TOKEN, KERBEROS]`.
Need to put `core-site.xml` to the `"${DORIS_HOME}/be/conf"` directory.
diff --git a/docs/en/docs/lakehouse/multi-catalog/jdbc.md
b/docs/en/docs/lakehouse/multi-catalog/jdbc.md
index 7200752064..bb45a3809f 100644
--- a/docs/en/docs/lakehouse/multi-catalog/jdbc.md
+++ b/docs/en/docs/lakehouse/multi-catalog/jdbc.md
@@ -716,7 +716,9 @@ When Doris connects to OceanBase, it will automatically
recognize that OceanBase
12. `Non supported character set (add orai18n.jar in your classpath):
ZHS16GBK` exception occurs when reading Oracle
- Download
[orai18n.jar](https://www.oracle.com/database/technologies/appdev/jdbc-downloads.html)
and put it in the lib directory of Doris FE and the lib/java_extensions
directory of BE (Doris versions before 2.0 need to be placed in the lib
directory of BE).
+ Download
[orai18n.jar](https://www.oracle.com/database/technologies/appdev/jdbc-downloads.html)
and put it in the lib directory of Doris FE and the `lib/java_extensions/`
directory of BE (Doris versions before 2.0 need to be placed in the lib
directory of BE).
+
+ Starting from version 2.0.2, this file can be placed in BE's `custom_lib/`
directory (if it does not exist, just create it manually) to prevent the file
from being lost due to the replacement of the lib directory when upgrading the
cluster.
13. `NoClassDefFoundError: net/jpountz/lz4/LZ4Factory` exception occurs when
reading Clickhouse data via jdbc catalog.
diff --git a/docs/en/docs/lakehouse/multi-catalog/paimon.md
b/docs/en/docs/lakehouse/multi-catalog/paimon.md
index 68bf4cb011..b095bdee3e 100644
--- a/docs/en/docs/lakehouse/multi-catalog/paimon.md
+++ b/docs/en/docs/lakehouse/multi-catalog/paimon.md
@@ -60,10 +60,13 @@ CREATE CATALOG `paimon_hdfs` PROPERTIES (
#### S3
->Note that.
+> Note that.
>
> user need download
> [paimon-s3-0.4.0-incubating.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-s3/0.4.0-incubating/paimon-s3-0.4.0-incubating.jar)
+>
> Place it in directory
> ${DORIS_HOME}/be/lib/java_extensions/preload-extensions and restart be
+>
+> Starting from version 2.0.2, this file can be placed in BE's `custom_lib/`
directory (if it does not exist, just create it manually) to prevent the file
from being lost due to the replacement of the lib directory when upgrading the
cluster.
```sql
CREATE CATALOG `paimon_s3` PROPERTIES (
diff --git a/docs/zh-CN/docs/lakehouse/multi-catalog/faq.md
b/docs/zh-CN/docs/lakehouse/multi-catalog/faq.md
index 11e2cc937f..18cd16e720 100644
--- a/docs/zh-CN/docs/lakehouse/multi-catalog/faq.md
+++ b/docs/zh-CN/docs/lakehouse/multi-catalog/faq.md
@@ -148,6 +148,10 @@ under the License.
);
```
+ 从 2.0.2 版本起，可以将这个文件放置在BE的 `custom_lib/` 目录下（如不存在，手动创建即可），以防止升级集群时因为 lib
目录被替换而导致文件丢失。
+
+## HDFS
+
14. 使用JDBC
Catalog将MySQL数据同步到Doris中，日期数据同步错误。需要校验下MySQL的版本是否与MySQL的驱动包是否对应，比如MySQL8以上需要使用驱动com.mysql.cj.jdbc.Driver。
15. 在Catalog中配置Kerberos时，如果报错`SIMPLE authentication is not enabled.
Available:[TOKEN,
KERBEROS]`，那么需要将`core-site.xml`文件放到`"${DORIS_HOME}/be/conf"`目录下。
diff --git a/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
b/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
index 79301cd1e1..732fa26363 100644
--- a/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
+++ b/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
@@ -713,6 +713,8 @@ CREATE CATALOG jdbc_oceanbase PROPERTIES (
下载
[orai18n.jar](https://www.oracle.com/database/technologies/appdev/jdbc-downloads.html)
并放到 Doris FE 的 lib 目录以及 BE 的 lib/java_extensions 目录 (Doris 2.0 之前的版本需放到 BE 的
lib 目录下) 下即可。
+ 从 2.0.2 版本起，可以将这个文件放置在BE的 `custom_lib/` 目录下（如不存在，手动创建即可），以防止升级集群时因为 lib
目录被替换而导致文件丢失。
+
13. 通过jdbc catalog 读取Clickhouse数据出现`NoClassDefFoundError:
net/jpountz/lz4/LZ4Factory` 错误信息
可以先下载[lz4-1.3.0.jar](https://repo1.maven.org/maven2/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.jar)包，然后放到DorisFE
lib 目录以及BE 的 `lib/lib/java_extensions`目录中（Doris 2.0 之前的版本需放到 BE 的 lib 目录下）。
diff --git a/docs/zh-CN/docs/lakehouse/multi-catalog/paimon.md
b/docs/zh-CN/docs/lakehouse/multi-catalog/paimon.md
index 73cab27a03..bd22d59af9 100644
--- a/docs/zh-CN/docs/lakehouse/multi-catalog/paimon.md
+++ b/docs/zh-CN/docs/lakehouse/multi-catalog/paimon.md
@@ -60,10 +60,13 @@ CREATE CATALOG `paimon_hdfs` PROPERTIES (
#### S3
->注意：
+> 注意：
>
> 用户需要手动下载[paimon-s3-0.4.0-incubating.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-s3/0.4.0-incubating/paimon-s3-0.4.0-incubating.jar)
-> 放在${DORIS_HOME}/be/lib/java_extensions/preload-extensions目录下并重启be
+
+> 放在${DORIS_HOME}/be/lib/java_extensions/preload-extensions目录下并重启be。
+>
+> 从 2.0.2 版本起，可以将这个文件放置在BE的 `custom_lib/` 目录下（如不存在，手动创建即可），以防止升级集群时因为 lib
目录被替换而导致文件丢失。
```sql
CREATE CATALOG `paimon_s3` PROPERTIES (
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
