This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 9c7601841e [Doc]broker load rpc timeout problem FQA (#10698)
9c7601841e is described below
commit 9c7601841ed99310cac16947ec52e0c0b413d1bf
Author: jiafeng.zhang <zhang...@gmail.com>
AuthorDate: Sat Jul 9 06:24:08 2022 +0800
[Doc]broker load rpc timeout problem FQA (#10698)
---
.../data-operate/import/import-way/broker-load-manual.md | 15 ++++++++++++++-
docs/en/docs/faq/data-faq.md | 15 +++++++++++++++
.../data-operate/import/import-way/broker-load-manual.md | 15 ++++++++++++++-
docs/zh-CN/docs/faq/data-faq.md | 13 +++++++++++++
4 files changed, 56 insertions(+), 2 deletions(-)
diff --git a/docs/en/docs/data-operate/import/import-way/broker-load-manual.md
b/docs/en/docs/data-operate/import/import-way/broker-load-manual.md
index cd2140bc3e..84504fa1f7 100644
--- a/docs/en/docs/data-operate/import/import-way/broker-load-manual.md
+++ b/docs/en/docs/data-operate/import/import-way/broker-load-manual.md
@@ -245,7 +245,7 @@ LOAD LABEL demo.label_20220402
)
with HDFS (
"fs.defaultFS"="hdfs://10.220.147.151:8020",
- "hdfs_user"="root"
+ "hadoop.username"="root"
)
PROPERTIES
(
@@ -407,6 +407,19 @@ Currently the Profile can only be viewed after the job has
been successfully exe
Please refer to the Best Practices section in the document to modify the FE
configuration items `max_bytes_per_broker_scanner` and `max_broker_concurrency`
+- `org.apache.thrift.transport.TTransportException: java.net.SocketException:
Broken pipe` during import
+
+ The reason for this problem may be that when importing data from external
storage (such as HDFS), because there are too many files in the directory, it
takes too long to list the file directory. Here, the Broker RPC Timeout
defaults to 10 seconds, and the timeout needs to be adjusted appropriately
here. time.
+
+ Modify the `fe.conf` configuration file to add the following parameters:
+
+ ````
+ broker_timeout_ms = 10000
+ ##The default here is 10 seconds, you need to increase this parameter
appropriately
+ ````
+
+ Adding parameters here requires restarting the FE service.
+
- Import error: `failed to send batch` or `TabletWriter add batch with unknown
id`
Modify `query_timeout` and `streaming_load_rpc_max_alive_time_sec`
appropriately.
diff --git a/docs/en/docs/faq/data-faq.md b/docs/en/docs/faq/data-faq.md
index d9dab3abee..ca5096e391 100644
--- a/docs/en/docs/faq/data-faq.md
+++ b/docs/en/docs/faq/data-faq.md
@@ -134,3 +134,18 @@ The second is that the packet size of rpc exceeds
max_body_size. This problem ma
```
brpc_max_body_size：default 3GB.
```
+
+### Q10. [ Broker load ] org.apache.thrift.transport.TTransportException:
java.net.SocketException: Broken pipe
+
+`org.apache.thrift.transport.TTransportException: java.net.SocketException:
Broken pipe` during import.
+
+The reason for this problem may be that when importing data from external
storage (such as HDFS), because there are too many files in the directory, it
takes too long to list the file directory. Here, the Broker RPC Timeout
defaults to 10 seconds, and the timeout needs to be adjusted appropriately
here. time.
+
+Modify the `fe.conf` configuration file to add the following parameters:
+
+````
+broker_timeout_ms = 10000
+##The default here is 10 seconds, you need to increase this parameter
appropriately
+````
+
+Adding parameters here requires restarting the FE service.
diff --git
a/docs/zh-CN/docs/data-operate/import/import-way/broker-load-manual.md
b/docs/zh-CN/docs/data-operate/import/import-way/broker-load-manual.md
index feb30e39be..7a0031074d 100644
--- a/docs/zh-CN/docs/data-operate/import/import-way/broker-load-manual.md
+++ b/docs/zh-CN/docs/data-operate/import/import-way/broker-load-manual.md
@@ -244,7 +244,7 @@ LOAD LABEL demo.label_20220402
)
with HDFS (
"fs.defaultFS"="hdfs://10.220.147.151:8020",
- "hdfs_user"="root"
+ "hadoop.username"="root"
)
PROPERTIES
(
@@ -407,6 +407,19 @@ FE 的配置参数 `async_loading_load_task_pool_size` 用于限制同时运行
请参照文档中最佳实践部分，修改 FE 配置项 `max_bytes_per_broker_scanner` 和
`max_broker_concurrency`
+- 导入过程中出现 `org.apache.thrift.transport.TTransportException:
java.net.SocketException: Broken pipe`
+
+ 出现这个问题的原因可能是到从外部存储（例如HDFS）导入数据的时候，因为目录下文件太多，列出文件目录的时间太长，这里Broker RPC Timeout
默认是10秒，这里需要适当调整超时时间。
+
+ 修改 `fe.conf` 配置文件，添加下面的参数：
+
+ ```
+ broker_timeout_ms = 10000
+ ##这里默认是10秒，需要适当加大这个参数
+ ```
+
+ 这里添加参数，需要重启 FE 服务。
+
- 导入报错：`failed to send batch` 或 `TabletWriter add batch with unknown id`
适当修改 `query_timeout` 和 `streaming_load_rpc_max_alive_time_sec`。
diff --git a/docs/zh-CN/docs/faq/data-faq.md b/docs/zh-CN/docs/faq/data-faq.md
index 81d8e6f6ad..ae7a126fdd 100644
--- a/docs/zh-CN/docs/faq/data-faq.md
+++ b/docs/zh-CN/docs/faq/data-faq.md
@@ -134,3 +134,16 @@ failed to initialize storage reader.
tablet=63416.1050661139.aa4d304e7a7aff9c-f0
```
brpc_max_body_size：默认 3GB.
```
+
+### Q10. [ Broker load ] org.apache.thrift.transport.TTransportException:
java.net.SocketException: Broken pipe
+
+出现这个问题的原因可能是到从外部存储（例如HDFS）导入数据的时候，因为目录下文件太多，列出文件目录的时间太长，这里Broker RPC Timeout
默认是10秒，这里需要适当调整超时时间。
+
+修改 `fe.conf` 配置文件，添加下面的参数：
+
+```
+broker_timeout_ms = 10000
+##这里默认是10秒，需要适当加大这个参数
+```
+
+这里添加参数，需要重启 FE 服务。
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
