This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push: new b866096f1c [doc]remove som be.conf (#1001) b866096f1c is described below commit b866096f1c456724e2d78e4fa4f4114ff3a54f05 Author: Mryange <59914473+mrya...@users.noreply.github.com> AuthorDate: Tue Sep 3 11:23:21 2024 +0800 [doc]remove som be.conf (#1001) https://github.com/apache/doris/pull/39219 --- docs/admin-manual/config/be-config.md | 22 ------------------- docs/query/query-variables/variables.md | 8 +++---- .../current/admin-manual/config/be-config.md | 25 ---------------------- .../current/query/query-variables/variables.md | 6 +++--- .../version-3.0/admin-manual/config/be-config.md | 25 ---------------------- .../version-3.0/query/query-variables/variables.md | 6 +++--- .../version-3.0/admin-manual/config/be-config.md | 22 ------------------- .../version-3.0/query/query-variables/variables.md | 6 +++--- 8 files changed, 13 insertions(+), 107 deletions(-) diff --git a/docs/admin-manual/config/be-config.md b/docs/admin-manual/config/be-config.md index e86d0a2698..6bd6915170 100644 --- a/docs/admin-manual/config/be-config.md +++ b/docs/admin-manual/config/be-config.md @@ -349,13 +349,6 @@ The maximum size of a (received) message of the thrift server, in bytes. If the * Description: Follow up asynic work create threads dynamically, with a maximum of 512 threads created. * Default value: 512 -#### `doris_max_scan_key_num` - -* Type: int -* Description: Used to limit the maximum number of scan keys that a scan node can split in a query request. When a conditional query request reaches the scan node, the scan node will try to split the conditions related to the key column in the query condition into multiple scan key ranges. After that, these scan key ranges will be assigned to multiple scanner threads for data scanning. A larger value usually means that more scanner threads can be used to increase the parallelism of the s [...] - - When the concurrency cannot be improved in high concurrency scenarios, try to reduce this value and observe the impact. -* Default value: 48 - #### `doris_scan_range_row_count` * Type: int32 @@ -397,21 +390,6 @@ The maximum size of a (received) message of the thrift server, in bytes. If the * Description: The size of the Buffer queue of the ExchangeNode node, in bytes. After the amount of data sent from the Sender side is larger than the Buffer size of ExchangeNode, subsequent data sent will block until the Buffer frees up space for writing. * Default value: 10485760 -#### `max_pushdown_conditions_per_column` - -* Type: int -* Description: Used to limit the maximum number of conditions that can be pushed down to the storage engine for a single column in a query request. During the execution of the query plan, the filter conditions on some columns can be pushed down to the storage engine, so that the index information in the storage engine can be used for data filtering, reducing the amount of data that needs to be scanned by the query. Such as equivalent conditions, conditions in IN predicates, etc. In most [...] -* Default value: 1024 - -* Example - - The table structure is' id INT, col2 INT, col3 varchar (32),... '. - - The query request is'WHERE id IN (v1, v2, v3, ...) -#### `max_send_batch_parallelism_per_job` - -* Type: int -* Description: Max send batch parallelism for OlapTableSink. The value set by the user for `send_batch_parallelism` is not allowed to exceed `max_send_batch_parallelism_per_job`, if exceed, the value of `send_batch_parallelism` would be `max_send_batch_parallelism_per_job`. -* Default value: 5 - #### `doris_scan_range_max_mb` * Type: int32 diff --git a/docs/query/query-variables/variables.md b/docs/query/query-variables/variables.md index 8c93d855ab..e45aab274a 100644 --- a/docs/query/query-variables/variables.md +++ b/docs/query/query-variables/variables.md @@ -319,12 +319,12 @@ Note that the comment must start with /*+ and can only follow the SELECT. For compatibility with JDBC connection pool C3P0. Has no real effect on Doris itself. If you encounter the error `Packet for query is too large (1,514,085 > 1,048,576). You can change this value on the server by setting the 'max_allowed_packet' variable.`, you can use `set GLOBAL max_allowed_packet = 1548576` to increase the value. * `max_pushdown_conditions_per_column` - - For the specific meaning of this variable, please refer to the description of `max_pushdown_conditions_per_column` in [BE Configuration](../../admin-manual/config/be-config.md). This variable is set to -1 by default, which means that the configuration value in `be.conf` is used. If the setting is greater than 0, the query in the current session will use the variable value, and ignore the configuration value in `be.conf`. + + Used to limit the maximum number of conditions for a single column that can be pushed down to the storage engine in a query request. During the execution of a query plan, some filtering conditions on columns can be pushed down to the storage engine. This allows the storage engine to utilize index information for data filtering, reducing the amount of data that needs to be scanned in the query. For example, equality conditions or conditions in an IN predicate. This parameter typically [...] * `max_scan_key_num` - For the specific meaning of this variable, please refer to the description of `doris_max_scan_key_num` in [BE Configuration](../../admin-manual/config/be-config.md). This variable is set to -1 by default, which means that the configuration value in `be.conf` is used. If the setting is greater than 0, the query in the current session will use the variable value, and ignore the configuration value in `be.conf`. + Used to limit the maximum number of scan keys that can be split by a scan node in a query request. When a query request with conditions reaches a scan node, the scan node will attempt to split the key-related conditions in the query into multiple scan key ranges. These scan key ranges are then allocated to multiple scanner threads for data scanning. A larger value typically allows for more scanner threads, enhancing the parallelism of the scanning operation. However, in high-concurre [...] * `net_buffer_length` @@ -377,7 +377,7 @@ Note that the comment must start with /*+ and can only follow the SELECT. * `send_batch_parallelism` - Used to set the default parallelism for sending batch when execute InsertStmt operation, if the value for parallelism exceed `max_send_batch_parallelism_per_job` in BE config, then the coordinator BE will use the value of `max_send_batch_parallelism_per_job`. + The maximum parallelism for sending batch data by the OlapTableSink. * `sql_mode` diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/config/be-config.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/config/be-config.md index b1417b9acb..b411425dab 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/config/be-config.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/config/be-config.md @@ -360,13 +360,6 @@ Thrift 服务器接收请求消息的大小(字节数)上限。如果客户 * 描述:根据后续任务动态创建线程,最大创建 512 个线程。 * 默认值:512 -#### `doris_max_scan_key_num` - -* 类型:int -* 描述:用于限制一个查询请求中,scan node 节点能拆分的最大 scan key 的个数。当一个带有条件的查询请求到达 scan node 节点时,scan node 会尝试将查询条件中 key 列相关的条件拆分成多个 scan key range。之后这些 scan key range 会被分配给多个 scanner 线程进行数据扫描。较大的数值通常意味着可以使用更多的 scanner 线程来提升扫描操作的并行度。但在高并发场景下,过多的线程可能会带来更大的调度开销和系统负载,反而会降低查询响应速度。一个经验数值为 50。该配置可以单独进行会话级别的配置,具体可参阅 [变量](../../query/query-variables/variables.md) 中 `max_scan_key_num` 的说明。 - - 当在高并发场景下发下并发度无法提升时,可以尝试降低该数值并观察影响。 -* 默认值:48 - #### `doris_scan_range_row_count` * 类型:int32 @@ -408,24 +401,6 @@ Thrift 服务器接收请求消息的大小(字节数)上限。如果客户 * 描述:ExchangeNode 节点 Buffer 队列的大小,单位为 byte。来自 Sender 端发送的数据量大于 ExchangeNode 的 Buffer 大小之后,后续发送的数据将阻塞直到 Buffer 腾出可写入的空间。 * 默认值:10485760 -#### `max_pushdown_conditions_per_column` - -* 类型:int -* 描述:用于限制一个查询请求中,针对单个列,能够下推到存储引擎的最大条件数量。在查询计划执行的过程中,一些列上的过滤条件可以下推到存储引擎,这样可以利用存储引擎中的索引信息进行数据过滤,减少查询需要扫描的数据量。比如等值条件、IN 谓词中的条件等。这个参数在绝大多数情况下仅影响包含 IN 谓词的查询。如 `WHERE colA IN (1,2,3,4,...)`。较大的数值意味值 IN 谓词中更多的条件可以推送给存储引擎,但过多的条件可能会导致随机读的增加,某些情况下可能会降低查询效率。该配置可以单独进行会话级别的配置,具体可参阅 [变量](../../query/query-variables/variables.md) 中 `max_pushdown_conditions_per_column ` 的说明。 -* 默认值:1024 - -* 示例 - - - 表结构为 `id INT, col2 INT, col3 varchar(32), ...`。 - - 查询请求为 `... WHERE id IN (v1, v2, v3, ...)` - - 如果 IN 谓词中的条件数量超过了该配置,则可以尝试增加该配置值,观察查询响应是否有所改善。 - -#### `max_send_batch_parallelism_per_job` - -* 类型:int -* 描述:OlapTableSink 发送批处理数据的最大并行度,用户为 `send_batch_parallelism` 设置的值不允许超过 `max_send_batch_parallelism_per_job` ,如果超过, `send_batch_parallelism` 将被设置为 `max_send_batch_parallelism_per_job` 的值。 -* 默认值:5 - #### `doris_scan_range_max_mb` * 类型:int32 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/query/query-variables/variables.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/query/query-variables/variables.md index 64bb3b938c..ffa3758530 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/query/query-variables/variables.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/query/query-variables/variables.md @@ -318,11 +318,11 @@ SELECT /*+ SET_VAR(query_timeout = 1, enable_partition_cache=true) */ sleep(3); - `max_pushdown_conditions_per_column` - 该变量的具体含义请参阅 [BE 配置项](../../admin-manual/config/be-config) 中 `max_pushdown_conditions_per_column` 的说明。该变量默认置为 -1,表示使用 `be.conf` 中的配置值。如果设置大于 0,则当前会话中的查询会使用该变量值,而忽略 `be.conf` 中的配置值。 + 用于限制一个查询请求中,针对单个列,能够下推到存储引擎的最大条件数量。在查询计划执行的过程中,一些列上的过滤条件可以下推到存储引擎,这样可以利用存储引擎中的索引信息进行数据过滤,减少查询需要扫描的数据量。比如等值条件、IN 谓词中的条件等。这个参数在绝大多数情况下仅影响包含 IN 谓词的查询。如 `WHERE colA IN (1,2,3,4,...)`。较大的数值意味值 IN 谓词中更多的条件可以推送给存储引擎,但过多的条件可能会导致随机读的增加,某些情况下可能会降低查询效率。 默认值:1024。如果 IN 谓词中的条件数量超过了该配置,则可以尝试增加该配置值,观察查询响应是否有所改善。 - `max_scan_key_num` - 该变量的具体含义请参阅 [BE 配置项](../../admin-manual/config/be-config) 中 `doris_max_scan_key_num` 的说明。该变量默认置为 -1,表示使用 `be.conf` 中的配置值。如果设置大于 0,则当前会话中的查询会使用该变量值,而忽略 `be.conf` 中的配置值。 + 用于限制一个查询请求中,scan node 节点能拆分的最大 scan key 的个数。当一个带有条件的查询请求到达 scan node 节点时,scan node 会尝试将查询条件中 key 列相关的条件拆分成多个 scan key range。之后这些 scan key range 会被分配给多个 scanner 线程进行数据扫描。较大的数值通常意味着可以使用更多的 scanner 线程来提升扫描操作的并行度。但在高并发场景下,过多的线程可能会带来更大的调度开销和系统负载,反而会降低查询响应速度。一个经验数值为 50。当在高并发场景下发下并发度无法提升时,可以尝试降低该数值并观察影响。默认值:48。 - `net_buffer_length` @@ -373,7 +373,7 @@ SELECT /*+ SET_VAR(query_timeout = 1, enable_partition_cache=true) */ sleep(3); - `send_batch_parallelism` - 用于设置执行 InsertStmt 操作时发送批处理数据的默认并行度,如果并行度的值超过 BE 配置中的 `max_send_batch_parallelism_per_job`,那么作为协调点的 BE 将使用 `max_send_batch_parallelism_per_job` 的值。 + OlapTableSink 发送批处理数据的最大并行度。 - `sql_mode` diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/config/be-config.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/config/be-config.md index 7a01d7a5ff..553d421a74 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/config/be-config.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/config/be-config.md @@ -360,13 +360,6 @@ Thrift 服务器接收请求消息的大小(字节数)上限。如果客户 * 描述:后续查询请求动态创建线程,最大创建 512 个线程。 * 默认值:2048 -#### `doris_max_scan_key_num` - -* 类型:int -* 描述:用于限制一个查询请求中,scan node 节点能拆分的最大 scan key 的个数。当一个带有条件的查询请求到达 scan node 节点时,scan node 会尝试将查询条件中 key 列相关的条件拆分成多个 scan key range。之后这些 scan key range 会被分配给多个 scanner 线程进行数据扫描。较大的数值通常意味着可以使用更多的 scanner 线程来提升扫描操作的并行度。但在高并发场景下,过多的线程可能会带来更大的调度开销和系统负载,反而会降低查询响应速度。一个经验数值为 50。该配置可以单独进行会话级别的配置,具体可参阅 [变量](../../query/query-variables/variables.md) 中 `max_scan_key_num` 的说明。 - - 当在高并发场景下发下并发度无法提升时,可以尝试降低该数值并观察影响。 -* 默认值:48 - #### `doris_scan_range_row_count` * 类型:int32 @@ -408,24 +401,6 @@ Thrift 服务器接收请求消息的大小(字节数)上限。如果客户 * 描述:ExchangeNode 节点 Buffer 队列的大小,单位为 byte。来自 Sender 端发送的数据量大于 ExchangeNode 的 Buffer 大小之后,后续发送的数据将阻塞直到 Buffer 腾出可写入的空间。 * 默认值:10485760 -#### `max_pushdown_conditions_per_column` - -* 类型:int -* 描述:用于限制一个查询请求中,针对单个列,能够下推到存储引擎的最大条件数量。在查询计划执行的过程中,一些列上的过滤条件可以下推到存储引擎,这样可以利用存储引擎中的索引信息进行数据过滤,减少查询需要扫描的数据量。比如等值条件、IN 谓词中的条件等。这个参数在绝大多数情况下仅影响包含 IN 谓词的查询。如 `WHERE colA IN (1,2,3,4,...)`。较大的数值意味值 IN 谓词中更多的条件可以推送给存储引擎,但过多的条件可能会导致随机读的增加,某些情况下可能会降低查询效率。该配置可以单独进行会话级别的配置,具体可参阅 [变量](../../query/query-variables/variables.md) 中 `max_pushdown_conditions_per_column ` 的说明。 -* 默认值:1024 - -* 示例 - - - 表结构为 `id INT, col2 INT, col3 varchar(32), ...`。 - - 查询请求为 `... WHERE id IN (v1, v2, v3, ...)` - - 如果 IN 谓词中的条件数量超过了该配置,则可以尝试增加该配置值,观察查询响应是否有所改善。 - -#### `max_send_batch_parallelism_per_job` - -* 类型:int -* 描述:OlapTableSink 发送批处理数据的最大并行度,用户为 `send_batch_parallelism` 设置的值不允许超过 `max_send_batch_parallelism_per_job` ,如果超过, `send_batch_parallelism` 将被设置为 `max_send_batch_parallelism_per_job` 的值。 -* 默认值:5 - #### `doris_scan_range_max_mb` * 类型:int32 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/query/query-variables/variables.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/query/query-variables/variables.md index 64bb3b938c..9988700802 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/query/query-variables/variables.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/query/query-variables/variables.md @@ -318,11 +318,11 @@ SELECT /*+ SET_VAR(query_timeout = 1, enable_partition_cache=true) */ sleep(3); - `max_pushdown_conditions_per_column` - 该变量的具体含义请参阅 [BE 配置项](../../admin-manual/config/be-config) 中 `max_pushdown_conditions_per_column` 的说明。该变量默认置为 -1,表示使用 `be.conf` 中的配置值。如果设置大于 0,则当前会话中的查询会使用该变量值,而忽略 `be.conf` 中的配置值。 + 用于限制一个查询请求中,针对单个列,能够下推到存储引擎的最大条件数量。在查询计划执行的过程中,一些列上的过滤条件可以下推到存储引擎,这样可以利用存储引擎中的索引信息进行数据过滤,减少查询需要扫描的数据量。比如等值条件、IN 谓词中的条件等。这个参数在绝大多数情况下仅影响包含 IN 谓词的查询。如 `WHERE colA IN (1,2,3,4,...)`。较大的数值意味值 IN 谓词中更多的条件可以推送给存储引擎,但过多的条件可能会导致随机读的增加,某些情况下可能会降低查询效率。 默认值:1024。如果 IN 谓词中的条件数量超过了该配置,则可以尝试增加该配置值,观察查询响应是否有所改善。 - `max_scan_key_num` - 该变量的具体含义请参阅 [BE 配置项](../../admin-manual/config/be-config) 中 `doris_max_scan_key_num` 的说明。该变量默认置为 -1,表示使用 `be.conf` 中的配置值。如果设置大于 0,则当前会话中的查询会使用该变量值,而忽略 `be.conf` 中的配置值。 + 用于限制一个查询请求中,scan node 节点能拆分的最大 scan key 的个数。当一个带有条件的查询请求到达 scan node 节点时,scan node 会尝试将查询条件中 key 列相关的条件拆分成多个 scan key range。之后这些 scan key range 会被分配给多个 scanner 线程进行数据扫描。较大的数值通常意味着可以使用更多的 scanner 线程来提升扫描操作的并行度。但在高并发场景下,过多的线程可能会带来更大的调度开销和系统负载,反而会降低查询响应速度。一个经验数值为 50。当在高并发场景下发下并发度无法提升时,可以尝试降低该数值并观察影响。默认值:48。 - `net_buffer_length` @@ -373,7 +373,7 @@ SELECT /*+ SET_VAR(query_timeout = 1, enable_partition_cache=true) */ sleep(3); - `send_batch_parallelism` - 用于设置执行 InsertStmt 操作时发送批处理数据的默认并行度,如果并行度的值超过 BE 配置中的 `max_send_batch_parallelism_per_job`,那么作为协调点的 BE 将使用 `max_send_batch_parallelism_per_job` 的值。 + OlapTableSink 发送批处理数据的最大并行度。 - `sql_mode` diff --git a/versioned_docs/version-3.0/admin-manual/config/be-config.md b/versioned_docs/version-3.0/admin-manual/config/be-config.md index 4a08abb464..cdf546d564 100644 --- a/versioned_docs/version-3.0/admin-manual/config/be-config.md +++ b/versioned_docs/version-3.0/admin-manual/config/be-config.md @@ -349,13 +349,6 @@ The maximum size of a (received) message of the thrift server, in bytes. If the * Description: Follow up query requests create threads dynamically, with a maximum of 512 threads created. * Default value: 2048 -#### `doris_max_scan_key_num` - -* Type: int -* Description: Used to limit the maximum number of scan keys that a scan node can split in a query request. When a conditional query request reaches the scan node, the scan node will try to split the conditions related to the key column in the query condition into multiple scan key ranges. After that, these scan key ranges will be assigned to multiple scanner threads for data scanning. A larger value usually means that more scanner threads can be used to increase the parallelism of the s [...] - - When the concurrency cannot be improved in high concurrency scenarios, try to reduce this value and observe the impact. -* Default value: 48 - #### `doris_scan_range_row_count` * Type: int32 @@ -397,21 +390,6 @@ The maximum size of a (received) message of the thrift server, in bytes. If the * Description: The size of the Buffer queue of the ExchangeNode node, in bytes. After the amount of data sent from the Sender side is larger than the Buffer size of ExchangeNode, subsequent data sent will block until the Buffer frees up space for writing. * Default value: 10485760 -#### `max_pushdown_conditions_per_column` - -* Type: int -* Description: Used to limit the maximum number of conditions that can be pushed down to the storage engine for a single column in a query request. During the execution of the query plan, the filter conditions on some columns can be pushed down to the storage engine, so that the index information in the storage engine can be used for data filtering, reducing the amount of data that needs to be scanned by the query. Such as equivalent conditions, conditions in IN predicates, etc. In most [...] -* Default value: 1024 - -* Example - - The table structure is' id INT, col2 INT, col3 varchar (32),... '. - - The query request is'WHERE id IN (v1, v2, v3, ...) -#### `max_send_batch_parallelism_per_job` - -* Type: int -* Description: Max send batch parallelism for OlapTableSink. The value set by the user for `send_batch_parallelism` is not allowed to exceed `max_send_batch_parallelism_per_job`, if exceed, the value of `send_batch_parallelism` would be `max_send_batch_parallelism_per_job`. -* Default value: 5 - #### `doris_scan_range_max_mb` * Type: int32 diff --git a/versioned_docs/version-3.0/query/query-variables/variables.md b/versioned_docs/version-3.0/query/query-variables/variables.md index 7d52e4b910..5e1b4afbca 100644 --- a/versioned_docs/version-3.0/query/query-variables/variables.md +++ b/versioned_docs/version-3.0/query/query-variables/variables.md @@ -320,11 +320,11 @@ Note that the comment must start with /*+ and can only follow the SELECT. * `max_pushdown_conditions_per_column` - For the specific meaning of this variable, please refer to the description of `max_pushdown_conditions_per_column` in [BE Configuration](../../admin-manual/config/be-config.md). This variable is set to -1 by default, which means that the configuration value in `be.conf` is used. If the setting is greater than 0, the query in the current session will use the variable value, and ignore the configuration value in `be.conf`. + Used to limit the maximum number of conditions for a single column that can be pushed down to the storage engine in a query request. During the execution of a query plan, some filtering conditions on columns can be pushed down to the storage engine. This allows the storage engine to utilize index information for data filtering, reducing the amount of data that needs to be scanned in the query. For example, equality conditions or conditions in an IN predicate. This parameter typically [...] * `max_scan_key_num` - For the specific meaning of this variable, please refer to the description of `doris_max_scan_key_num` in [BE Configuration](../../admin-manual/config/be-config.md). This variable is set to -1 by default, which means that the configuration value in `be.conf` is used. If the setting is greater than 0, the query in the current session will use the variable value, and ignore the configuration value in `be.conf`. + Used to limit the maximum number of scan keys that can be split by a scan node in a query request. When a query request with conditions reaches a scan node, the scan node will attempt to split the key-related conditions in the query into multiple scan key ranges. These scan key ranges are then allocated to multiple scanner threads for data scanning. A larger value typically allows for more scanner threads, enhancing the parallelism of the scanning operation. However, in high-concurrenc [...] * `net_buffer_length` @@ -377,7 +377,7 @@ Note that the comment must start with /*+ and can only follow the SELECT. * `send_batch_parallelism` - Used to set the default parallelism for sending batch when execute InsertStmt operation, if the value for parallelism exceed `max_send_batch_parallelism_per_job` in BE config, then the coordinator BE will use the value of `max_send_batch_parallelism_per_job`. + The maximum parallelism for sending batch data by the OlapTableSink. * `sql_mode` --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org