This is an automated email from the ASF dual-hosted git repository. kassiez pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push: new c6bb83bf8cc [opt](group commit) remove group_commit_memory_rows_for_max_filter_ra… (#2329) c6bb83bf8cc is described below commit c6bb83bf8cc6a45536b75ef9b3dfa87b1cedb109 Author: meiyi <me...@selectdb.com> AuthorDate: Sun Apr 27 18:23:39 2025 +0800 [opt](group commit) remove group_commit_memory_rows_for_max_filter_ra… (#2329) …tio config ## Versions - [x] dev - [x] 3.0 - [x] 2.1 - [ ] 2.0 ## Languages - [x] Chinese - [x] English ## Docs Checklist - [ ] Checked by AI - [ ] Test Cases Built --- docs/admin-manual/config/be-config.md | 5 ----- docs/data-operate/import/group-commit-manual.md | 22 ++++++---------------- .../current/admin-manual/config/be-config.md | 5 ----- .../data-operate/import/group-commit-manual.md | 22 ++++++---------------- .../version-2.1/admin-manual/config/be-config.md | 5 ----- .../data-operate/import/group-commit-manual.md | 22 ++++++---------------- .../version-3.0/admin-manual/config/be-config.md | 5 ----- .../data-operate/import/group-commit-manual.md | 22 ++++++---------------- .../version-2.1/admin-manual/config/be-config.md | 5 ----- .../data-operate/import/group-commit-manual.md | 16 +++------------- .../version-3.0/admin-manual/config/be-config.md | 5 ----- .../data-operate/import/group-commit-manual.md | 22 ++++++---------------- 12 files changed, 33 insertions(+), 123 deletions(-) diff --git a/docs/admin-manual/config/be-config.md b/docs/admin-manual/config/be-config.md index 71c0cd381b6..b0d89f06816 100644 --- a/docs/admin-manual/config/be-config.md +++ b/docs/admin-manual/config/be-config.md @@ -1356,11 +1356,6 @@ Indicates how many tablets failed to load in the data directory. At the same tim group_commit_wal_path=/data1/storage/wal;/data2/storage/wal;/data3/storage/wal ``` -#### `group_commit_memory_rows_for_max_filter_ratio` - -* Description: The `max_filter_ratio` limit can only work if the total rows of `group commit` is less than this value. See [Group Commit](../../data-operate/import/group-commit-manual.md) for more details -* Default: 10000 - ### Compute and Storage Disaggregated Mode #### `deploy_mode` diff --git a/docs/data-operate/import/group-commit-manual.md b/docs/data-operate/import/group-commit-manual.md index 315b10359e3..1377bb63ef9 100644 --- a/docs/data-operate/import/group-commit-manual.md +++ b/docs/data-operate/import/group-commit-manual.md @@ -437,12 +437,6 @@ Recommend balancing based on system memory resources and data reliability requir group_commit_wal_path=/data1/storage/wal;/data2/storage/wal;/data3/storage/wal ``` -2. `group_commit_memory_rows_for_max_filter_ratio` - - * Description: `max_filter_ratio` works normally when group commit load total rows don't exceed this value, otherwise it doesn't work - - * Default: 10000 - ## Usage Limitations * **Group Commit Limitations** @@ -463,10 +457,6 @@ Recommend balancing based on system memory resources and data reliability requir * **Unique Model** - Group Commit doesn't guarantee commit order, recommend using Sequence column to ensure data consistency. -* **max_filter_ratio Support** - - In default loads, `filter_ratio` is calculated through failed rows and total rows. - - In Group Commit mode, `max_filter_ratio` works when total rows don't exceed `group_commit_memory_rows_for_max_filter_ratio`. - * **WAL Limitations** - `async_mode` writes data to WAL, deletes after success, recovers through WAL on failure. - WAL files are stored with a single replica on one BE, disk damage or accidental file deletion may cause data loss. @@ -647,21 +637,21 @@ JMeter Parameter Settings as Shown in the Images **Performance Test with 30 Concurrent Users in Sync Mode, 5 BEs, and 3 Replicas** - | Group commit internal | 10ms | 20ms | 50ms | 100ms | + | Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 891.8 | 701.1 | 400.0 | 237.5 | |enable_nereids_planner=false| 885.8 | 688.1 | 398.7 | 232.9 | **Performance Test with 100 Concurrent Users in Sync Mode, 5 BEs, and 3 Replicas** - | Group commit internal | 10ms | 20ms | 50ms | 100ms | + | Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 2427.8 | 2068.9 | 1259.4 | 764.9 | |enable_nereids_planner=false| 2320.4 | 1899.3 | 1206.2 |749.7| **Performance Test with 500 Concurrent Users in Sync Mode, 5 BEs, and 3 Replicas** - | Group commit internal | 10ms | 20ms | 50ms | 100ms | + | Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 5567.5 | 5713.2 | 4681.0 | 3131.2 | |enable_nereids_planner=false| 4471.6 | 5042.5 | 4932.2 | 3641.1 | @@ -698,21 +688,21 @@ JMeter Parameter Settings as Shown in the Images **Performance Test with 30 Concurrent Users in Sync Mode, 5 BEs, and 3 Replicas** -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 9.1K | 11.1K | 11.4K | 11.1K | |enable_nereids_planner=false| 157.8K | 159.9K | 154.1K | 120.4K | **Performance Test with 100 Concurrent Users in Sync Mode, 5 BEs, and 3 Replicas** -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 10.0K |9.2K | 8.9K | 8.9K | |enable_nereids_planner=false| 130.4k | 131.0K | 130.4K | 124.1K | **Performance Test with 500 Concurrent Users in Sync Mode, 5 BEs, and 3 Replicas** -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 2.5K | 2.5K | 2.3K | 2.1K | |enable_nereids_planner=false| 94.2K | 95.1K | 94.4K | 94.8K | diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/config/be-config.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/config/be-config.md index 39a4c4a9b63..0f4b062de76 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/config/be-config.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/config/be-config.md @@ -1336,11 +1336,6 @@ load tablets from header failed, failed tablets size: xxx, path=xxx group_commit_wal_path=/data1/storage/wal;/data2/storage/wal;/data3/storage/wal ``` -#### `group_commit_memory_rows_for_max_filter_ratio` - -* 描述:当 Group Commit 导入的总行数不高于该值,`max_filter_ratio` 正常工作,否则不工作,请参考 [Group Commit](../../data-operate/import/group-commit-manual.md) -* 默认值:10000 - ### 存算分离模式 #### `deploy_mode` diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/group-commit-manual.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/group-commit-manual.md index 4ddae61a832..af3d80e044b 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/group-commit-manual.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/group-commit-manual.md @@ -441,12 +441,6 @@ ALTER TABLE dt SET ("group_commit_data_bytes" = "134217728"); group_commit_wal_path=/data1/storage/wal;/data2/storage/wal;/data3/storage/wal ``` -2. `group_commit_memory_rows_for_max_filter_ratio` - - * 描述:当 group commit 导入的总行数不高于该值,`max_filter_ratio` 正常工作,否则不工作 - - * 默认值:10000 - ## 使用限制 * **Group Commit 限制条件** @@ -467,10 +461,6 @@ ALTER TABLE dt SET ("group_commit_data_bytes" = "134217728"); * **Unique 模型** - Group Commit 不保证提交顺序,建议使用 Sequence 列来保证数据一致性。 -* **max_filter_ratio 支持** - - 默认导入中,`filter_ratio` 通过失败行数和总行数计算。 - - Group Commit 模式下,`max_filter_ratio` 在总行数不超过 `group_commit_memory_rows_for_max_filter_ratio` 时有效。 - * **WAL 限制** - `async_mode` 写入会将数据写入 WAL,成功后删除,失败时通过 WAL 恢复。 - WAL 文件是单副本存储的,如果对应磁盘损坏或文件误删可能导致数据丢失。 @@ -641,21 +631,21 @@ PROPERTIES ( #### 30 并发 sync 模式 5 个 BE3 副本性能测试 -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 891.8 | 701.1 | 400.0 | 237.5 | |enable_nereids_planner=false| 885.8 | 688.1 | 398.7 | 232.9 | #### 100 并发 sync 模式 5 个 BE3 副本性能测试 -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 2427.8 | 2068.9 | 1259.4 | 764.9 | |enable_nereids_planner=false| 2320.4 | 1899.3 | 1206.2 |749.7| #### 500 并发 sync 模式 5 个 BE3 副本性能测试 -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 5567.5 | 5713.2 | 4681.0 | 3131.2 | |enable_nereids_planner=false| 4471.6 | 5042.5 | 4932.2 | 3641.1 | @@ -719,21 +709,21 @@ PROPERTIES ( #### 30 并发 sync 模式 5 个 BE3 副本性能测试 -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 9.1K | 11.1K | 11.4K | 11.1K | |enable_nereids_planner=false| 157.8K | 159.9K | 154.1K | 120.4K | #### 100 并发 sync 模式 5 个 BE3 副本性能测试 -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 10.0K |9.2K | 8.9K | 8.9K | |enable_nereids_planner=false| 130.4k | 131.0K | 130.4K | 124.1K | #### 500 并发 sync 模式 5 个 BE3 副本性能测试 -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 2.5K | 2.5K | 2.3K | 2.1K | |enable_nereids_planner=false| 94.2K | 95.1K | 94.4K | 94.8K | diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/config/be-config.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/config/be-config.md index 3f0e8e4aeb1..7c89cbdd8aa 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/config/be-config.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/config/be-config.md @@ -1504,11 +1504,6 @@ load tablets from header failed, failed tablets size: xxx, path=xxx group_commit_wal_path=/data1/storage/wal;/data2/storage/wal;/data3/storage/wal ``` -#### `group_commit_memory_rows_for_max_filter_ratio` - -* 描述:当 group commit 导入的总行数不高于该值,`max_filter_ratio` 正常工作,否则不工作,请参考 [Group Commit](../../data-operate/import/group-commit-manual.md) -* 默认值:10000 - #### `default_tzfiles_path` * 描述:Doris 自带的时区数据库。如果系统目录下未找到时区文件,则启用该目录下的数据。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/data-operate/import/group-commit-manual.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/data-operate/import/group-commit-manual.md index b23fee4725c..3bfa8daea60 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/data-operate/import/group-commit-manual.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/data-operate/import/group-commit-manual.md @@ -441,12 +441,6 @@ ALTER TABLE dt SET ("group_commit_data_bytes" = "134217728"); group_commit_wal_path=/data1/storage/wal;/data2/storage/wal;/data3/storage/wal ``` -2. `group_commit_memory_rows_for_max_filter_ratio` - - * 描述:当 group commit 导入的总行数不高于该值,`max_filter_ratio` 正常工作,否则不工作 - - * 默认值:10000 - ## 使用限制 * **Group Commit 限制条件** @@ -467,10 +461,6 @@ ALTER TABLE dt SET ("group_commit_data_bytes" = "134217728"); * **Unique 模型** - Group Commit 不保证提交顺序,建议使用 Sequence 列来保证数据一致性。 -* **max_filter_ratio 支持** - - 默认导入中,`filter_ratio` 通过失败行数和总行数计算。 - - Group Commit 模式下,`max_filter_ratio` 在总行数不超过 `group_commit_memory_rows_for_max_filter_ratio` 时有效。 - * **WAL 限制** - `async_mode` 写入会将数据写入 WAL,成功后删除,失败时通过 WAL 恢复。 - WAL 文件是单副本存储的,如果对应磁盘损坏或文件误删可能导致数据丢失。 @@ -635,20 +625,20 @@ PROPERTIES ( **30 并发 sync 模式 5 个 BE3 副本性能测试** -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| | | 321.5 | 307.3 | 285.8 | 224.3 | **100 并发 sync 模式性能测试** -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| | | 1175.2 | 1108.7 | 1016.3 | 704.5 | **500 并发 sync 模式性能测试** -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| | | 3289.8 | 3686.7 | 3280.7 | 2609.2 | @@ -711,18 +701,18 @@ PROPERTIES ( **30 并发 sync 模式性能测试** -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| | | 92.2K | 85.9K | 84K | 83.2K | **100 并发 sync 模式性能测试** -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| | | 70.4K |70.5K | 73.2K | 69.4K | **500 并发 sync 模式性能测试** -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| | | 46.3K | 47.7K | 47.4K | 46.5K | diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/config/be-config.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/config/be-config.md index b2ec7c919c7..d8d13fe6c62 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/config/be-config.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/config/be-config.md @@ -1491,11 +1491,6 @@ load tablets from header failed, failed tablets size: xxx, path=xxx group_commit_wal_path=/data1/storage/wal;/data2/storage/wal;/data3/storage/wal ``` -#### `group_commit_memory_rows_for_max_filter_ratio` - -* 描述:当 Group Commit 导入的总行数不高于该值,`max_filter_ratio` 正常工作,否则不工作,请参考 [Group Commit](../../data-operate/import/group-commit-manual.md) -* 默认值:10000 - #### `default_tzfiles_path` * 描述:Doris 自带的时区数据库。如果系统目录下未找到时区文件,则启用该目录下的数据。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/data-operate/import/group-commit-manual.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/data-operate/import/group-commit-manual.md index 4ddae61a832..af3d80e044b 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/data-operate/import/group-commit-manual.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/data-operate/import/group-commit-manual.md @@ -441,12 +441,6 @@ ALTER TABLE dt SET ("group_commit_data_bytes" = "134217728"); group_commit_wal_path=/data1/storage/wal;/data2/storage/wal;/data3/storage/wal ``` -2. `group_commit_memory_rows_for_max_filter_ratio` - - * 描述:当 group commit 导入的总行数不高于该值,`max_filter_ratio` 正常工作,否则不工作 - - * 默认值:10000 - ## 使用限制 * **Group Commit 限制条件** @@ -467,10 +461,6 @@ ALTER TABLE dt SET ("group_commit_data_bytes" = "134217728"); * **Unique 模型** - Group Commit 不保证提交顺序,建议使用 Sequence 列来保证数据一致性。 -* **max_filter_ratio 支持** - - 默认导入中,`filter_ratio` 通过失败行数和总行数计算。 - - Group Commit 模式下,`max_filter_ratio` 在总行数不超过 `group_commit_memory_rows_for_max_filter_ratio` 时有效。 - * **WAL 限制** - `async_mode` 写入会将数据写入 WAL,成功后删除,失败时通过 WAL 恢复。 - WAL 文件是单副本存储的,如果对应磁盘损坏或文件误删可能导致数据丢失。 @@ -641,21 +631,21 @@ PROPERTIES ( #### 30 并发 sync 模式 5 个 BE3 副本性能测试 -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 891.8 | 701.1 | 400.0 | 237.5 | |enable_nereids_planner=false| 885.8 | 688.1 | 398.7 | 232.9 | #### 100 并发 sync 模式 5 个 BE3 副本性能测试 -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 2427.8 | 2068.9 | 1259.4 | 764.9 | |enable_nereids_planner=false| 2320.4 | 1899.3 | 1206.2 |749.7| #### 500 并发 sync 模式 5 个 BE3 副本性能测试 -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 5567.5 | 5713.2 | 4681.0 | 3131.2 | |enable_nereids_planner=false| 4471.6 | 5042.5 | 4932.2 | 3641.1 | @@ -719,21 +709,21 @@ PROPERTIES ( #### 30 并发 sync 模式 5 个 BE3 副本性能测试 -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 9.1K | 11.1K | 11.4K | 11.1K | |enable_nereids_planner=false| 157.8K | 159.9K | 154.1K | 120.4K | #### 100 并发 sync 模式 5 个 BE3 副本性能测试 -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 10.0K |9.2K | 8.9K | 8.9K | |enable_nereids_planner=false| 130.4k | 131.0K | 130.4K | 124.1K | #### 500 并发 sync 模式 5 个 BE3 副本性能测试 -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 2.5K | 2.5K | 2.3K | 2.1K | |enable_nereids_planner=false| 94.2K | 95.1K | 94.4K | 94.8K | diff --git a/versioned_docs/version-2.1/admin-manual/config/be-config.md b/versioned_docs/version-2.1/admin-manual/config/be-config.md index 7acbb528fa8..abd9b0529f2 100644 --- a/versioned_docs/version-2.1/admin-manual/config/be-config.md +++ b/versioned_docs/version-2.1/admin-manual/config/be-config.md @@ -1513,11 +1513,6 @@ This configuration is supported since the Apache Doris 1.2 version group_commit_wal_path=/data1/storage/wal;/data2/storage/wal;/data3/storage/wal ``` -#### `group_commit_memory_rows_for_max_filter_ratio` - -* Description: The `max_filter_ratio` limit can only work if the total rows of `group commit` is less than this value. See [Group Commit](../../data-operate/import/group-commit-manual.md) for more details -* Default: 10000 - #### `default_tzfiles_path` * Description: Doris comes with its own time zone database. If the time zone file is not found in the system directory, the data in that directory is enabled. diff --git a/versioned_docs/version-2.1/data-operate/import/group-commit-manual.md b/versioned_docs/version-2.1/data-operate/import/group-commit-manual.md index a24fccecd71..92e787b6960 100644 --- a/versioned_docs/version-2.1/data-operate/import/group-commit-manual.md +++ b/versioned_docs/version-2.1/data-operate/import/group-commit-manual.md @@ -437,12 +437,6 @@ Recommend balancing based on system memory resources and data reliability requir group_commit_wal_path=/data1/storage/wal;/data2/storage/wal;/data3/storage/wal ``` -2. `group_commit_memory_rows_for_max_filter_ratio` - - * Description: `max_filter_ratio` works normally when group commit load total rows don't exceed this value, otherwise it doesn't work - - * Default: 10000 - ## Usage Limitations * **Group Commit Limitations** @@ -463,10 +457,6 @@ Recommend balancing based on system memory resources and data reliability requir * **Unique Model** - Group Commit doesn't guarantee commit order, recommend using Sequence column to ensure data consistency. -* **max_filter_ratio Support** - - In default loads, `filter_ratio` is calculated through failed rows and total rows. - - In Group Commit mode, `max_filter_ratio` works when total rows don't exceed `group_commit_memory_rows_for_max_filter_ratio`. - * **WAL Limitations** - `async_mode` writes data to WAL, deletes after success, recovers through WAL on failure. - WAL files are stored with a single replica on one BE, disk damage or accidental file deletion may cause data loss. @@ -687,18 +677,18 @@ Ensure that the imported values match the data types one by one. **Performance Test with 30 Concurrent Users in Sync Mode, 5 BEs, and 3 Replicas** -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| | | 92.2K | 85.9K | 84K | 83.2K | **Performance Test with 100 Concurrent Users in Sync Mode, 5 BEs, and 3 Replicas** -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| | | 70.4K |70.5K | 73.2K | 69.4K | **Performance Test with 500 Concurrent Users in Sync Mode, 5 BEs, and 3 Replicas** -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| | | 46.3K | 47.7K | 47.4K | 46.5K | diff --git a/versioned_docs/version-3.0/admin-manual/config/be-config.md b/versioned_docs/version-3.0/admin-manual/config/be-config.md index 2a6ae232239..abffa7e09c0 100644 --- a/versioned_docs/version-3.0/admin-manual/config/be-config.md +++ b/versioned_docs/version-3.0/admin-manual/config/be-config.md @@ -1360,11 +1360,6 @@ This configuration is supported since the Apache Doris 3.0.4 version group_commit_wal_path=/data1/storage/wal;/data2/storage/wal;/data3/storage/wal ``` -#### `group_commit_memory_rows_for_max_filter_ratio` - -* Description: The `max_filter_ratio` limit can only work if the total rows of `group commit` is less than this value. See [Group Commit](../../data-operate/import/group-commit-manual.md) for more details -* Default: 10000 - ### Compute and Storage Disaggregated Mode #### `deploy_mode` diff --git a/versioned_docs/version-3.0/data-operate/import/group-commit-manual.md b/versioned_docs/version-3.0/data-operate/import/group-commit-manual.md index d87ec074069..0b65dc066d7 100644 --- a/versioned_docs/version-3.0/data-operate/import/group-commit-manual.md +++ b/versioned_docs/version-3.0/data-operate/import/group-commit-manual.md @@ -437,12 +437,6 @@ Recommend balancing based on system memory resources and data reliability requir group_commit_wal_path=/data1/storage/wal;/data2/storage/wal;/data3/storage/wal ``` -2. `group_commit_memory_rows_for_max_filter_ratio` - - * Description: `max_filter_ratio` works normally when group commit load total rows don't exceed this value, otherwise it doesn't work - - * Default: 10000 - ## Usage Limitations * **Group Commit Limitations** @@ -463,10 +457,6 @@ Recommend balancing based on system memory resources and data reliability requir * **Unique Model** - Group Commit doesn't guarantee commit order, recommend using Sequence column to ensure data consistency. -* **max_filter_ratio Support** - - In default loads, `filter_ratio` is calculated through failed rows and total rows. - - In Group Commit mode, `max_filter_ratio` works when total rows don't exceed `group_commit_memory_rows_for_max_filter_ratio`. - * **WAL Limitations** - `async_mode` writes data to WAL, deletes after success, recovers through WAL on failure. - WAL files are stored with a single replica on one BE, disk damage or accidental file deletion may cause data loss. @@ -647,21 +637,21 @@ JMeter Parameter Settings as Shown in the Images **Performance Test with 30 Concurrent Users in Sync Mode, 5 BEs, and 3 Replicas** - | Group commit internal | 10ms | 20ms | 50ms | 100ms | + | Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 891.8 | 701.1 | 400.0 | 237.5 | |enable_nereids_planner=false| 885.8 | 688.1 | 398.7 | 232.9 | **Performance Test with 100 Concurrent Users in Sync Mode, 5 BEs, and 3 Replicas** - | Group commit internal | 10ms | 20ms | 50ms | 100ms | + | Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 2427.8 | 2068.9 | 1259.4 | 764.9 | |enable_nereids_planner=false| 2320.4 | 1899.3 | 1206.2 |749.7| **Performance Test with 500 Concurrent Users in Sync Mode, 5 BEs, and 3 Replicas** - | Group commit internal | 10ms | 20ms | 50ms | 100ms | + | Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 5567.5 | 5713.2 | 4681.0 | 3131.2 | |enable_nereids_planner=false| 4471.6 | 5042.5 | 4932.2 | 3641.1 | @@ -698,21 +688,21 @@ JMeter Parameter Settings as Shown in the Images **Performance Test with 30 Concurrent Users in Sync Mode, 5 BEs, and 3 Replicas** -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 9.1K | 11.1K | 11.4K | 11.1K | |enable_nereids_planner=false| 157.8K | 159.9K | 154.1K | 120.4K | **Performance Test with 100 Concurrent Users in Sync Mode, 5 BEs, and 3 Replicas** -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 10.0K |9.2K | 8.9K | 8.9K | |enable_nereids_planner=false| 130.4k | 131.0K | 130.4K | 124.1K | **Performance Test with 500 Concurrent Users in Sync Mode, 5 BEs, and 3 Replicas** -| Group commit internal | 10ms | 20ms | 50ms | 100ms | +| Group commit interval | 10ms | 20ms | 50ms | 100ms | |-----------------------|---------------|---------------|---------------|---------------| |enable_nereids_planner=true| 2.5K | 2.5K | 2.3K | 2.1K | |enable_nereids_planner=false| 94.2K | 95.1K | 94.4K | 94.8K | --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org