This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new ec42ad9fe1 add workload management docs (#1028)
ec42ad9fe1 is described below
commit ec42ad9fe16bd96407205eaa6a8772f2e4b01aab
Author: wangbo <wan...@apache.org>
AuthorDate: Thu Aug 22 00:13:53 2024 +0800
add workload management docs (#1028)
---
.../resource-admin/use-workload-cpu-limit.md | 82 ++++++++++++
.../resource-admin/use-workload-local-io-limit.md | 133 +++++++++++++++++++
.../resource-admin/use-workload-mem-limit.md | 130 ++++++++++++++++++
.../resource-admin/use-workload-remote-io-limit.md | 145 ++++++++++++++++++++
.../admin-manual/resource-admin/workload-fusing.md | 100 ++++++++++++++
.../resource-admin/use-workload-cpu-limit.md | 77 +++++++++++
.../resource-admin/use-workload-local-io-limit.md | 135 +++++++++++++++++++
.../resource-admin/use-workload-mem-limit.md | 132 +++++++++++++++++++
.../resource-admin/use-workload-remote-io-limit.md | 146 +++++++++++++++++++++
.../admin-manual/resource-admin/workload-fusing.md | 99 ++++++++++++++
sidebars.json | 7 +-
static/images/workload-management/use_wg_cpu_1.png | Bin 0 -> 58448 bytes
static/images/workload-management/use_wg_cpu_2.png | Bin 0 -> 69564 bytes
static/images/workload-management/use_wg_io_1.png | Bin 0 -> 228840 bytes
static/images/workload-management/use_wg_io_2.png | Bin 0 -> 147990 bytes
static/images/workload-management/use_wg_rio_1.png | Bin 0 -> 62269 bytes
static/images/workload-management/use_wg_rio_2.png | Bin 0 -> 58419 bytes
17 files changed, 1185 insertions(+), 1 deletion(-)
diff --git a/docs/admin-manual/resource-admin/use-workload-cpu-limit.md
b/docs/admin-manual/resource-admin/use-workload-cpu-limit.md
new file mode 100644
index 0000000000..99e8668198
--- /dev/null
+++ b/docs/admin-manual/resource-admin/use-workload-cpu-limit.md
@@ -0,0 +1,82 @@
+---
+{
+"title": "Use Workload CPU Limit",
+"language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Use Workload Group limit CPU
+
+The workloads in Doris can generally be divided into three categories:
+
+Core Report Queries: These are typically used by company executives to view
reports. The load may not be very high, but the availability requirements are
high. These queries can be grouped together and assigned a higher-priority soft
limit, ensuring they receive more CPU resources when CPU availability is
limited.
+
+Ad-hoc Queries: These queries are usually exploratory, with random SQL and
unpredictable resource usage. They typically have a lower priority, so they can
be managed with a CPU hard limit set to a lower value, preventing them from
consuming excessive CPU resources and reducing cluster availability.
+
+ETL Queries: These queries have relatively fixed SQL and stable resource
usage, though occasionally, a sudden increase in upstream data volume can cause
a spike in resource usage. Therefore, they can be managed with a CPU hard limit
configuration.
+
+Different workloads consume CPU resources differently, and users have varying
requirements for response latency. When the BE's CPU is heavily utilized,
availability decreases and response latency increases. For example, an ad-hoc
analytical query might fully utilize the cluster's CPU, causing increased
latency for core reports and impacting the SLA. Therefore, a CPU isolation
mechanism is needed to separate different types of workloads, ensuring cluster
availability and meeting SLAs.
+
+Workload Groups support both soft and hard CPU limits, with a current
recommendation to configure Workload Groups with hard limits in production
environments. This is because soft limits only come into effect when the CPU is
fully utilized, but when the CPU is maxed out, internal components of Doris
(e.g., RPC components) and available CPU resources for the operating system are
reduced, leading to a significant decline in overall cluster availability.
Therefore, in production environment [...]
+
+## CPU hard limit Test
+
+### Test env
+1FE, 1BE(96 cores)
+Test data is clickbench，sql is q29.
+
+### Test
+1. Run query in 3 concurrent, using top command we can see it uses 76 cores.
+
+
+
+2. Alter workload group.
+```
+alter workload group g2 properties('cpu_hard_limit'='10%');
+```
+
+3. Enable cpu hard limit，then all workload group could convert to hard limit.
+```
+ADMIN SET FRONTEND CONFIG ("enable_cpu_hard_limit" = "true");
+```
+
+4. Test again, the BE using 9 ~ 10 cores, about 10%.
+
+
+
+It should be noted that it's best to use query workloads for testing here, as
they are more likely to reflect the intended effects. If you use
high-throughput data import instead, it may trigger compaction, causing the
observed values to be higher than the configured Workload Group limits.
Currently, the compaction workload is not managed by the Workload Group.
+
+5. Use system table to show cpu usage, it's about 10%;
+```
+mysql [information_schema]>select CPU_USAGE_PERCENT from
workload_group_resource_usage where WORKLOAD_GROUP_ID=11201;
++-------------------+
+| CPU_USAGE_PERCENT |
++-------------------+
+| 9.57 |
++-------------------+
+1 row in set (0.02 sec)
+```
+
+## NOTE
+1. When configuring in practice, it's best not to have the total CPU
allocation of all Groups add up to exactly 100%. This is primarily to ensure
availability in low-latency scenarios, as some resources need to be reserved
for other components. However, if the scenario is not sensitive to latency and
the goal is maximum resource utilization, you can consider configuring the
total CPU allocation of all Groups to equal 100%.
+
+2. Currently, the time interval for the Frontend (FE) to synchronize Workload
Group metadata to the Backend (BE) is 30 seconds, so it may take up to 30
seconds for changes to the Workload Group to take effect.
\ No newline at end of file
diff --git a/docs/admin-manual/resource-admin/use-workload-local-io-limit.md
b/docs/admin-manual/resource-admin/use-workload-local-io-limit.md
new file mode 100644
index 0000000000..475ffd7584
--- /dev/null
+++ b/docs/admin-manual/resource-admin/use-workload-local-io-limit.md
@@ -0,0 +1,133 @@
+---
+{
+"title": "Use Workload Local IO Limit",
+"language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Use Workload Group limit local IO
+
+In OLAP systems, when performing ETL or large ad-hoc queries, a significant
amount of data needs to be read. To speed up data analysis, Doris internally
uses multithreading to scan multiple disk files in parallel, which generates a
large amount of disk I/O and can negatively impact other queries, such as
report analysis.
+
+By using Workload Groups, you can group offline ETL data processing and online
report queries separately and limit the I/O bandwidth for offline data
processing, thereby reducing its impact on online report analysis.
+
+## Test limit local IO
+### Test
+1FE,1BE(96 cores), test data is clickbench
+
+### Not limit IO
+1. Clear cache.
+```
+// clear OS cache.
+sync; echo 3 > /proc/sys/vm/drop_caches
+
+// disable BE's cache.
+disable_storage_page_cache = true
+```
+
+2. Run query one by one.
+```
+set dry_run_query = true;
+select * from hits.hits;
+```
+
+3. Show local IO by system table, is's 3G/s.
+```
+mysql [information_schema]>select LOCAL_SCAN_BYTES_PER_SECOND / 1024 / 1024 as
mb_per_sec from workload_group_resource_usage where WORKLOAD_GROUP_ID=11201;
++--------------------+
+| mb_per_sec |
++--------------------+
+| 1146.6208400726318 |
++--------------------+
+1 row in set (0.03 sec)
+
+mysql [information_schema]>select LOCAL_SCAN_BYTES_PER_SECOND / 1024 / 1024 as
mb_per_sec from workload_group_resource_usage where WORKLOAD_GROUP_ID=11201;
++--------------------+
+| mb_per_sec |
++--------------------+
+| 3496.2762966156006 |
++--------------------+
+1 row in set (0.04 sec)
+
+mysql [information_schema]>select LOCAL_SCAN_BYTES_PER_SECOND / 1024 / 1024 as
mb_per_sec from workload_group_resource_usage where WORKLOAD_GROUP_ID=11201;
++--------------------+
+| mb_per_sec |
++--------------------+
+| 2192.7690029144287 |
++--------------------+
+1 row in set (0.02 sec)
+```
+
+4.Show IO by pidstat, the first column in picture is process id, the second
column is IO(kb/s), it's 2G/s.
+
+
+
+
+### Test IO limit.
+1. Clear cache.
+```
+// clear os cache
+sync; echo 3 > /proc/sys/vm/drop_caches
+
+// disable BE cache
+disable_storage_page_cache = true
+```
+
+2. Alter workload group.
+```
+alter workload group g2 properties('read_bytes_per_second'='104857600');
+```
+
+3. Show IO by system table, it's about 98M/s.
+```
+mysql [information_schema]>select LOCAL_SCAN_BYTES_PER_SECOND / 1024 / 1024 as
mb_per_sec from workload_group_resource_usage where WORKLOAD_GROUP_ID=11201;
++--------------------+
+| mb_per_sec |
++--------------------+
+| 97.94296646118164 |
++--------------------+
+1 row in set (0.03 sec)
+
+mysql [information_schema]>select LOCAL_SCAN_BYTES_PER_SECOND / 1024 / 1024 as
mb_per_sec from workload_group_resource_usage where WORKLOAD_GROUP_ID=11201;
++--------------------+
+| mb_per_sec |
++--------------------+
+| 98.37584781646729 |
++--------------------+
+1 row in set (0.04 sec)
+
+mysql [information_schema]>select LOCAL_SCAN_BYTES_PER_SECOND / 1024 / 1024 as
mb_per_sec from workload_group_resource_usage where WORKLOAD_GROUP_ID=11201;
++--------------------+
+| mb_per_sec |
++--------------------+
+| 98.06641292572021 |
++--------------------+
+1 row in set (0.02 sec)
+```
+
+4. Show IO by pidstat, the process IO is about 131M/s。
+
+
+
+## NOTE
+1. The LOCAL_SCAN_BYTES_PER_SECOND field in the system table represents the
aggregated statistics at the process level for the current Workload Group. For
example, if 12 file paths are configured, LOCAL_SCAN_BYTES_PER_SECOND
represents the maximum I/O value across these 12 file paths. If you want to see
the I/O throughput for each file path individually, you can view detailed
values in Grafana or through the BE's bvar monitoring.
+2. Due to the presence of the operating system's and Doris's Page Cache, the
I/O values observed using Linux's I/O monitoring scripts are usually smaller
than those seen in the system table.
\ No newline at end of file
diff --git a/docs/admin-manual/resource-admin/use-workload-mem-limit.md
b/docs/admin-manual/resource-admin/use-workload-mem-limit.md
new file mode 100644
index 0000000000..6ec6b70e79
--- /dev/null
+++ b/docs/admin-manual/resource-admin/use-workload-mem-limit.md
@@ -0,0 +1,130 @@
+---
+{
+"title": "Use Workload Memory Limit",
+"language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Use Workload Group limit memory
+
+Ad-hoc queries typically have unpredictable SQL input and uncertain memory
usage, which carries the risk of a few queries consuming a large amount of
memory. This type of load can be assigned to a separate group. By using the
hard memory limits feature of the Workload Group, sudden large queries can be
prevented from consuming all available memory, which would otherwise leave no
memory for other queries or cause an Out of Memory (OOM) error.
+
+When the memory usage of this Workload Group exceeds the configured hard
limit, memory will be freed by killing queries, thus preventing the process
memory from being completely consumed.
+
+## Test hard memory limit
+### Test env
+1FE，1BE，BE(96 cores)，memory is 375G。
+
+Test data is clickbench, run q29 in 3 concurrent.
+
+### Not using Workload Group's memory limit.
+1. Show process memory usage, ps shows memory usage, the memory is 7.7G.
+```
+[ ~]$ ps -eo pid,comm,%mem,rss | grep 1407481
+1407481 doris_be 2.0 7896792
+[ ~]$ ps -eo pid,comm,%mem,rss | grep 1407481
+1407481 doris_be 2.0 7929692
+[ ~]$ ps -eo pid,comm,%mem,rss | grep 1407481
+1407481 doris_be 2.0 8101232
+```
+
+2. Show Workload Group memory by system table, it's 5.8G.
+```
+mysql [information_schema]>select MEMORY_USAGE_BYTES / 1024/ 1024 as
wg_mem_used_mb from workload_group_resource_usage where workload_group_id=11201;
++-------------------+
+| wg_mem_used_mb |
++-------------------+
+| 5797.524360656738 |
++-------------------+
+1 row in set (0.01 sec)
+
+mysql [information_schema]>select MEMORY_USAGE_BYTES / 1024/ 1024 as
wg_mem_used_mb from workload_group_resource_usage where workload_group_id=11201;
++-------------------+
+| wg_mem_used_mb |
++-------------------+
+| 5840.246627807617 |
++-------------------+
+1 row in set (0.02 sec)
+
+mysql [information_schema]>select MEMORY_USAGE_BYTES / 1024/ 1024 as
wg_mem_used_mb from workload_group_resource_usage where workload_group_id=11201;
++-------------------+
+| wg_mem_used_mb |
++-------------------+
+| 5878.394917488098 |
++-------------------+
+1 row in set (0.02 sec)
+```
+Here, you can see that the memory usage of the process is usually much greater
than the memory usage of a single Workload Group, even if only one Workload
Group is running in the process. This is because the Workload Group only
accounts for the memory used by queries and some parts of data import. Other
components within the process, such as metadata and various caches, are not
included in the Workload Group's memory usage and are not managed by the
Workload Group.
+
+### Use Workload Group limit memory
+1. Alter workload group.
+```
+alter workload group g2 properties('memory_limit'='0.5%');
+alter workload group g2 properties('enable_memory_overcommit'='false');
+```
+
+2. Run test, the workload group uses 1.5G memory.
+```
+mysql [information_schema]>select MEMORY_USAGE_BYTES / 1024/ 1024 as
wg_mem_used_mb from workload_group_resource_usage where workload_group_id=11201;
++--------------------+
+| wg_mem_used_mb |
++--------------------+
+| 1575.3877239227295 |
++--------------------+
+1 row in set (0.02 sec)
+
+mysql [information_schema]>select MEMORY_USAGE_BYTES / 1024/ 1024 as
wg_mem_used_mb from workload_group_resource_usage where workload_group_id=11201;
++------------------+
+| wg_mem_used_mb |
++------------------+
+| 1668.77405834198 |
++------------------+
+1 row in set (0.01 sec)
+
+mysql [information_schema]>select MEMORY_USAGE_BYTES / 1024/ 1024 as
wg_mem_used_mb from workload_group_resource_usage where workload_group_id=11201;
++--------------------+
+| wg_mem_used_mb |
++--------------------+
+| 499.96760272979736 |
++--------------------+
+1 row in set (0.01 sec)
+```
+
+3. Show memory by ps command, the max memory is 3.8G.
+```
+[ ~]$ ps -eo pid,comm,%mem,rss | grep 1407481
+1407481 doris_be 1.0 4071364
+[ ~]$ ps -eo pid,comm,%mem,rss | grep 1407481
+1407481 doris_be 1.0 4059012
+[ ~]$ ps -eo pid,comm,%mem,rss | grep 1407481
+1407481 doris_be 1.0 4057068
+```
+
+4. There are many query failed because of oom.
+```
+1724074250162,14126,1c_sql,HY000 1105,"java.sql.SQLException: errCode = 2,
detailMessage = (127.0.0.1)[MEM_LIMIT_EXCEEDED]GC wg for hard limit, wg
id:11201, name:g2, used:1.71 GB, limit:1.69 GB, backend:10.16.10.8. cancel top
memory used tracker <Query#Id=4a0689936c444ac8-a0d01a50b944f6e7> consumption
1.71 GB. details:process memory used 3.01 GB exceed soft limit 304.41 GB or sys
available memory 101.16 GB less than warning water mark 12.80 GB., Execute
again after enough memory, details [...]
+```
+In this error message, you can see that the Workload Group used 1.7GB of
memory, while the Workload Group's limit is 1.69GB. The calculation works as
follows: 1.69GB = Physical machine memory (375GB) * mem_limit (value in
be.conf, default is 0.9) * 0.5% (Workload Group's configuration). This means
that the memory percentage configured in the Workload Group is calculated based
on the available memory of the Doris process.
+
+
+## Suggestions
+As demonstrated in the previous test, the hard limit can control the memory
usage of a Workload Group, but it releases memory by killing queries, which can
be a very unfriendly experience for users and may cause all queries to fail in
extreme cases. Therefore, in a production environment, it is recommended to use
memory hard limits in conjunction with query queuing. This approach can limit
memory usage while ensuring the success rate of queries.
\ No newline at end of file
diff --git a/docs/admin-manual/resource-admin/use-workload-remote-io-limit.md
b/docs/admin-manual/resource-admin/use-workload-remote-io-limit.md
new file mode 100644
index 0000000000..4f5972b5c9
--- /dev/null
+++ b/docs/admin-manual/resource-admin/use-workload-remote-io-limit.md
@@ -0,0 +1,145 @@
+---
+{
+"title": "Use Workload Remote IO Limit",
+"language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Use Workload Group limit remote IO
+
+BrokerLoad and S3Load are commonly used methods for importing large volumes of
data. Users can first upload data to HDFS or S3 and then use BrokerLoad and
S3Load for parallel data imports. To speed up the import process, Doris uses
multithreading to pull data from HDFS/S3 in parallel. This can put significant
pressure on HDFS/S3, potentially causing instability for other jobs running on
HDFS/S3.
+
+You can use the remote I/O limitation feature of Workload Groups to limit the
bandwidth used during the import process, thereby reducing the impact on other
operations.
+
+## Test limit remote IO
+### Test env
+1FE,1BE(16 cores, 64G), test data is clickbench,Before testing, the dataset
needs to be uploaded to S3. To save time, we will upload only 10 million rows
of data, and then use the TVF (Table-Valued Function) feature to query the data
from S3.
+
+Show schema info after upload.
+```
+DESC FUNCTION s3 (
+ "URI" = "https://bucketname/1kw.tsv";,
+ "s3.access_key"= "ak",
+ "s3.secret_key" = "sk",
+ "format" = "csv",
+ "use_path_style"="true"
+);
+```
+
+### Test not limit remote IO
+1. Run query one by one.
+```
+// just scan, not return value.
+set dry_run_query = true;
+
+SELECT * FROM s3(
+ "URI" = "https://bucketname/1kw.tsv";,
+ "s3.access_key"= "ak",
+ "s3.secret_key" = "sk",
+ "format" = "csv",
+ "use_path_style"="true"
+);
+```
+
+2. Show remote IO by system table,it's about 837M/s, It should be noted that
the actual I/O throughput here is significantly affected by the environment. If
the machine hosting the BE has a low bandwidth connection to external storage,
the actual throughput may be lower.
+```
+MySQL [(none)]> select cast(REMOTE_SCAN_BYTES_PER_SECOND/1024/1024 as int) as
read_mb from information_schema.workload_group_resource_usage;
++---------+
+| read_mb |
++---------+
+| 837 |
++---------+
+1 row in set (0.104 sec)
+
+MySQL [(none)]> select cast(REMOTE_SCAN_BYTES_PER_SECOND/1024/1024 as int) as
read_mb from information_schema.workload_group_resource_usage;
++---------+
+| read_mb |
++---------+
+| 867 |
++---------+
+1 row in set (0.070 sec)
+
+MySQL [(none)]> select cast(REMOTE_SCAN_BYTES_PER_SECOND/1024/1024 as int) as
read_mb from information_schema.workload_group_resource_usage;
++---------+
+| read_mb |
++---------+
+| 867 |
++---------+
+1 row in set (0.186 sec)
+```
+
+3. Using sar(sar -n DEV 1 3600) to show network bandwidth of the machine, the
max value is about 1033M/s.The first column of the output is the number of
bytes received per second by a certain network card of the current machine, in
KB per second.
+
+
+
+
+### Test limit remote IO
+1. Alter workload group.
+```
+alter workload group normal
properties('remote_read_bytes_per_second'='104857600');
+```
+
+2. Run query one by one.
+```
+// just scan not return.
+set dry_run_query = true;
+
+
+SELECT * FROM s3(
+ "URI" = "https://bucketname/1kw.tsv";,
+ "s3.access_key"= "ak",
+ "s3.secret_key" = "sk",
+ "format" = "csv",
+ "use_path_style"="true"
+);
+```
+
+3. Use the system table to check the current remote read I/O throughput. At
this moment, the I/O throughput is around 100M, with some fluctuation. This
fluctuation is influenced by the current algorithm design and typically
includes a peak, but it does not last long and is considered normal.
+```
+MySQL [(none)]> select cast(REMOTE_SCAN_BYTES_PER_SECOND/1024/1024 as int) as
read_mb from information_schema.workload_group_resource_usage;
++---------+
+| read_mb |
++---------+
+| 56 |
++---------+
+1 row in set (0.010 sec)
+
+MySQL [(none)]> select cast(REMOTE_SCAN_BYTES_PER_SECOND/1024/1024 as int) as
read_mb from information_schema.workload_group_resource_usage;
++---------+
+| read_mb |
++---------+
+| 131 |
++---------+
+1 row in set (0.009 sec)
+
+MySQL [(none)]> select cast(REMOTE_SCAN_BYTES_PER_SECOND/1024/1024 as int) as
read_mb from information_schema.workload_group_resource_usage;
++---------+
+| read_mb |
++---------+
+| 111 |
++---------+
+1 row in set (0.009 sec)
+```
+
+4. Using sar(sar -n DEV 1 3600) to show network bandwidth, the max IO is about
207M, This indicates that remote limit IO works. However, since the sar command
shows machine-level traffic, the values may be higher than those reported by
Doris.
+
+
diff --git a/docs/admin-manual/resource-admin/workload-fusing.md
b/docs/admin-manual/resource-admin/workload-fusing.md
new file mode 100644
index 0000000000..62622235f9
--- /dev/null
+++ b/docs/admin-manual/resource-admin/workload-fusing.md
@@ -0,0 +1,100 @@
+---
+{
+"title": "Workload Fusing",
+"language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Workload Fusing
+
+Workload Fusing is typically applied in scenarios where certain queries
excessively consume resources, leading to a decline in the overall availability
of the online cluster. In such cases, the cluster can be restored to normal
operation by detecting and blocking the abnormal big queries during runtime.
+
+If users have already grouped their workload based on query latency, such as
assigning big queries to one Workload Group and small queries to another, the
big query circuit breaking method can also be used to improve the availability
of the small query group.
+
+Currently, the big query circuit breaking strategy is mainly implemented
through the Workload Policy feature, allowing users to configure specific
policies to achieve automatic circuit breaking of big queries.
+
+## Workload Fusing Test
+
+### Test Env
+1FE,1BE(96 cores), test data is clickbench.
+
+### Workload Fusing By Memory
+1. Test sql is q29, the query 's peak memory is（peakMemoryBytes）8G in
fe.audit.log.
+```
+|User=root|Ctl=internal|Db=hits|State=EOF|ErrorCode=0|ErrorMessage=|Time(ms)=4614|ScanBytes=13107527680|ScanRows=81032736|ReturnRows=11|StmtId=526|QueryId=e5b6c62d624146e4-b7291221492a7cc2|IsQuery=true|isNereids=true|feIp=10.16.10.8|StmtType=SELECT|Stmt=SELECT
REGEXP_REPLACE(Referer, '^https?://(?:www\.)?([^/]+)/.*$', '\1') AS k,
AVG(length(Referer)) AS l, COUNT(*) AS c, MIN(Referer) FROM hits WHERE Referer
<> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT
25|CpuTimeMS=1056 [...]
+```
+
+2. Create a policy which kill the query which using memory exceeds 1G.
+```
+create workload policy cancel_1g_query conditions(query_be_memory_bytes >
1073741824) actions(cancel_query);
+
+mysql [information_schema]>select * from workload_policy;
++-------+-----------------+------------------------------------+--------------+----------+---------+---------+----------------+
+| ID | NAME | CONDITION | ACTION
| PRIORITY | ENABLED | VERSION | WORKLOAD_GROUP |
++-------+-----------------+------------------------------------+--------------+----------+---------+---------+----------------+
+| 11313 | cancel_1g_query | query_be_memory_bytes > 1073741824 | cancel_query
| 0 | 1 | 0 | |
++-------+-----------------+------------------------------------+--------------+----------+---------+---------+----------------+
+1 row in set (0.02 sec)
+```
+
+3. Test again, we can see query is killed by policy cancel_1g_query.
+```
+ mysql [information_schema]>SELECT REGEXP_REPLACE(Referer,
'^https?://(?:www\.)?([^/]+)/.*$', '\1') AS k, AVG(length(Referer)) AS l,
COUNT(*) AS c, MIN(Referer) FROM hits.hits WHERE Referer <> '' GROUP BY k
HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;
+ERROR 1105 (HY000): errCode = 2, detailMessage =
(127.0.0.1)[INTERNAL_ERROR]query ceb0553d359e454f-8939599b2e602d07 cancelled by
workload policy: cancel_1g_query, id:11313
+```
+
+### Workload Fusing By Scan Rows
+1. Create the policy.
+```
+// disable other policy first.
+alter workload policy cancel_1g_query properties('enabled'='false');
+
+create workload policy canel_scan_5kw_query conditions(be_scan_rows >
50000000) actions(cancel_query);
+```
+
+2. Test query, the query failed because of cancel_1g_query.
+```
+mysql [hits]>SELECT REGEXP_REPLACE(Referer, '^https?://(?:www\.)?([^/]+)/.*$',
'\1') AS k, AVG(length(Referer)) AS l, COUNT(*) AS c, MIN(Referer) FROM
hits.hits WHERE Referer <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l
DESC LIMIT 25;
+
+ERROR 1105 (HY000): errCode = 2, detailMessage =
(127.0.0.1)[INTERNAL_ERROR]query f9d15001bfc94322-9b63a8b84aef9dee cancelled by
workload policy: canel_scan_5kw_query, id:11314
+```
+It is important to note that be_scan_rows mainly tracks the amount of data
scanned by a query on a single BE. In a production environment, the data for a
table is usually distributed across multiple BEs, and the number of rows
scanned by a query as seen in the audit is the sum from all BEs.
+
+Therefore, when a query fails due to the SCAN data volume, the number of
scanned rows in the audit is typically much larger than the value configured in
the Policy.
+
+### Workload Fusing By Query Time
+1. Create the policy.
+```
+// disable other policy first.
+alter workload policy canel_scan_5kw_query properties('enabled'='false');
+
+create workload policy cancel_time_1s_query conditions(query_time > 1000)
actions(cancel_query);
+```
+2. Test query, it may be failed because of cancel_time_1s_query.
+```
+mysql [hits]>SELECT REGEXP_REPLACE(Referer, '^https?://(?:www\.)?([^/]+)/.*$',
'\1') AS k, AVG(length(Referer)) AS l, COUNT(*) AS c, MIN(Referer) FROM
hits.hits WHERE Referer <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l
DESC LIMIT 25;
+ERROR 1105 (HY000): errCode = 2, detailMessage =
(127.0.0.1)[INTERNAL_ERROR]query c6aebdf403e24c62-965daf0bdff179f7 cancelled by
workload policy: cancel_time_1s_query, id:11315
+```
+
+### NOTE
+1. Currently, the time interval for the Frontend (FE) to synchronize Workload
Policy metadata to the Backend (BE) is 30 seconds, meaning that it may take up
to 30 seconds for any policy changes to take effect.
+2. Currently, the Backend (BE) executes the strategies configured in the
Policy through a scheduled thread with a time interval of 500ms. This means
that if a query is very short and completes within 500ms, it may not be subject
to the Workload Policy constraints.
\ No newline at end of file
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/resource-admin/use-workload-cpu-limit.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/resource-admin/use-workload-cpu-limit.md
new file mode 100644
index 0000000000..180165c56b
--- /dev/null
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/resource-admin/use-workload-cpu-limit.md
@@ -0,0 +1,77 @@
+---
+{
+"title": "Use Workload CPU Limit",
+"language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# 使用Workload Group管理CPU资源
+
+Doris 的负载大体可以分为三类：
+1.
核心报表查询，通常给公司高层查看报表使用，负载不一定很高，但是对可用性要求较高，这类查询可以划分到一个分组，配置较高优先级的软限，保证CPU资源不够时可以获得更多的CPU资源。
+2.
Adhoc类查询，这类查询通常偏探索分析，SQL比较随机，具体的资源用量也比较未知，优先级通常不高。因此可以使用CPU硬限进行管理，并配置较低的值，避免占用过多CPU资源降低集群可用性。
+3. ETL类查询，这类查询的SQL比较固定，资源用量通常也比较稳定，偶尔会出现上游数据量增长导致资源用量暴涨的情况，因此可以使用CPU硬限进行配置。
+
+不同的负载对CPU的消耗不一样，用户对响应延时的需求也不一样。当BE的CPU
被用的很满时，可用性会变差，响应延时会变高。比如可能一个Adhoc的分析类查询把整个集群的CPU打满，导致核心报表的延时变大，影响到了SLA。所以需要CPU
隔离机制来对不同的业务进行隔离，保障集群的可用性和SLA。
+Workload Group支持CPU的软限和硬限，目前比较推荐在线上环境把Workload
Group配置成硬限。原因是CPU的软限通常在CPU被打满时才能体现出优先级的作用，但是在CPU被用满时，Doris的内部组件（例如rpc组件）以及操作系统可用的CPU会减少，此时集群整体的可用性是下降比较严重的，因此生产环境通常需要避免CPU资源被打满的情况，当然其他资源也一样，内存资源同理。
+
+## CPU硬限测试
+
+### 测试环境
+1FE，1BE，96核机器。
+数据集为clickbench，测试sql为q29。
+
+### 测试
+1.
使用jmeter发起3并发查询，把BE进程的CPU使用压到比较高的使用率，这里测试的机器是96核，使用top命令看到BE进程CPU使用率为7600%的含义是该进程目前使用中的核数是76个。
+
+
+
+2. 修改使用中的Workload Group的CPU硬限为10%。
+```
+alter workload group g2 properties('cpu_hard_limit'='10%');
+```
+
+3. 集群开启硬限模式，此时集群中所有Group都会切换为硬限。
+```
+ADMIN SET FRONTEND CONFIG ("enable_cpu_hard_limit" = "true");
+```
+
+4. 重新压测查询负载，可以看到当前进程只能使用9到10个核，占总核数的10%左右。
+
+
+
+需要注意的是，这里的测试最好使用查询负载会比较能体现出效果，因为如果是高吞吐导入的话，可能会触发Compaction，使得实际观测的值要比Workload
Group配置的值大。而Compaction的负载目前是没有归入Workload Group的管理的。
+
+5. 除了使用Linux 的系统命令外，还可以通过使用Doris的系统表观察Group目前的CPU使用为10%左右。
+```
+mysql [information_schema]>select CPU_USAGE_PERCENT from
workload_group_resource_usage where WORKLOAD_GROUP_ID=11201;
++-------------------+
+| CPU_USAGE_PERCENT |
++-------------------+
+| 9.57 |
++-------------------+
+1 row in set (0.02 sec)
+```
+
+## 注意事项
+1.
在实际配置的时候，所有Group的CPU累加值最好不要正好等于100%，这主要是为了保证低延迟场景的可用性。因为需要让出一部分资源给其他组件使用。当然如果对延迟不是很敏感的场景，期望最高的资源利用率，那么可以考虑所有Group的CPU累加值配置等于100%。
+2. 目前FE向BE同步Workload Group元数据的时间间隔为30秒，因此对于Workload Group的变更最大需要等待30秒才能生效。
\ No newline at end of file
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/resource-admin/use-workload-local-io-limit.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/resource-admin/use-workload-local-io-limit.md
new file mode 100644
index 0000000000..0f12a7958a
--- /dev/null
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/resource-admin/use-workload-local-io-limit.md
@@ -0,0 +1,135 @@
+---
+{
+"title": "Use Workload Local IO Limit",
+"language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# 使用Workload Group管理本地IO
+
+OLAP 系统在做ETL或者大的Adhoc 查询时，需要读取大量的数据，Doris
为了加速数据分析过程，内部会使用多线程并行的方式对多个磁盘文件扫描，会产生巨大的磁盘IO，就会对其他的查询（比如报表分析）产生影响。
+可以通过Workload Group 对离线的ETL数据处理和在线的报表查询做分组，限制离线数据处理IO带宽的方式，降低它对在线报表分析的影响。
+
+## 测试本地IO硬限
+### 测试环境
+1FE,1BE, 配置为96核。
+
+测试数据集为clickbench。
+
+### 不开启IO硬限测试
+1. 关闭缓存。
+```
+// 清空操作系统缓存
+sync; echo 3 > /proc/sys/vm/drop_caches
+
+// 禁用BE的page cache
+disable_storage_page_cache = true
+```
+
+2. 对clickbench的表执行全表扫描，执行单并发查询即可。
+```
+set dry_run_query = true;
+select * from hits.hits;
+```
+
+3. 通过Doris的内表查看当前Group的最大吞吐为3GB每秒。
+```
+mysql [information_schema]>select LOCAL_SCAN_BYTES_PER_SECOND / 1024 / 1024 as
mb_per_sec from workload_group_resource_usage where WORKLOAD_GROUP_ID=11201;
++--------------------+
+| mb_per_sec |
++--------------------+
+| 1146.6208400726318 |
++--------------------+
+1 row in set (0.03 sec)
+
+mysql [information_schema]>select LOCAL_SCAN_BYTES_PER_SECOND / 1024 / 1024 as
mb_per_sec from workload_group_resource_usage where WORKLOAD_GROUP_ID=11201;
++--------------------+
+| mb_per_sec |
++--------------------+
+| 3496.2762966156006 |
++--------------------+
+1 row in set (0.04 sec)
+
+mysql [information_schema]>select LOCAL_SCAN_BYTES_PER_SECOND / 1024 / 1024 as
mb_per_sec from workload_group_resource_usage where WORKLOAD_GROUP_ID=11201;
++--------------------+
+| mb_per_sec |
++--------------------+
+| 2192.7690029144287 |
++--------------------+
+1 row in set (0.02 sec)
+```
+
+4. 使用pidstat命令查看进程IO，图中第一列是进程id，第二列是读IO的吞吐（单位是kb/s）。可以看到不限制IO时，最大吞吐为2G每秒。
+
+
+
+
+### 开启IO硬限后测试
+1. 关闭缓存。
+```
+// 清空操作系统缓存
+sync; echo 3 > /proc/sys/vm/drop_caches
+
+// 禁用BE的page cache
+disable_storage_page_cache = true
+```
+
+2. 修改Workload Group的配置，限制每秒最大吞吐为100M。
+```
+// 限制当前Group的读吞吐为每秒100M
+alter workload group g2 properties('read_bytes_per_second'='104857600');
+```
+
+3. 使用Doris系统表查看Workload Group的最大IO吞吐为每秒98M。
+```
+mysql [information_schema]>select LOCAL_SCAN_BYTES_PER_SECOND / 1024 / 1024 as
mb_per_sec from workload_group_resource_usage where WORKLOAD_GROUP_ID=11201;
++--------------------+
+| mb_per_sec |
++--------------------+
+| 97.94296646118164 |
++--------------------+
+1 row in set (0.03 sec)
+
+mysql [information_schema]>select LOCAL_SCAN_BYTES_PER_SECOND / 1024 / 1024 as
mb_per_sec from workload_group_resource_usage where WORKLOAD_GROUP_ID=11201;
++--------------------+
+| mb_per_sec |
++--------------------+
+| 98.37584781646729 |
++--------------------+
+1 row in set (0.04 sec)
+
+mysql [information_schema]>select LOCAL_SCAN_BYTES_PER_SECOND / 1024 / 1024 as
mb_per_sec from workload_group_resource_usage where WORKLOAD_GROUP_ID=11201;
++--------------------+
+| mb_per_sec |
++--------------------+
+| 98.06641292572021 |
++--------------------+
+1 row in set (0.02 sec)
+```
+
+4. 使用pid工具查看进程最大IO吞吐为每秒131M。
+
+
+
+## 注意事项
+1. 系统表中的LOCAL_SCAN_BYTES_PER_SECOND字段代表的是当前Workload
Group在进程粒度的统计汇总值，比如配置了12个文件路径，那么LOCAL_SCAN_BYTES_PER_SECOND就是这12个文件路径IO的最大值，如果期望查看每个文件路径分别的IO吞吐，可以在grafana上或者BE的bvar监控查看明细的值。
+2. 由于操作系统和Doris的Page Cache的存在，通过linux的IO监控脚本看到的IO通常要比系统表看到的要小。
\ No newline at end of file
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/resource-admin/use-workload-mem-limit.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/resource-admin/use-workload-mem-limit.md
new file mode 100644
index 0000000000..c313223c83
--- /dev/null
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/resource-admin/use-workload-mem-limit.md
@@ -0,0 +1,132 @@
+---
+{
+"title": "Use Workload Memory Limit",
+"language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# 使用Workload Group管理内存资源
+
+Adhoc类查询通常输入的SQL不确定，使用的内存资源也不确定，因此存在少数查询占用很大内存的风险。
+可以对这类负载可以划分到独立的分组，通过Workload Group对内存的硬限的功能，避免突发性的大查询占满所有内存，导致其他查询没有可用内存或者OOM。
+当这个Workload Group的内存使用超过配置的硬限值时，会通过杀死查询的方式释放内存，避免进程内存被打满。
+
+## 测试内存硬限
+### 测试环境
+1FE，1BE，BE配置为96核，内存大小为375G。
+
+测试数据集为clickbench，测试方法为使用jmeter起三并发执行q29。
+
+### 测试不开启Workload Group的内存硬限
+1. 查看进程使用内存。ps命令输出第四列代表进程使用的物理内存的用量，单位为kb，可以看到当前测试负载下，进程的内存使用为7.7G左右。
+```
+[ ~]$ ps -eo pid,comm,%mem,rss | grep 1407481
+1407481 doris_be 2.0 7896792
+[ ~]$ ps -eo pid,comm,%mem,rss | grep 1407481
+1407481 doris_be 2.0 7929692
+[ ~]$ ps -eo pid,comm,%mem,rss | grep 1407481
+1407481 doris_be 2.0 8101232
+```
+
+2. 使用Doris系统表查看当前Workload Group的内存用量，Workload Group的内存用量为5.8G左右。
+```
+mysql [information_schema]>select MEMORY_USAGE_BYTES / 1024/ 1024 as
wg_mem_used_mb from workload_group_resource_usage where workload_group_id=11201;
++-------------------+
+| wg_mem_used_mb |
++-------------------+
+| 5797.524360656738 |
++-------------------+
+1 row in set (0.01 sec)
+
+mysql [information_schema]>select MEMORY_USAGE_BYTES / 1024/ 1024 as
wg_mem_used_mb from workload_group_resource_usage where workload_group_id=11201;
++-------------------+
+| wg_mem_used_mb |
++-------------------+
+| 5840.246627807617 |
++-------------------+
+1 row in set (0.02 sec)
+
+mysql [information_schema]>select MEMORY_USAGE_BYTES / 1024/ 1024 as
wg_mem_used_mb from workload_group_resource_usage where workload_group_id=11201;
++-------------------+
+| wg_mem_used_mb |
++-------------------+
+| 5878.394917488098 |
++-------------------+
+1 row in set (0.02 sec)
+```
+这里可以看到进程的内存使用通常要远大于一个Workload Group的内存用量，即使进程内只有一个Workload
Group在跑，这是因为Workload Group只统计了查询和部分导入的内存，进程内的其他组件比如元数据，各种Cache的内存是不计算Workload
Group内的，也不由Workload Group管理。
+
+### 测试开启Workload Group的内存硬限
+1. 执行SQL命令修改内存配置。
+```
+alter workload group g2 properties('memory_limit'='0.5%');
+alter workload group g2 properties('enable_memory_overcommit'='false');
+```
+
+2. 执行同样的测试，查看系统表的内存用量，内存用量为1.5G左右。
+```
+mysql [information_schema]>select MEMORY_USAGE_BYTES / 1024/ 1024 as
wg_mem_used_mb from workload_group_resource_usage where workload_group_id=11201;
++--------------------+
+| wg_mem_used_mb |
++--------------------+
+| 1575.3877239227295 |
++--------------------+
+1 row in set (0.02 sec)
+
+mysql [information_schema]>select MEMORY_USAGE_BYTES / 1024/ 1024 as
wg_mem_used_mb from workload_group_resource_usage where workload_group_id=11201;
++------------------+
+| wg_mem_used_mb |
++------------------+
+| 1668.77405834198 |
++------------------+
+1 row in set (0.01 sec)
+
+mysql [information_schema]>select MEMORY_USAGE_BYTES / 1024/ 1024 as
wg_mem_used_mb from workload_group_resource_usage where workload_group_id=11201;
++--------------------+
+| wg_mem_used_mb |
++--------------------+
+| 499.96760272979736 |
++--------------------+
+1 row in set (0.01 sec)
+```
+
+3. 使用ps命令查看进程的内存用量，内存用量为3.8G左右。
+```
+[ ~]$ ps -eo pid,comm,%mem,rss | grep 1407481
+1407481 doris_be 1.0 4071364
+[ ~]$ ps -eo pid,comm,%mem,rss | grep 1407481
+1407481 doris_be 1.0 4059012
+[ ~]$ ps -eo pid,comm,%mem,rss | grep 1407481
+1407481 doris_be 1.0 4057068
+```
+
+4. 同时客户端会观察到大量由于内存不足导致的查询失败。
+```
+1724074250162,14126,1c_sql,HY000 1105,"java.sql.SQLException: errCode = 2,
detailMessage = (127.0.0.1)[MEM_LIMIT_EXCEEDED]GC wg for hard limit, wg
id:11201, name:g2, used:1.71 GB, limit:1.69 GB, backend:10.16.10.8. cancel top
memory used tracker <Query#Id=4a0689936c444ac8-a0d01a50b944f6e7> consumption
1.71 GB. details:process memory used 3.01 GB exceed soft limit 304.41 GB or sys
available memory 101.16 GB less than warning water mark 12.80 GB., Execute
again after enough memory, details [...]
+```
+这个报错信息中可以看到，Workload Group使用了1.7G的内存，但是Workload
Group的限制是1.69G，这里的计算方式是这样的1.69G = 物理机内存(375) * mem_limit(be.conf中的值，默认为0.9) *
0.5%（Workload Group的配置）
+也就是说，Workload Group中配置的内存百分比是基于Doris进程可用内存再次进行计算的。
+
+
+## 使用建议
+如上文测试，硬限可以控制Workload
Group的内存使用，但却是通过杀死查询的方式释放内存，这对用户来说体验会非常不友好，极端情况下可能会导致所有查询都失败。
+因此在生产环境中推荐内存硬限配合查询排队的功能一起使用，可以在限制内存使用的同时保证查询的成功率。
\ No newline at end of file
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/resource-admin/use-workload-remote-io-limit.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/resource-admin/use-workload-remote-io-limit.md
new file mode 100644
index 0000000000..e2bb996794
--- /dev/null
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/resource-admin/use-workload-remote-io-limit.md
@@ -0,0 +1,146 @@
+---
+{
+"title": "Use Workload Remote IO Limit",
+"language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# 使用Workload Group管理远程IO
+
+BrokerLoad和S3Load 是常用的大批量数据导入方式，用户可以把数据先上传到HDFS或者S3，然后通过Brokerload和S3Load
对数据进行并行导入。 Doris为了加快导入速度，会使用多线程并行的方式从HDFS/S3拉取数据，此时会对HDFS/S3
产生巨大的压力，会导致HDFS/S3上运行的别的作业不稳定。
+可以通过Workload Group 远程IO的限制功能来限制导入过程中对HDFS/S3的带宽，降低对其他业务的影响。
+
+## 测试远程IO限制
+### 测试环境
+1FE，1BE部署在同一台机器，配置为16核64G内存。测试数据为clickbench数据集，测试前需要把数据集上传到S3上。考虑到上传时间的问题，我们只取其中的1千万行数据上传，然后使用tvf的功能查询s3的数据。
+
+上传成功后可以使用命令查看Schema信息。
+```
+// 查看schema
+DESC FUNCTION s3 (
+ "URI" = "https://bucketname/1kw.tsv";,
+ "s3.access_key"= "ak",
+ "s3.secret_key" = "sk",
+ "format" = "csv",
+ "use_path_style"="true"
+);
+```
+
+### 测试不限制远程读的IO
+1. 发起单并发测试，全表扫描clickbench表。
+```
+// 设置只scan数据，不返回结果
+set dry_run_query = true;
+
+// 使用tvf查询s3的数据
+SELECT * FROM s3(
+ "URI" = "https://bucketname/1kw.tsv";,
+ "s3.access_key"= "ak",
+ "s3.secret_key" = "sk",
+ "format" = "csv",
+ "use_path_style"="true"
+);
+```
+
+2.
使用系统表查看此时的远程IO吞吐。可以看到这个查询的远程IO吞吐为837M每秒，需要注意的是，这里的实际IO吞吐受环境影响较大，如果BE所在的机器连接外部存储的带宽比较低，那么可能实际的吞吐会小。
+```
+MySQL [(none)]> select cast(REMOTE_SCAN_BYTES_PER_SECOND/1024/1024 as int) as
read_mb from information_schema.workload_group_resource_usage;
++---------+
+| read_mb |
++---------+
+| 837 |
++---------+
+1 row in set (0.104 sec)
+
+MySQL [(none)]> select cast(REMOTE_SCAN_BYTES_PER_SECOND/1024/1024 as int) as
read_mb from information_schema.workload_group_resource_usage;
++---------+
+| read_mb |
++---------+
+| 867 |
++---------+
+1 row in set (0.070 sec)
+
+MySQL [(none)]> select cast(REMOTE_SCAN_BYTES_PER_SECOND/1024/1024 as int) as
read_mb from information_schema.workload_group_resource_usage;
++---------+
+| read_mb |
++---------+
+| 867 |
++---------+
+1 row in set (0.186 sec)
+```
+
+3. 使用sar(sar -n DEV 1 3600)命令查看机器的网络带宽，可以看到机器级别最大网络带宽为1033M每秒。
+ 输出的第一列为当前机器某个网卡每秒接收的字节数，单位为KB每秒。
+
+
+
+### 测试限制远程读的IO
+1. 修改Workload Group的配置，限制远程读的IO吞吐为100M每秒。
+```
+alter workload group normal
properties('remote_read_bytes_per_second'='104857600');
+```
+
+2. 发起单并发扫全表的查询。
+```
+// 设置只scan数据，不返回结果
+set dry_run_query = true;
+
+// 使用tvf查询s3的数据
+SELECT * FROM s3(
+ "URI" = "https://bucketname/1kw.tsv";,
+ "s3.access_key"= "ak",
+ "s3.secret_key" = "sk",
+ "format" = "csv",
+ "use_path_style"="true"
+);
+```
+
+3.
使用系统表查看此时的远程读IO吞吐，此时的IO吞吐在100M左右，会有一定的波动，这个波动是受目前算法设计的影响，通常会有一个高峰，但不会持续很长时间，属于正常情况。
+```
+MySQL [(none)]> select cast(REMOTE_SCAN_BYTES_PER_SECOND/1024/1024 as int) as
read_mb from information_schema.workload_group_resource_usage;
++---------+
+| read_mb |
++---------+
+| 56 |
++---------+
+1 row in set (0.010 sec)
+
+MySQL [(none)]> select cast(REMOTE_SCAN_BYTES_PER_SECOND/1024/1024 as int) as
read_mb from information_schema.workload_group_resource_usage;
++---------+
+| read_mb |
++---------+
+| 131 |
++---------+
+1 row in set (0.009 sec)
+
+MySQL [(none)]> select cast(REMOTE_SCAN_BYTES_PER_SECOND/1024/1024 as int) as
read_mb from information_schema.workload_group_resource_usage;
++---------+
+| read_mb |
++---------+
+| 111 |
++---------+
+1 row in set (0.009 sec)
+```
+
+4. 使用sar命令（sar -n DEV 1
3600）查看目前的网卡接收流量，第一列为每秒接收的数据量，可以看到最大值变成了207M每秒，说明读IO的限制是生效的，但是由于sar命令看到的是机器级别的流量，因此要比Doris统计到的会大一些。
+
+
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/resource-admin/workload-fusing.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/resource-admin/workload-fusing.md
new file mode 100644
index 0000000000..d8cd0b8a7a
--- /dev/null
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/resource-admin/workload-fusing.md
@@ -0,0 +1,99 @@
+---
+{
+"title": "Workload Fusing",
+"language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# 大查询熔断
+
+大查询熔断通常适用于线上集群经常出现个别占用资源过多的查询导致集群整体可用性下降的场景，此时可以通过在运行时检测异常的大查询，并且阻止这个查询运行的方式使得集群恢复正常。
+如果用户已经根据查询延迟对负载进行了分组，比如大查询划分到一个Workload Group，小查询划分到一个Workload
Group，那么也可以通过大查询熔断的方式提高小查询分组的可用性。
+目前的大查询熔断策略主要是通过Workload Policy功能实现的，用户可以通过配置特定的策略实现大查询的自动熔断。
+
+## 大查询熔断测试
+
+### 测试环境
+1FE，1BE，BE配置为96核，测试数据集为clickbench。
+
+### 基于内存容量的熔断
+1. 测试SQL使用ckbench的q29，可以通过审计看到这个查询的峰值内存用量（peakMemoryBytes）为8G。
+```
+|User=root|Ctl=internal|Db=hits|State=EOF|ErrorCode=0|ErrorMessage=|Time(ms)=4614|ScanBytes=13107527680|ScanRows=81032736|ReturnRows=11|StmtId=526|QueryId=e5b6c62d624146e4-b7291221492a7cc2|IsQuery=true|isNereids=true|feIp=10.16.10.8|StmtType=SELECT|Stmt=SELECT
REGEXP_REPLACE(Referer, '^https?://(?:www\.)?([^/]+)/.*$', '\1') AS k,
AVG(length(Referer)) AS l, COUNT(*) AS c, MIN(Referer) FROM hits WHERE Referer
<> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT
25|CpuTimeMS=1056 [...]
+```
+
+2. 新建一个Workload Policy，如果查询内存超过1G就kill。
+```
+create workload policy cancel_1g_query conditions(query_be_memory_bytes >
1073741824) actions(cancel_query);
+
+// 通过系统表查看新建的policy
+mysql [information_schema]>select * from workload_policy;
++-------+-----------------+------------------------------------+--------------+----------+---------+---------+----------------+
+| ID | NAME | CONDITION | ACTION
| PRIORITY | ENABLED | VERSION | WORKLOAD_GROUP |
++-------+-----------------+------------------------------------+--------------+----------+---------+---------+----------------+
+| 11313 | cancel_1g_query | query_be_memory_bytes > 1073741824 | cancel_query
| 0 | 1 | 0 | |
++-------+-----------------+------------------------------------+--------------+----------+---------+---------+----------------+
+1 row in set (0.02 sec)
+```
+
+3. 再次执行查询，可以看到此时查询会被新建的Policy给cancel掉。
+```
+ mysql [information_schema]>SELECT REGEXP_REPLACE(Referer,
'^https?://(?:www\.)?([^/]+)/.*$', '\1') AS k, AVG(length(Referer)) AS l,
COUNT(*) AS c, MIN(Referer) FROM hits.hits WHERE Referer <> '' GROUP BY k
HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;
+ERROR 1105 (HY000): errCode = 2, detailMessage =
(127.0.0.1)[INTERNAL_ERROR]query ceb0553d359e454f-8939599b2e602d07 cancelled by
workload policy: cancel_1g_query, id:11313
+```
+
+### 基于SCAN数据量的熔断
+1. 使用ckbench的q29进行测试，新建一个限制扫描行数的Policy。
+```
+// 先禁用掉已经创建的policy
+alter workload policy cancel_1g_query properties('enabled'='false');
+
+// 创建一个新的基于scan函数熔断的policy
+create workload policy canel_scan_5kw_query conditions(be_scan_rows >
50000000) actions(cancel_query);
+```
+
+2. 测试查询执行情况，此时查询会由于SCAN行数超过配置的值而失败。
+```
+mysql [hits]>SELECT REGEXP_REPLACE(Referer, '^https?://(?:www\.)?([^/]+)/.*$',
'\1') AS k, AVG(length(Referer)) AS l, COUNT(*) AS c, MIN(Referer) FROM
hits.hits WHERE Referer <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l
DESC LIMIT 25;
+
+ERROR 1105 (HY000): errCode = 2, detailMessage =
(127.0.0.1)[INTERNAL_ERROR]query f9d15001bfc94322-9b63a8b84aef9dee cancelled by
workload policy: canel_scan_5kw_query, id:11314
+```
+需要注意的是，be_scan_rows主要统计的是某个查询在单BE上的扫描数据量，线上环境下一个表的数据通常分布在多个BE上，而审计中看到的查询的扫描行数是多个BE的汇总，
+因此当查询由于SCAN数据量失败时，审计里的扫描行数通常要比Policy中配置的大很多。
+
+### 基于查询时间的熔断
+1. 使用ckbench的q29进行测试，新建一个取消执行时间超过1s的查询的Policy。
+```
+// 先禁用已经创建的policy
+alter workload policy canel_scan_5kw_query properties('enabled'='false');
+
+create workload policy cancel_time_1s_query conditions(query_time > 1000)
actions(cancel_query);
+```
+2. 执行查询，可以看到该查询由于执行时间超过1秒而失败。
+```
+mysql [hits]>SELECT REGEXP_REPLACE(Referer, '^https?://(?:www\.)?([^/]+)/.*$',
'\1') AS k, AVG(length(Referer)) AS l, COUNT(*) AS c, MIN(Referer) FROM
hits.hits WHERE Referer <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l
DESC LIMIT 25;
+ERROR 1105 (HY000): errCode = 2, detailMessage =
(127.0.0.1)[INTERNAL_ERROR]query c6aebdf403e24c62-965daf0bdff179f7 cancelled by
workload policy: cancel_time_1s_query, id:11315
+```
+
+### 注意事项
+1. 目前FE向BE同步Workload Policy元数据的时间间隔为30秒，也就是对于policy的变更最大需要30秒才会生效。
+2. BE目前是通过定时线程来执行Policy中配置的策略，时间间隔为500ms，这意味着如果查询的时间过短，在500ms以内，可能不会受到Workload
Policy的约束。
\ No newline at end of file
diff --git a/sidebars.json b/sidebars.json
index e5199411b5..1cc877ced4 100644
--- a/sidebars.json
+++ b/sidebars.json
@@ -395,7 +395,12 @@
"admin-manual/resource-admin/workload-group",
"admin-manual/resource-admin/workload-policy",
"admin-manual/resource-admin/workload-system-table",
- "admin-manual/resource-admin/multi-tenant"
+ "admin-manual/resource-admin/multi-tenant",
+ "admin-manual/resource-admin/workload-fusing",
+ "admin-manual/resource-admin/use-workload-mem-limit",
+ "admin-manual/resource-admin/use-workload-cpu-limit",
+
"admin-manual/resource-admin/use-workload-local-io-limit",
+
"admin-manual/resource-admin/use-workload-remote-io-limit"
]
},
{
diff --git a/static/images/workload-management/use_wg_cpu_1.png
b/static/images/workload-management/use_wg_cpu_1.png
new file mode 100644
index 0000000000..6d6cd8c817
Binary files /dev/null and b/static/images/workload-management/use_wg_cpu_1.png
differ
diff --git a/static/images/workload-management/use_wg_cpu_2.png
b/static/images/workload-management/use_wg_cpu_2.png
new file mode 100644
index 0000000000..ab689a0330
Binary files /dev/null and b/static/images/workload-management/use_wg_cpu_2.png
differ
diff --git a/static/images/workload-management/use_wg_io_1.png
b/static/images/workload-management/use_wg_io_1.png
new file mode 100644
index 0000000000..34979196a5
Binary files /dev/null and b/static/images/workload-management/use_wg_io_1.png
differ
diff --git a/static/images/workload-management/use_wg_io_2.png
b/static/images/workload-management/use_wg_io_2.png
new file mode 100644
index 0000000000..ec8328de44
Binary files /dev/null and b/static/images/workload-management/use_wg_io_2.png
differ
diff --git a/static/images/workload-management/use_wg_rio_1.png
b/static/images/workload-management/use_wg_rio_1.png
new file mode 100644
index 0000000000..070001fe0e
Binary files /dev/null and b/static/images/workload-management/use_wg_rio_1.png
differ
diff --git a/static/images/workload-management/use_wg_rio_2.png
b/static/images/workload-management/use_wg_rio_2.png
new file mode 100644
index 0000000000..018cebb935
Binary files /dev/null and b/static/images/workload-management/use_wg_rio_2.png
differ
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
