This is an automated email from the ASF dual-hosted git repository.
gabriellee pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new ea35ed32224 Refactor pipeline document (#1138)
ea35ed32224 is described below
commit ea35ed32224dc15902ffb262ccc0e72806aeb75a
Author: Gabriel <gabrielleeb...@gmail.com>
AuthorDate: Tue Sep 24 10:07:53 2024 +0800
Refactor pipeline document (#1138)
# Versions
- [x] dev
- [x] 3.0
- [ ] 2.1
- [ ] 2.0
# Languages
- [x] Chinese
- [x] English
---
docs/query/pipeline/pipeline-execution-engine.md | 38 ++++++++---
docs/query/pipeline/pipeline-x-execution-engine.md | 78 ----------------------
.../query/pipeline/pipeline-execution-engine.md | 40 ++++++++---
.../query/pipeline/pipeline-x-execution-engine.md | 76 ---------------------
.../query/pipeline/pipeline-execution-engine.md | 28 ++++++--
.../query/pipeline/pipeline-x-execution-engine.md | 76 ---------------------
sidebars.json | 9 +--
.../query/pipeline/pipeline-execution-engine.md | 26 +++++++-
.../query/pipeline/pipeline-x-execution-engine.md | 78 ----------------------
versioned_sidebars/version-3.0-sidebars.json | 9 +--
10 files changed, 110 insertions(+), 348 deletions(-)
diff --git a/docs/query/pipeline/pipeline-execution-engine.md
b/docs/query/pipeline/pipeline-execution-engine.md
index 1d58fa8d4fc..9e5145908ac 100644
--- a/docs/query/pipeline/pipeline-execution-engine.md
+++ b/docs/query/pipeline/pipeline-execution-engine.md
@@ -1,9 +1,9 @@
---
{
- "title": "Pipeline Execution Engine",
- "language": "en",
- "toc_min_heading_level": 2,
- "toc_max_heading_level": 4
+"title": "Pipeline Execution Engine",
+"language": "en",
+"toc_min_heading_level": 2,
+"toc_max_heading_level": 4
}
---
@@ -27,10 +27,11 @@ under the License.
-->
+The Pipeline Execution Engine is an experimental feature in Apache Doris 2.0,
which was later optimized and upgraded in version 2.1 (i.e., PipelineX). In
versions 3.0 and later, PipelineX is used as the only execution engine in Doris
and renamed to Pipeline Execution Engine.
The goal of pipeline execution engine is to replace the current execution
engine of Doris's volcano model, fully release the computing power of
multi-core CPUs, and limit the number of Doris's query threads to solve the
problem of Doris's execution thread bloat.
-Its specific design, implementation and effects can be found in
[DSIP-027]([DSIP-027: Support Pipeline Exec Engine - DORIS - Apache Software
Foundation](https://cwiki.apache.org/confluence/display/DORIS/DSIP-027%3A+Support+Pipeline+Exec+Engine)).
+Its specific design, implementation and effects can be found in
[DSIP-027]([DSIP-027: Support Pipeline Exec Engine - DORIS - Apache Software
Foundation](https://cwiki.apache.org/confluence/display/DORIS/DSIP-027%3A+Support+Pipeline+Exec+Engine))
and [DSIP-035]([DSIP-035: PipelineX Execution Engine - DORIS - Apache Software
Foundation](https://cwiki.apache.org/confluence/display/DORIS/DSIP-035%3A+PipelineX+Execution+Engine)).
## Principle
@@ -38,8 +39,8 @@ The current Doris SQL execution engine is designed based on
the traditional volc
* Inability to take full advantage of multi-core computing power to improve
query performance,**most scenarios require manual setting of parallelism** for
performance tuning, which is almost difficult to set in production environments.
* Each instance of a standalone query corresponds to one thread of the thread
pool, which introduces two additional problems.
- * Once the thread pool is hit full. **Doris' query engine will enter a
pseudo-deadlock** and will not respond to subsequent queries. **At the same
time there is a certain probability of entering a logical deadlock** situation:
for example, all threads are executing an instance's probe task.
- * Blocking arithmetic will take up thread resources,**blocking thread
resources can not be yielded to instances that can be scheduled**, the overall
resource utilization does not go up.
+ * Once the thread pool is hit full. **Doris' query engine will enter a
pseudo-deadlock** and will not respond to subsequent queries. **At the same
time there is a certain probability of entering a logical deadlock** situation:
for example, all threads are executing an instance's probe task.
+ * Blocking arithmetic will take up thread resources,**blocking thread
resources can not be yielded to instances that can be scheduled**, the overall
resource utilization does not go up.
* Blocking arithmetic relies on the OS thread scheduling mechanism, **thread
switching overhead (especially in the scenario of system mixing))**
@@ -52,6 +53,11 @@ And as shown in the figure below (quoted from[Push versus
pull-based loop fusion
1. Transformation of the traditional pull pull logic-driven execution process
into a data-driven execution engine for the push model
2. Blocking operations are asynchronous, reducing the execution overhead
caused by thread switching and thread blocking and making more efficient use of
the CPU
3. Controls the number of threads to be executed and reduces the resource
congestion of large queries on small queries in mixed load scenarios by
controlling time slice switching
+4. In terms of execution concurrency, pipelineX introduces local exchange
optimization to fully utilize CPU resources, and distribute data evenly across
different tasks to minimize data skewing. In addition, pipelineX will no longer
be constrained by the number of tablets.
+5. Logically, multiple pipeline tasks share all shared states of the same
pipeline and eliminate additional initialization overhead, such as expressions
and some const variables.
+6. In terms of scheduling logic, the blocking conditions of all pipeline tasks
are encapsulated using Dependency, and the execution logic of the tasks is
triggered by external events (such as rpc completion) to enter the runnable
queue, thereby eliminating the overhead of blocking polling threads.
+7. Profile: Provide users with simple and easy to understand metrics.
+
This improves the efficiency of CPU execution on mixed-load SQL and enhances
the performance of SQL queries.
@@ -77,6 +83,22 @@ set parallel_pipeline_task_num = 0;
You can limit the automatically configured concurrency by setting
`max_instance_num`（The default value is 64)
+3. enable_local_shuffle
+
+Set `enable_local_shuffle` to true will enable local shuffle optimization.
Local shuffle will try to evenly distribute data among different pipeline tasks
to avoid data skewing as much as possible.
+
+```sql
+set enable_local_shuffle = true;
+```
+
+4. ignore_storage_data_distribution
+
+Settings `ignore_storage_data_distribution` is true, it means ignoring the
data distribution of the storage layer. When used in conjunction with local
shuffle, the concurrency capability of the pipelineX engine will no longer be
constrained by the number of storage layer tables, thus fully utilizing machine
resources.
+
+```sql
+set ignore_storage_data_distribution = true;
+```
+
### Load
-The engine selected for import are detailed in the
[Import](../../data-operate/import/load-manual) documentation.
+The engine selected for import are detailed in the
[Load](../../data-operate/import/load-manual) documentation.
diff --git a/docs/query/pipeline/pipeline-x-execution-engine.md
b/docs/query/pipeline/pipeline-x-execution-engine.md
deleted file mode 100644
index f926ed37141..00000000000
--- a/docs/query/pipeline/pipeline-x-execution-engine.md
+++ /dev/null
@@ -1,78 +0,0 @@
----
-{
- "title": "PipelineX Execution Engine",
- "language": "en",
- "toc_min_heading_level": 2,
- "toc_max_heading_level": 4
-}
----
-
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-
-
-## Background
-
-The PipelineX execution engine is an experimental feature in Doris 2.1.0,
expected to address the four major issues of the Doris pipeline engine:
-1. In terms of execution concurrency, Doris is currently constrained by two
factors: one is the parameters set by FE, and the other is limited by the
number of buckets. This concurrent strategy prevents the execution engine from
fully utilizing machine resources.
-2. In terms of execution logic, Doris currently has some fixed additional
overhead. For example, the common expression for all instances will be
initialized multiple times due to independence between all instances.
-3. In terms of scheduling logic, the scheduler of the current pipeline will
put all blocking tasks into a blocking queue, and a blocking queue scheduler
will be responsible for polling and extracting executable tasks from the
blocking queue and placing them in the runnable queue. Therefore, during the
query execution process, a CPU core will always be occupied to do scheduling
instead of execution.
-4. In terms of profile, currently the pipeline cannot provide users concise
and clear metrics.
-
-Its specific design and implementation can be found in [DSIP-035]([DSIP-035:
PipelineX Execution Engine - DORIS - Apache Software
Foundation](https://cwiki.apache.org/confluence/display/DORIS/DSIP-035%3A+PipelineX+Execution+Engine)).
-
-## Goals
-
-1. In terms of execution concurrency, pipelineX introduces local exchange
optimization to fully utilize CPU resources, and distribute data evenly across
different tasks to minimize data skewing. In addition, pipelineX will no longer
be constrained by the number of tablets.
-2. Logically, multiple pipeline tasks share all shared states of the same
pipeline and eliminate additional initialization overhead, such as expressions
and some const variables.
-3. In terms of scheduling logic, the blocking conditions of all pipeline tasks
are encapsulated using Dependency, and the execution logic of the tasks is
triggered by external events (such as rpc completion) to enter the runnable
queue, thereby eliminating the overhead of blocking polling threads.
-4. Profile: Provide users with simple and easy to understand metrics.
-
-## User Interface changes
-
-### Set session variable
-
-#### enable_pipeline_x_engine
-
-Set `enable_pipeline_x_engine ` to `true`, BE will use PipelineX to execute by
default.
-
-```
-set enable_pipeline_x_engine = true;
-```
-
-#### enable_local_shuffle
-
-Set `enable_local_shuffle` to true will enable local shuffle optimization.
Local shuffle will try to evenly distribute data among different pipeline tasks
to avoid data skewing as much as possible.
-
-```
-set enable_local_shuffle = true;
-```
-
-#### ignore_storage_data_distribution
-
-Settings `ignore_storage_data_distribution` is true, it means ignoring the
data distribution of the storage layer. When used in conjunction with local
shuffle, the concurrency capability of the pipelineX engine will no longer be
constrained by the number of storage layer tables, thus fully utilizing machine
resources.
-
-```
-set ignore_storage_data_distribution = true;
-```
-
-### Load
-
-The engine selected for import are detailed in the
[Import](../../data-operate/import/load-manual) documentation.
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/query/pipeline/pipeline-execution-engine.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/query/pipeline/pipeline-execution-engine.md
index 47596be46b7..f556bab6a0c 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/query/pipeline/pipeline-execution-engine.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/query/pipeline/pipeline-execution-engine.md
@@ -1,9 +1,9 @@
---
{
- "title": "Pipeline 执行引擎",
- "language": "zh-CN",
- "toc_min_heading_level": 2,
- "toc_max_heading_level": 4
+"title": "Pipeline 执行引擎",
+"language": "zh-CN",
+"toc_min_heading_level": 2,
+"toc_max_heading_level": 4
}
---
@@ -30,12 +30,12 @@ under the License.
:::info 备注
-Pipeline 执行引擎 是 Doris 在 2.0 版本加入的实验性功能。
+Pipeline 执行引擎 是 Doris 在 2.0 版本加入的实验性功能，随后在 2.1 版本进行了优化与升级 （即 PipelineX）。在 3.0
以及之后的版本中，Doris只使用PipelineX作为唯一执行引擎，并且更名为Pipeline执行引擎。
:::
-Pipeline 执行引擎的目标是为了替换当前 Doris 的火山模型的执行引擎，充分释放多核 CPU 的计算能力，并对 Doris
的查询线程的数目进行限制，解决 Doris 的执行线程膨胀的问题。
+Pipeline 执行引擎的主要目标是为了替换之前 Doris 基于火山模型的执行引擎，充分释放多核 CPU 的计算能力，并对 Doris
的查询线程的数目进行限制，解决 Doris 的执行线程膨胀的问题。
-它的具体设计、实现和效果可以参阅 [DSIP-027]([DSIP-027: Support Pipeline Exec Engine - DORIS -
Apache Software
Foundation](https://cwiki.apache.org/confluence/display/DORIS/DSIP-027%3A+Support+Pipeline+Exec+Engine))。
+它的具体设计、实现和效果可以参阅 [DSIP-027]([DSIP-027: Support Pipeline Exec Engine - DORIS -
Apache Software
Foundation](https://cwiki.apache.org/confluence/display/DORIS/DSIP-027%3A+Support+Pipeline+Exec+Engine))
以及 [DSIP-035]([DSIP-035: PipelineX Execution Engine - DORIS - Apache Software
Foundation](https://cwiki.apache.org/confluence/display/DORIS/DSIP-035%3A+PipelineX+Execution+Engine))。
## 原理
@@ -43,8 +43,8 @@ Pipeline 执行引擎的目标是为了替换当前 Doris 的火山模型的执
* 无法充分利用多核计算能力，提升查询性能，**多数场景下进行性能调优时需要手动设置并行度**，在生产环境中几乎很难进行设定。
* 单机查询的每个 Instance 对应线程池的一个线程，这会带来额外的两个问题。
- * 线程池一旦打满。**Doris
的查询引擎会进入假性死锁**，对后续的查询无法响应。**同时有一定概率进入逻辑死锁**的情况：比如所有的线程都在执行一个 Instance 的 Probe
任务。
- * 阻塞的算子会占用线程资源，**而阻塞的线程资源无法让渡给能够调度的 Instance**，整体资源利用率上不去。
+ * 线程池一旦打满。**Doris
的查询引擎会进入假性死锁**，对后续的查询无法响应。**同时有一定概率进入逻辑死锁**的情况：比如所有的线程都在执行一个 Instance 的 Probe
任务。
+ * 阻塞的算子会占用线程资源，**而阻塞的线程资源无法让渡给能够调度的 Instance**，整体资源利用率上不去。
* 阻塞算子依赖操作系统的线程调度机制，**线程切换开销较大（尤其在系统混布的场景中）**
@@ -57,6 +57,10 @@ Pipeline 执行引擎的目标是为了替换当前 Doris 的火山模型的执
1. 将传统 Pull 拉取的逻辑驱动的执行流程改造为 Push 模型的数据驱动的执行引擎
2. 阻塞操作异步化，减少了线程切换，线程阻塞导致的执行开销，对于 CPU 的利用更为高效
3. 控制了执行线程的数目，通过时间片的切换的控制，在混合负载的场景中，减少大查询对于小查询的资源挤占问题
+4. 执行并发上，依赖 Local Exchange 使 Pipeline 充分并发，可以让数据被均匀分布到不同的 Task
中，尽可能减少数据倾斜，此外，Pipeline 也将不再受存储层 Tablet 数量的制约。
+5. 执行逻辑上，多个 Pipeline Task 共享同一个 Pipeline 的全部共享状态，例如表达式和一些 Const 变量，消除了额外的初始化开销。
+6. 调度逻辑上，所有 Pipeline Task 的阻塞条件都使用 Dependency 进行了封装，通过外部事件（例如 RPC 完成）触发 task
的执行逻辑进入 Runnable 队列，从而消除了阻塞轮询线程的开销。
+7. Profile：为用户提供简单易懂的指标。
从而提高了 CPU 在混合负载 SQL 上执行时的效率，提升了 SQL 查询的性能。
@@ -66,7 +70,7 @@ Pipeline 执行引擎的目标是为了替换当前 Doris 的火山模型的执
1. enable_pipeline_engine
-将 session 变量 `enable_pipeline_engine` 设置为 `true`，则 BE 在进行查询执行时将会使用 Pipeline
执行引擎。
+将 Session 变量 `enable_pipeline_engine` 设置为 `true`，则 BE 在进行查询执行时将会使用 Pipeline
执行引擎。
```sql
set enable_pipeline_engine = true;
@@ -82,6 +86,22 @@ set parallel_pipeline_task_num = 0;
可以通过设置 `max_instance_num` 来限制自动设置的并发数 (默认为 64)
+3. enable_local_shuffle
+
+设置`enable_local_shuffle`为 True 则打开 Local Shuffle 优化。Local Shuffle
将尽可能将数据均匀分布给不同的 Pipeline Task 从而尽可能避免数据倾斜。
+
+```sql
+set enable_local_shuffle = true;
+```
+
+4. ignore_storage_data_distribution
+
+设置`ignore_storage_data_distribution`为 True 则表示忽略存储层的数据分布。结合 Local Shuffle
一起使用，则 Pipeline 引擎的并发能力将不再受到存储层 Tablet 数量的制约，从而充分利用机器资源。
+
+```sql
+set ignore_storage_data_distribution = true;
+```
+
### 导入
导入的引擎选择设置，详见[导入](../../data-operate/import/load-manual)文档。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/query/pipeline/pipeline-x-execution-engine.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/query/pipeline/pipeline-x-execution-engine.md
deleted file mode 100644
index 59e77313bb7..00000000000
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/query/pipeline/pipeline-x-execution-engine.md
+++ /dev/null
@@ -1,76 +0,0 @@
----
-{
- "title": "PipelineX 执行引擎",
- "language": "zh-CN",
- "toc_min_heading_level": 2,
- "toc_max_heading_level": 4
-}
----
-
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-
-## 背景
-PipelineX 执行引擎的目标是为了解决 Doris pipeline 引擎的四大问题：
-1. 执行并发上，当前 Doris 执行并发收到两个因素的制约，一个是 fe 设置的参数，另一个是受存储层 bucket
数量的限制，这样的静态并发使得执行引擎无法充分利用机器资源。
-2. 执行逻辑上，当前 Doris 有一些固定的额外开销，例如表达式部分各个 instance 彼此独立，而 instance
的初始化参数有很多公共部分，所以需要额外进行很多重复的初始化步骤。
-3. 调度逻辑上，当前 pipeline 的调度器会把阻塞 task 全部放入一个阻塞队列中，由一个线程负责轮询并从阻塞队列中取出可执行 task 放入
runnable 队列，所以在有查询执行的过程中，会固定有一个核的资源作为调度的开销。
-4. profile 方面，目前 pipeline 无法为用户提供简单易懂的指标。
-
-它的具体设计、实现和效果可以参阅 [DSIP-035]([DSIP-035: PipelineX Execution Engine - DORIS -
Apache Software
Foundation](https://cwiki.apache.org/confluence/display/DORIS/DSIP-035%3A+PipelineX+Execution+Engine))。
-
-## 预期效果
-
-1. 执行并发上，依赖 local exchange 使 pipelinex 充分并发，可以让数据被均匀分布到不同的 task
中，尽可能减少数据倾斜，此外，pipelineX 也将不再受存储层 tablet 数量的制约。
-2. 执行逻辑上，多个 pipeline task 共享同一个 pipeline 的全部共享状态，例如表达式和一些 const 变量，消除了额外的初始化开销。
-3. 调度逻辑上，所有 pipeline task 的阻塞条件都使用 Dependency 进行了封装，通过外部事件（例如 rpc 完成）触发 task
的执行逻辑进入 runnable 队列，从而消除了阻塞轮询线程的开销。
-4. profile：为用户提供简单易懂的指标。
-
-## 用户接口变更
-
-### 设置 Session 变量
-
-#### enable_pipeline_x_engine
-
-将 session 变量`enable_pipeline_x_engine` 设置为`true`，则 BE 在进行查询执行时就会默认将 SQL
的执行模型转变 PipelineX 的执行方式。如果 Pipeline 和 PipelineX 引擎都设置为开启，Doris 将优先使用 PipelineX
引擎。
-
-```
-set enable_pipeline_x_engine = true;
-```
-
-#### enable_local_shuffle
-
-设置`enable_local_shuffle`为 true 则打开 local shuffle 优化。local shuffle
将尽可能将数据均匀分布给不同的 pipeline task 从而尽可能避免数据倾斜。
-
-```
-set enable_local_shuffle = true;
-```
-
-#### ignore_storage_data_distribution
-
-设置`ignore_storage_data_distribution`为 true 则表示忽略存储层的数据分布。结合 local shuffle
一起使用，则 pipelineX 引擎的并发能力将不再受到存储层 tablet 数量的制约，从而充分利用机器资源。
-
-```
-set ignore_storage_data_distribution = true;
-```
-
-### 导入
-
-导入的引擎选择设置，详见[导入](../../data-operate/import/load-manual)文档。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/query/pipeline/pipeline-execution-engine.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/query/pipeline/pipeline-execution-engine.md
index 47596be46b7..708370a255f 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/query/pipeline/pipeline-execution-engine.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/query/pipeline/pipeline-execution-engine.md
@@ -30,12 +30,12 @@ under the License.
:::info 备注
-Pipeline 执行引擎 是 Doris 在 2.0 版本加入的实验性功能。
+Pipeline 执行引擎 是 Doris 在 2.0 版本加入的实验性功能，随后在 2.1 版本进行了优化与升级 （即 PipelineX）。在 3.0
以及之后的版本中，Doris只使用PipelineX作为唯一执行引擎，并且更名为Pipeline执行引擎。
:::
-Pipeline 执行引擎的目标是为了替换当前 Doris 的火山模型的执行引擎，充分释放多核 CPU 的计算能力，并对 Doris
的查询线程的数目进行限制，解决 Doris 的执行线程膨胀的问题。
+Pipeline 执行引擎的主要目标是为了替换之前 Doris 基于火山模型的执行引擎，充分释放多核 CPU 的计算能力，并对 Doris
的查询线程的数目进行限制，解决 Doris 的执行线程膨胀的问题。
-它的具体设计、实现和效果可以参阅 [DSIP-027]([DSIP-027: Support Pipeline Exec Engine - DORIS -
Apache Software
Foundation](https://cwiki.apache.org/confluence/display/DORIS/DSIP-027%3A+Support+Pipeline+Exec+Engine))。
+它的具体设计、实现和效果可以参阅 [DSIP-027]([DSIP-027: Support Pipeline Exec Engine - DORIS -
Apache Software
Foundation](https://cwiki.apache.org/confluence/display/DORIS/DSIP-027%3A+Support+Pipeline+Exec+Engine))
以及 [DSIP-035]([DSIP-035: PipelineX Execution Engine - DORIS - Apache Software
Foundation](https://cwiki.apache.org/confluence/display/DORIS/DSIP-035%3A+PipelineX+Execution+Engine))。
## 原理
@@ -57,6 +57,10 @@ Pipeline 执行引擎的目标是为了替换当前 Doris 的火山模型的执
1. 将传统 Pull 拉取的逻辑驱动的执行流程改造为 Push 模型的数据驱动的执行引擎
2. 阻塞操作异步化，减少了线程切换，线程阻塞导致的执行开销，对于 CPU 的利用更为高效
3. 控制了执行线程的数目，通过时间片的切换的控制，在混合负载的场景中，减少大查询对于小查询的资源挤占问题
+4. 执行并发上，依赖 Local Exchange 使 Pipeline 充分并发，可以让数据被均匀分布到不同的 Task
中，尽可能减少数据倾斜，此外，Pipeline 也将不再受存储层 Tablet 数量的制约。
+5. 执行逻辑上，多个 Pipeline Task 共享同一个 Pipeline 的全部共享状态，例如表达式和一些 Const 变量，消除了额外的初始化开销。
+6. 调度逻辑上，所有 Pipeline Task 的阻塞条件都使用 Dependency 进行了封装，通过外部事件（例如 RPC 完成）触发 task
的执行逻辑进入 Runnable 队列，从而消除了阻塞轮询线程的开销。
+7. Profile：为用户提供简单易懂的指标。
从而提高了 CPU 在混合负载 SQL 上执行时的效率，提升了 SQL 查询的性能。
@@ -66,7 +70,7 @@ Pipeline 执行引擎的目标是为了替换当前 Doris 的火山模型的执
1. enable_pipeline_engine
-将 session 变量 `enable_pipeline_engine` 设置为 `true`，则 BE 在进行查询执行时将会使用 Pipeline
执行引擎。
+将 Session 变量 `enable_pipeline_engine` 设置为 `true`，则 BE 在进行查询执行时将会使用 Pipeline
执行引擎。
```sql
set enable_pipeline_engine = true;
@@ -82,6 +86,22 @@ set parallel_pipeline_task_num = 0;
可以通过设置 `max_instance_num` 来限制自动设置的并发数 (默认为 64)
+3. enable_local_shuffle
+
+设置`enable_local_shuffle`为 True 则打开 Local Shuffle 优化。Local Shuffle
将尽可能将数据均匀分布给不同的 Pipeline Task 从而尽可能避免数据倾斜。
+
+```sql
+set enable_local_shuffle = true;
+```
+
+4. ignore_storage_data_distribution
+
+设置`ignore_storage_data_distribution`为 True 则表示忽略存储层的数据分布。结合 Local Shuffle
一起使用，则 Pipeline 引擎的并发能力将不再受到存储层 Tablet 数量的制约，从而充分利用机器资源。
+
+```sql
+set ignore_storage_data_distribution = true;
+```
+
### 导入
导入的引擎选择设置，详见[导入](../../data-operate/import/load-manual)文档。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/query/pipeline/pipeline-x-execution-engine.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/query/pipeline/pipeline-x-execution-engine.md
deleted file mode 100644
index 59e77313bb7..00000000000
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/query/pipeline/pipeline-x-execution-engine.md
+++ /dev/null
@@ -1,76 +0,0 @@
----
-{
- "title": "PipelineX 执行引擎",
- "language": "zh-CN",
- "toc_min_heading_level": 2,
- "toc_max_heading_level": 4
-}
----
-
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-
-## 背景
-PipelineX 执行引擎的目标是为了解决 Doris pipeline 引擎的四大问题：
-1. 执行并发上，当前 Doris 执行并发收到两个因素的制约，一个是 fe 设置的参数，另一个是受存储层 bucket
数量的限制，这样的静态并发使得执行引擎无法充分利用机器资源。
-2. 执行逻辑上，当前 Doris 有一些固定的额外开销，例如表达式部分各个 instance 彼此独立，而 instance
的初始化参数有很多公共部分，所以需要额外进行很多重复的初始化步骤。
-3. 调度逻辑上，当前 pipeline 的调度器会把阻塞 task 全部放入一个阻塞队列中，由一个线程负责轮询并从阻塞队列中取出可执行 task 放入
runnable 队列，所以在有查询执行的过程中，会固定有一个核的资源作为调度的开销。
-4. profile 方面，目前 pipeline 无法为用户提供简单易懂的指标。
-
-它的具体设计、实现和效果可以参阅 [DSIP-035]([DSIP-035: PipelineX Execution Engine - DORIS -
Apache Software
Foundation](https://cwiki.apache.org/confluence/display/DORIS/DSIP-035%3A+PipelineX+Execution+Engine))。
-
-## 预期效果
-
-1. 执行并发上，依赖 local exchange 使 pipelinex 充分并发，可以让数据被均匀分布到不同的 task
中，尽可能减少数据倾斜，此外，pipelineX 也将不再受存储层 tablet 数量的制约。
-2. 执行逻辑上，多个 pipeline task 共享同一个 pipeline 的全部共享状态，例如表达式和一些 const 变量，消除了额外的初始化开销。
-3. 调度逻辑上，所有 pipeline task 的阻塞条件都使用 Dependency 进行了封装，通过外部事件（例如 rpc 完成）触发 task
的执行逻辑进入 runnable 队列，从而消除了阻塞轮询线程的开销。
-4. profile：为用户提供简单易懂的指标。
-
-## 用户接口变更
-
-### 设置 Session 变量
-
-#### enable_pipeline_x_engine
-
-将 session 变量`enable_pipeline_x_engine` 设置为`true`，则 BE 在进行查询执行时就会默认将 SQL
的执行模型转变 PipelineX 的执行方式。如果 Pipeline 和 PipelineX 引擎都设置为开启，Doris 将优先使用 PipelineX
引擎。
-
-```
-set enable_pipeline_x_engine = true;
-```
-
-#### enable_local_shuffle
-
-设置`enable_local_shuffle`为 true 则打开 local shuffle 优化。local shuffle
将尽可能将数据均匀分布给不同的 pipeline task 从而尽可能避免数据倾斜。
-
-```
-set enable_local_shuffle = true;
-```
-
-#### ignore_storage_data_distribution
-
-设置`ignore_storage_data_distribution`为 true 则表示忽略存储层的数据分布。结合 local shuffle
一起使用，则 pipelineX 引擎的并发能力将不再受到存储层 tablet 数量的制约，从而充分利用机器资源。
-
-```
-set ignore_storage_data_distribution = true;
-```
-
-### 导入
-
-导入的引擎选择设置，详见[导入](../../data-operate/import/load-manual)文档。
diff --git a/sidebars.json b/sidebars.json
index 59af3872dd8..5daaa8965e4 100644
--- a/sidebars.json
+++ b/sidebars.json
@@ -230,14 +230,6 @@
"query/nereids/statistics"
]
},
- {
- "type": "category",
- "label": "Pipeline Execution Engine",
- "items": [
- "query/pipeline/pipeline-execution-engine",
- "query/pipeline/pipeline-x-execution-engine"
- ]
- },
{
"type": "category",
"label": "Caches",
@@ -275,6 +267,7 @@
"query/duplicate/using-hll"
]
},
+ "query/pipeline/pipeline-execution-engine",
"query/high-concurrent-point-query",
"query/topn-query",
{
diff --git
a/versioned_docs/version-3.0/query/pipeline/pipeline-execution-engine.md
b/versioned_docs/version-3.0/query/pipeline/pipeline-execution-engine.md
index 1d58fa8d4fc..d6bacb88bdd 100644
--- a/versioned_docs/version-3.0/query/pipeline/pipeline-execution-engine.md
+++ b/versioned_docs/version-3.0/query/pipeline/pipeline-execution-engine.md
@@ -27,10 +27,11 @@ under the License.
-->
+The Pipeline Execution Engine is an experimental feature in Apache Doris 2.0,
which was later optimized and upgraded in version 2.1 (i.e., PipelineX). In
versions 3.0 and later, PipelineX is used as the only execution engine in Doris
and renamed to Pipeline Execution Engine.
The goal of pipeline execution engine is to replace the current execution
engine of Doris's volcano model, fully release the computing power of
multi-core CPUs, and limit the number of Doris's query threads to solve the
problem of Doris's execution thread bloat.
-Its specific design, implementation and effects can be found in
[DSIP-027]([DSIP-027: Support Pipeline Exec Engine - DORIS - Apache Software
Foundation](https://cwiki.apache.org/confluence/display/DORIS/DSIP-027%3A+Support+Pipeline+Exec+Engine)).
+Its specific design, implementation and effects can be found in
[DSIP-027]([DSIP-027: Support Pipeline Exec Engine - DORIS - Apache Software
Foundation](https://cwiki.apache.org/confluence/display/DORIS/DSIP-027%3A+Support+Pipeline+Exec+Engine))
and [DSIP-035]([DSIP-035: PipelineX Execution Engine - DORIS - Apache Software
Foundation](https://cwiki.apache.org/confluence/display/DORIS/DSIP-035%3A+PipelineX+Execution+Engine)).
## Principle
@@ -52,6 +53,11 @@ And as shown in the figure below (quoted from[Push versus
pull-based loop fusion
1. Transformation of the traditional pull pull logic-driven execution process
into a data-driven execution engine for the push model
2. Blocking operations are asynchronous, reducing the execution overhead
caused by thread switching and thread blocking and making more efficient use of
the CPU
3. Controls the number of threads to be executed and reduces the resource
congestion of large queries on small queries in mixed load scenarios by
controlling time slice switching
+4. In terms of execution concurrency, pipelineX introduces local exchange
optimization to fully utilize CPU resources, and distribute data evenly across
different tasks to minimize data skewing. In addition, pipelineX will no longer
be constrained by the number of tablets.
+5. Logically, multiple pipeline tasks share all shared states of the same
pipeline and eliminate additional initialization overhead, such as expressions
and some const variables.
+6. In terms of scheduling logic, the blocking conditions of all pipeline tasks
are encapsulated using Dependency, and the execution logic of the tasks is
triggered by external events (such as rpc completion) to enter the runnable
queue, thereby eliminating the overhead of blocking polling threads.
+7. Profile: Provide users with simple and easy to understand metrics.
+
This improves the efficiency of CPU execution on mixed-load SQL and enhances
the performance of SQL queries.
@@ -77,6 +83,22 @@ set parallel_pipeline_task_num = 0;
You can limit the automatically configured concurrency by setting
`max_instance_num`（The default value is 64)
+3. enable_local_shuffle
+
+Set `enable_local_shuffle` to true will enable local shuffle optimization.
Local shuffle will try to evenly distribute data among different pipeline tasks
to avoid data skewing as much as possible.
+
+```sql
+set enable_local_shuffle = true;
+```
+
+4. ignore_storage_data_distribution
+
+Settings `ignore_storage_data_distribution` is true, it means ignoring the
data distribution of the storage layer. When used in conjunction with local
shuffle, the concurrency capability of the pipelineX engine will no longer be
constrained by the number of storage layer tables, thus fully utilizing machine
resources.
+
+```sql
+set ignore_storage_data_distribution = true;
+```
+
### Load
-The engine selected for import are detailed in the
[Import](../../data-operate/import/load-manual) documentation.
+The engine selected for import are detailed in the
[Load](../../data-operate/import/load-manual) documentation.
diff --git
a/versioned_docs/version-3.0/query/pipeline/pipeline-x-execution-engine.md
b/versioned_docs/version-3.0/query/pipeline/pipeline-x-execution-engine.md
deleted file mode 100644
index f926ed37141..00000000000
--- a/versioned_docs/version-3.0/query/pipeline/pipeline-x-execution-engine.md
+++ /dev/null
@@ -1,78 +0,0 @@
----
-{
- "title": "PipelineX Execution Engine",
- "language": "en",
- "toc_min_heading_level": 2,
- "toc_max_heading_level": 4
-}
----
-
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-
-
-## Background
-
-The PipelineX execution engine is an experimental feature in Doris 2.1.0,
expected to address the four major issues of the Doris pipeline engine:
-1. In terms of execution concurrency, Doris is currently constrained by two
factors: one is the parameters set by FE, and the other is limited by the
number of buckets. This concurrent strategy prevents the execution engine from
fully utilizing machine resources.
-2. In terms of execution logic, Doris currently has some fixed additional
overhead. For example, the common expression for all instances will be
initialized multiple times due to independence between all instances.
-3. In terms of scheduling logic, the scheduler of the current pipeline will
put all blocking tasks into a blocking queue, and a blocking queue scheduler
will be responsible for polling and extracting executable tasks from the
blocking queue and placing them in the runnable queue. Therefore, during the
query execution process, a CPU core will always be occupied to do scheduling
instead of execution.
-4. In terms of profile, currently the pipeline cannot provide users concise
and clear metrics.
-
-Its specific design and implementation can be found in [DSIP-035]([DSIP-035:
PipelineX Execution Engine - DORIS - Apache Software
Foundation](https://cwiki.apache.org/confluence/display/DORIS/DSIP-035%3A+PipelineX+Execution+Engine)).
-
-## Goals
-
-1. In terms of execution concurrency, pipelineX introduces local exchange
optimization to fully utilize CPU resources, and distribute data evenly across
different tasks to minimize data skewing. In addition, pipelineX will no longer
be constrained by the number of tablets.
-2. Logically, multiple pipeline tasks share all shared states of the same
pipeline and eliminate additional initialization overhead, such as expressions
and some const variables.
-3. In terms of scheduling logic, the blocking conditions of all pipeline tasks
are encapsulated using Dependency, and the execution logic of the tasks is
triggered by external events (such as rpc completion) to enter the runnable
queue, thereby eliminating the overhead of blocking polling threads.
-4. Profile: Provide users with simple and easy to understand metrics.
-
-## User Interface changes
-
-### Set session variable
-
-#### enable_pipeline_x_engine
-
-Set `enable_pipeline_x_engine ` to `true`, BE will use PipelineX to execute by
default.
-
-```
-set enable_pipeline_x_engine = true;
-```
-
-#### enable_local_shuffle
-
-Set `enable_local_shuffle` to true will enable local shuffle optimization.
Local shuffle will try to evenly distribute data among different pipeline tasks
to avoid data skewing as much as possible.
-
-```
-set enable_local_shuffle = true;
-```
-
-#### ignore_storage_data_distribution
-
-Settings `ignore_storage_data_distribution` is true, it means ignoring the
data distribution of the storage layer. When used in conjunction with local
shuffle, the concurrency capability of the pipelineX engine will no longer be
constrained by the number of storage layer tables, thus fully utilizing machine
resources.
-
-```
-set ignore_storage_data_distribution = true;
-```
-
-### Load
-
-The engine selected for import are detailed in the
[Import](../../data-operate/import/load-manual) documentation.
diff --git a/versioned_sidebars/version-3.0-sidebars.json
b/versioned_sidebars/version-3.0-sidebars.json
index f1104f2ce6d..31607737a6f 100644
--- a/versioned_sidebars/version-3.0-sidebars.json
+++ b/versioned_sidebars/version-3.0-sidebars.json
@@ -222,14 +222,6 @@
"query/nereids/statistics"
]
},
- {
- "type": "category",
- "label": "Pipeline Execution Engine",
- "items": [
- "query/pipeline/pipeline-execution-engine",
- "query/pipeline/pipeline-x-execution-engine"
- ]
- },
{
"type": "category",
"label": "Caches",
@@ -267,6 +259,7 @@
"query/duplicate/using-hll"
]
},
+ "query/pipeline/pipeline-execution-engine",
"query/high-concurrent-point-query",
"query/topn-query",
{
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
