[GitHub] [doris] github-actions[bot] commented on pull request #14394: [refactor](decimalv3) Refine code for DecimalV3
github-actions[bot] commented on PR #14394: URL: https://github.com/apache/doris/pull/14394#issuecomment-1320835577 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14394: [refactor](decimalv3) Refine code for DecimalV3
github-actions[bot] commented on PR #14394: URL: https://github.com/apache/doris/pull/14394#issuecomment-1320835583 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 merged pull request #14394: [refactor](decimalv3) Refine code for DecimalV3
Gabriel39 merged PR #14394: URL: https://github.com/apache/doris/pull/14394 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen opened a new pull request, #14418: [fix](exec) fix thread token shutdown
zhannngchen opened a new pull request, #14418: URL: https://github.com/apache/doris/pull/14418 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #14217: [test](hive catalog)add hive catalog test case
morningman merged PR #14217: URL: https://github.com/apache/doris/pull/14217 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (2c42f0a905 -> 5dfe5ef965)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 2c42f0a905 [refactor](decimalv3) Refine code for DecimalV3 (#14394) add 5dfe5ef965 [test](hive catalog)add hive catalog test case (#14217) No new revisions were added by this update. Summary of changes: .licenserc.yaml| 1 + .../docker-compose/hive/scripts/create.hql | 148 + .../hive/scripts/data_case/account_fund/data_file | 2 + .../hive/scripts/data_case/hive01/data_file| 2 + .../hive/scripts/data_case/lineorder/data_file | 4 + .../hive/scripts/data_case/sale_table/data_file| 2 + .../hive/scripts/data_case/string_table/data_file | 3 + .../hive/scripts/data_case/student/data_file | 2 + .../hive/scripts/data_case/t_hive/data_file| 2 + .../hive/scripts/data_case/test1/data_file | 4 + .../hive/scripts/data_case/test2/data_file | 3 + .../scripts/data_case/test_hive_doris/data_file| 4 + .../docker-compose/hive/scripts/hive-metastore.sh | 2 + .../docker-compose/mysql/init/04-insert.sql| 1 - .../data/hive_catalog_p0/hive_catalog_case.out | 65 + .../hive_catalog_p0/hive_catalog_case.groovy | 80 +++ 16 files changed, 324 insertions(+), 1 deletion(-) create mode 100644 docker/thirdparties/docker-compose/hive/scripts/data_case/account_fund/data_file create mode 100644 docker/thirdparties/docker-compose/hive/scripts/data_case/hive01/data_file create mode 100644 docker/thirdparties/docker-compose/hive/scripts/data_case/lineorder/data_file create mode 100644 docker/thirdparties/docker-compose/hive/scripts/data_case/sale_table/data_file create mode 100644 docker/thirdparties/docker-compose/hive/scripts/data_case/string_table/data_file create mode 100644 docker/thirdparties/docker-compose/hive/scripts/data_case/student/data_file create mode 100644 docker/thirdparties/docker-compose/hive/scripts/data_case/t_hive/data_file create mode 100644 docker/thirdparties/docker-compose/hive/scripts/data_case/test1/data_file create mode 100644 docker/thirdparties/docker-compose/hive/scripts/data_case/test2/data_file create mode 100644 docker/thirdparties/docker-compose/hive/scripts/data_case/test_hive_doris/data_file create mode 100644 regression-test/data/hive_catalog_p0/hive_catalog_case.out create mode 100644 regression-test/suites/hive_catalog_p0/hive_catalog_case.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #14418: [fix](exec) fix thread token shutdown
hello-stephen commented on PR #14418: URL: https://github.com/apache/doris/pull/14418#issuecomment-1320845830 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 36.62 seconds load time: 475 seconds storage size: 17170556941 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221119093903_clickbench_pr_49122.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring commented on pull request #14418: [fix](exec) fix thread token shutdown
dataroaring commented on PR #14418: URL: https://github.com/apache/doris/pull/14418#issuecomment-1320866552 Just modify some logs? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated (3f4740e1746 -> f1cb9c65e6f)
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git from 3f4740e1746 Datax doris writer doc fix new 692b07d947c Performance test documentation update new f1cb9c65e6f fix The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/benchmark/ssb.md | 378 --- docs/benchmark/tpch.md | 718 +++- docs/install/source-install/compilation.md | 77 +-- .../current/benchmark/ssb.md | 381 --- .../current/benchmark/tpch.md | 721 +++-- .../current/install/source-install/compilation.md | 65 +- static/images/image-20220614114351241.png | Bin 92508 -> 81771 bytes static/images/ssb_12_11_015.png| Bin 0 -> 75967 bytes static/images/ssb_v11_v015_compare.png | Bin 86762 -> 81539 bytes 9 files changed, 2028 insertions(+), 312 deletions(-) create mode 100644 static/images/ssb_12_11_015.png - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] 02/02: fix
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git commit f1cb9c65e6fe601a9fe56eb7cf767806a145d6fd Author: jiafeng.zhang AuthorDate: Fri Nov 11 16:25:00 2022 +0800 fix --- docs/install/source-install/compilation.md | 77 +- .../current/install/source-install/compilation.md | 65 +- 2 files changed, 62 insertions(+), 80 deletions(-) diff --git a/docs/install/source-install/compilation.md b/docs/install/source-install/compilation.md index e3184ee9389..3eb47e3a62a 100644 --- a/docs/install/source-install/compilation.md +++ b/docs/install/source-install/compilation.md @@ -59,31 +59,34 @@ This document focuses on how to code Doris through source code. | apache/incubator-doris:build-env-latest | before [0efef1b](https://github.com/apache/doris/commit/0efef1b332300887ee0473f9df9bdd9d7297d824) | | | apache/doris:build-env-for-1.0.0| | 1.0.0 | | apache/doris:build-env-for-1.1.0| | 1.1.0 | -| apache/doris:build-env-ldb-toolchain-latest | trunk | | +| apache/doris:build-env-ldb-toolchain-latest | | 1.1.x, trunk | +| apache/doris:build-env-ldb-toolchain-no-avx2-latest | | 1.1.x, trunk | **note**: -> 1. Dev docker image [ChangeLog](https://github.com/apache/doris/blob/master/thirdparty/CHANGELOG.md) +> 1. Third-party libraries in images with no-avx2 in their names that can run on CPUs that do not support avx2 instructions. Doris can be compiled with the USE_AVX2=0 option. -> 2. Doris version 0.14.0 still uses apache/incubator-doris:build-env-1.2 to compile, and the 0.14.x code will use apache/incubator-doris:build-env-1.3.1. +> 2. Dev docker image [ChangeLog](https://github.com/apache/doris/blob/master/thirdparty/CHANGELOG.md) -> 3. From docker image of build-env-1.3.1, both OpenJDK 8 and OpenJDK 11 are included, and OpenJDK 11 is used for compilation by default. Please make sure that the JDK version used for compiling is the same as the JDK version used at runtime, otherwise it may cause unexpected operation errors. You can use the following command to switch the default JDK version in container: -> -> Switch to JDK 8: -> -> ``` -> $ alternatives --set java java-1.8.0-openjdk.x86_64 -> $ alternatives --set javac java-1.8.0-openjdk.x86_64 -> $ export JAVA_HOME=/usr/lib/jvm/java-1.8.0 -> ``` -> -> Switch to JDK 11: -> -> ``` -> $ alternatives --set java java-11-openjdk.x86_64 -> $ alternatives --set javac java-11-openjdk.x86_64 -> $ export JAVA_HOME=/usr/lib/jvm/java-11 -> ``` +> 3. Doris version 0.14.0 still uses apache/incubator-doris:build-env-1.2 to compile, and the 0.14.x code will use apache/incubator-doris:build-env-1.3.1. + +> 4. Since the docker image of build-env-1.3.1 includes both OpenJDK 8 and OpenJDK 11, please confirm the default JDK version with `java -version`. You can also switch versions in the following ways (it is recommended to use JDK8 by default) +> +> Switch to JDK 8: +> +> ``` +> alternatives --set java java-1.8.0-openjdk.x86_64 +> alternatives --set javac java-1.8.0-openjdk.x86_64 +> export JAVA_HOME=/usr/lib/jvm/java-1.8.0 +> ``` +> +> Switch to JDK 11: +> +> ``` +> alternatives --set java java-11-openjdk.x86_64 +> alternatives --set javac java-11-openjdk.x86_64 +> export JAVA_HOME=/usr/lib/jvm/java-11 +> ``` 2. Running Mirror @@ -130,21 +133,13 @@ This document focuses on how to code Doris through source code. After compilation, the output files are in the `output/` directory. > **Note:** -> -> If you are using `build-env-for-0.15.0` or `build-env-for-1.0.0` for the first time, use the following command when compiling: -> ->``` ->sh build.sh --clean --be --fe --ui ->``` -> -> `build-env-ldb-toolchain-latest` and later, use the following command: -> ->``` ->sh build.sh --clean --be --fe ->``` -> -> This is because from build-env-for-0.15.0, we upgraded thrift (0.9 -> 0.13), you need to use the --clean command to force the use of the new version of thrift to generate code files, otherwise incompatible code will appear. - + > + > If you are using `build-env-for-0.15.0` or later version for the first time, use the following command when compiling: + > + > `sh build.sh --clean --be --fe --ui` + > + > This is because from build-env-for-0.15.0, we upgraded thrift (0.9 -> 0.13), you need to use the --clean command to force the use of the new version of thrift to generate code files, otherwise incompatible code will appear. + After compilation, the output file is in the `output/` directory. ### Self-compiling Development Environment Mirror @@ -152,7 +147,7 @@ This document focuses on how to code Doris through source code. You can also create a Doris development en
[doris-website] 01/02: Performance test documentation update
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git commit 692b07d947c33767e0b1ac2a60e44dd8423a055d Author: jiafeng.zhang AuthorDate: Thu Nov 10 11:42:10 2022 +0800 Performance test documentation update Performance test documentation update --- docs/benchmark/ssb.md | 378 --- docs/benchmark/tpch.md | 718 +++- .../current/benchmark/ssb.md | 381 --- .../current/benchmark/tpch.md | 721 +++-- static/images/image-20220614114351241.png | Bin 92508 -> 81771 bytes static/images/ssb_12_11_015.png| Bin 0 -> 75967 bytes static/images/ssb_v11_v015_compare.png | Bin 86762 -> 81539 bytes 7 files changed, 1966 insertions(+), 232 deletions(-) diff --git a/docs/benchmark/ssb.md b/docs/benchmark/ssb.md index e73ba27e9e7..39aa606ba4e 100644 --- a/docs/benchmark/ssb.md +++ b/docs/benchmark/ssb.md @@ -28,16 +28,22 @@ under the License. [Star Schema Benchmark(SSB)](https://www.cs.umb.edu/~poneil/StarSchemaB.PDF) is a performance test set in a lightweight data warehouse scenario. Based on [TPC-H](http://www.tpc.org/tpch/), SSB provides a simplified version of the star schema dataset, which is mainly used to test the performance of multi-table association queries under the star schema. . In addition, the industry usually flattens SSB as a wide table model (hereinafter referred to as: SSB flat) to test the performance of t [...] -This document mainly introduces the performance of Doris on the SSB test set. +This document mainly introduces the performance of Doris on the SSB 100G test set. > Note 1: The standard test set including SSB is usually far from the actual > business scenario, and some tests will perform parameter tuning for the test > set. Therefore, the test results of the standard test set can only reflect > the performance of the database in specific scenarios. Users are advised to > conduct further testing with actual business data. > > Note 2: The operations involved in this document are all performed in the > Ubuntu Server 20.04 environment, and CentOS 7 can also be tested. -On the 13 queries on the SSB standard test dataset, we performed peer-to-peer tests based on Doris 1.1 version and Doris 0.15.0 RC04 version, and the overall performance was improved by 2-3 times. +We conducted pairwise testing on 13 queries on the SSB standard test dataset based on Apache Doris 1.2.0-rc01, Apache Doris 1.1.3 and Apache Doris 0.15.0 RC04 versions. + +The overall performance improvement on SSB FlAT wide tables was nearly 4x on Apache Doris 1.2.0-rc01 compared to Apache Doris 1.1.3, and nearly 10x on Apache Doris 0.15.0 RC04.  +On the standard SSB test SQL, Apache Doris 1.2.0-rc01 delivers an overall performance improvement of nearly 2X over Apache Doris 1.1.3 and nearly 31X over Apache Doris 0.15.0 RC04. + + + ## 1. Hardware Environment | Number of machines | 4 Tencent Cloud hosts (1 FE, 3 BE)| @@ -52,7 +58,7 @@ On the 13 queries on the SSB standard test dataset, we performed peer-to-peer te - Doris deploys 3BE 1FE; - Kernel version: Linux version 5.4.0-96-generic (buildd@lgw01-amd64-051) - OS version: Ubuntu Server 20.04 LTS 64 bit -- Doris software version: Apache Doris 1.1, Apache Doris 0.15.0 RC04 +- Doris software version: Apache Doris 1.2.0-rc01, Apache Doris 1.1.3 , Apache Doris 0.15.0 RC04 - JDK: openjdk version "11.0.14" 2022-01-18 ## 3. Test data volume @@ -68,51 +74,68 @@ On the 13 queries on the SSB standard test dataset, we performed peer-to-peer te ## 4. Test Results -Here we use the upcoming Doris-1.1 version and Doris-0.15.0 RC04 version for comparative testing. The test results are as follows: - -| Query | Doris-1.1(ms) | Doris-0.15.0 RC04(ms) | -| - | - | - | -| Q1.1 | 90| 250 | -| Q1.2 | 10| 30| -| Q1.3 | 70| 120 | -| Q2.1 | 360 | 900 | -| Q2.2 | 340 | 1020 | -| Q2.3 | 260 | 770 | -| Q3.1 | 550 | 1710 | -| Q3.2 | 290 | 670 | -| Q3.3 | 240 | 550 | -| Q3.4 | 20| 30| -| Q4.1 | 480 | 1250 | -| Q4.2 | 240 | 400 | -| Q4.3 | 200 | 330 | +Here we use Apache Doris 1.2.0-rc01, Apache Doris 1.1.3 and Apache Doris 0.15.0 RC04 versions for comparative testing, with the following results. + +| Query | Ap
[GitHub] [doris] github-actions[bot] commented on pull request #14147: [typo](docs)Performance test documentation update[dont merge]
github-actions[bot] commented on PR #14147: URL: https://github.com/apache/doris/pull/14147#issuecomment-1320867924 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14147: [typo](docs)Performance test documentation update[dont merge]
github-actions[bot] commented on PR #14147: URL: https://github.com/apache/doris/pull/14147#issuecomment-1320867931 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: fix benchmark doc
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 044672300ec fix benchmark doc
044672300ec is described below
commit 044672300ec64fd262fb54b50ff2703aca77a489
Author: jiafeng.zhang
AuthorDate: Sat Nov 19 20:50:09 2022 +0800
fix benchmark doc
---
.../current/benchmark/ssb.md| 19 +++
.../current/benchmark/tpch.md | 4 ++--
static/images/image-20220614114351241.png | Bin 81771 -> 81771 bytes
static/images/ssb_12_11_015.png | Bin 75967 -> 84078 bytes
static/images/ssb_v11_v015_compare.png | Bin 81539 -> 83556 bytes
5 files changed, 13 insertions(+), 10 deletions(-)
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/benchmark/ssb.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/benchmark/ssb.md
index 403feb6c35e..a9c328ab53e 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/benchmark/ssb.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/benchmark/ssb.md
@@ -26,13 +26,13 @@ under the License.
# Star Schema Benchmark
-[Star Schema Benchmark(SSB)](https://www.cs.umb.edu/~poneil/StarSchemaB.PDF)
是一个轻量级的数仓场景下的性能测试集。SSB基于 [TPC-H](http://www.tpc.org/tpch/)
提供了一个简化版的星型模型数据集，主要用于测试在星型模型下，多表关联查询的性能表现。另外，业界内通常也会将SSB打平为宽表模型（以下简称：SSB
flat），来测试查询引擎的性能，参考[Clickhouse](https://clickhouse.com/docs/zh/getting-started/example-datasets/star-schema)。
+[Star Schema Benchmark(SSB)](https://www.cs.umb.edu/~poneil/StarSchemaB.PDF)
是一个轻量级的数仓场景下的性能测试集。SSB 基于 [TPC-H](http://www.tpc.org/tpch/)
提供了一个简化版的星型模型数据集，主要用于测试在星型模型下，多表关联查询的性能表现。另外，业界内通常也会将 SSB 打平为宽表模型（以下简称：SSB
flat），来测试查询引擎的性能，参考[Clickhouse](https://clickhouse.com/docs/zh/getting-started/example-datasets/star-schema)。
本文档主要介绍Apache Doris 在 SSB 100G 测试集上的性能表现。
-> 注1：包括 SSB
在内的标准测试集通常和实际业务场景差距较大，并且部分测试会针对测试集进行参数调优。所以标准测试集的测试结果仅能反映数据库在特定场景下的性能表现。建议用户使用实际业务数据进行进一步的测试。
+> 注 1：包括 SSB
在内的标准测试集通常和实际业务场景差距较大，并且部分测试会针对测试集进行参数调优。所以标准测试集的测试结果仅能反映数据库在特定场景下的性能表现。建议用户使用实际业务数据进行进一步的测试。
>
-> 注2：本文档涉及的操作都在 Ubuntu Server 20.04 环境进行，CentOS 7 也可测试。
+> 注 2：本文档涉及的操作都在 Ubuntu Server 20.04 环境进行，CentOS 7 也可测试。
在 SSB 标准测试数据集上的 13 个查询上，我们基于 Apache Doris 1.2.0-rc01， Apache Doris 1.1.3 及
Apache Doris 0.15.0 RC04 版本进行了对别测试。
@@ -55,7 +55,7 @@ under the License.
## 2. 软件环境
-- Doris部署 3BE 1FE；
+- Doris 部署 3BE 1FE；
- 内核版本：Linux version 5.4.0-96-generic (buildd@lgw01-amd64-051)
- 操作系统版本：Ubuntu Server 20.04 LTS 64位
- Doris 软件版本： Apache Doris 1.2.0-rc01、Apache Doris 1.1.3 及 Apache Doris 0.15.0
RC04
@@ -96,8 +96,8 @@ under the License.
**结果说明**
-- 测试结果对应的数据集为scale 100, 约6亿条。
-- 测试环境配置为用户常用配置，云服务器4台，16核 64G SSD，1 FE 3 BE 部署。
+- 测试结果对应的数据集为 scale 100, 约 6 亿条。
+- 测试环境配置为用户常用配置，云服务器 4 台，16 核 64G SSD，1 FE 3 BE 部署。
- 选用用户常见配置测试以降低用户选型评估成本，但整个测试过程中不会消耗如此多的硬件资源。
## 5. 标准 SSB 测试结果
@@ -127,6 +127,7 @@ under the License.
- 测试环境配置为用户常用配置，云服务器4台，16核 64G SSD，1 FE 3 BE 部署。
- 选用用户常见配置测试以降低用户选型评估成本，但整个测试过程中不会消耗如此多的硬件资源。
+
## 6. 环境准备
请先参照 [官方文档](../install/install-deploy.md) 进行 Apache Doris 的安装部署，以获得一个正常运行中的
Doris 集群（至少包含 1 FE 1 BE，推荐 1 FE 3 BE）。
@@ -257,11 +258,11 @@ PROPERTIES (
);
```
-
### 7.4 导入数据
我们使用以下命令完成 SSB 测试集所有数据导入及 SSB FLAT 宽表数据合成并导入到表里。
+
```shell
sh bin/load-ssb-data.sh -c 10
```
@@ -274,6 +275,7 @@ sh bin/load-ssb-data.sh -c 10
>
> 2. flat 表数据采用 'INSERT INTO ... SELECT ... ' 的方式导入。
+
### 7.5 检查导入数据
```sql
@@ -300,10 +302,12 @@ select count(*) from lineorder_flat;
SSB-FlAT 查询语句
：[ssb-flat-queries](https://github.com/apache/doris/tree/master/tools/ssb-tools/ssb-flat-queries)
+
标准 SSB 查询语句
：[ssb-queries](https://github.com/apache/doris/tree/master/tools/ssb-tools/ssb-queries)
7.6.1 SSB FLAT 测试 SQL
+
```sql
--Q1.1
SELECT SUM(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue
@@ -607,4 +611,3 @@ WHERE
GROUP BY d_year, s_city, p_brand
ORDER BY d_year, s_city, p_brand;
```
-
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/benchmark/tpch.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/benchmark/tpch.md
index 2ece066ab60..33d4ca138b5 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/benchmark/tpch.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/benchmark/tpch.md
@@ -1,6 +1,6 @@
---
{
-"title": "TPC-H benchmark",
+"title": "TPC-H Benchmark",
"language": "zh-CN"
}
---
@@ -24,7 +24,7 @@ specific language governing permissions and limitations
under the License.
-->
-# TPC-H benchmark
+# TPC-H Benchmark
TPC-H是一个决策支持基准（Decision Support
Benchmark），它由一套面向业务的特别查询和并发数据修改组成。查询和填充数据库的数据具有广泛的行业相关性。这个基准测试演示了检查大量数据、执行高度复杂的查询并回答关键业务问题的决策支持系统。TPC-H报告的性能指标称为TPC-H每小时复合查询性能指标(QphH@Size)，反映了系统处理查询能力的多个方面。这些方面包括执行查询时所选择的数据库大小，由单个流提交查询时的查询处理能力，以及由多个并发用户提交查询时的查询吞吐量。
diff --git a/static/images/image-20220614114351241.png
b/sta
[GitHub] [doris] xinyiZzz commented on pull request #14410: [chore](macOS) Fix linker errors
xinyiZzz commented on PR #14410: URL: https://github.com/apache/doris/pull/14410#issuecomment-1320879648 @adonis0147 In `mem_info.cpp:193`, when it is __APPLE_, there is an empty `refresh_proc_meminfo`, does this not work? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14418: [fix](exec) fix thread token shutdown
github-actions[bot] commented on PR #14418: URL: https://github.com/apache/doris/pull/14418#issuecomment-1320882868 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14418: [fix](exec) fix thread token shutdown
github-actions[bot] commented on PR #14418: URL: https://github.com/apache/doris/pull/14418#issuecomment-1320882883 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: rename image name
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 4decadcd1ef rename image name
4decadcd1ef is described below
commit 4decadcd1efacd26288e45cc3e11fbcf72931b7e
Author: jiafeng.zhang
AuthorDate: Sat Nov 19 21:19:44 2022 +0800
rename image name
---
docs/benchmark/ssb.md | 4 ++--
docs/benchmark/tpch.md | 2 +-
.../docusaurus-plugin-content-docs/current/benchmark/ssb.md | 4 ++--
.../current/benchmark/tpch.md | 2 +-
static/images/{ssb_12_11_015.png => ssb.png}| Bin
static/images/{ssb_v11_v015_compare.png => ssb_flat.png}| Bin
static/images/{image-20220614114351241.png => tpch.png} | Bin
7 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/docs/benchmark/ssb.md b/docs/benchmark/ssb.md
index 39aa606ba4e..eeca264fb6c 100644
--- a/docs/benchmark/ssb.md
+++ b/docs/benchmark/ssb.md
@@ -38,11 +38,11 @@ We conducted pairwise testing on 13 queries on the SSB
standard test dataset bas
The overall performance improvement on SSB FlAT wide tables was nearly 4x on
Apache Doris 1.2.0-rc01 compared to Apache Doris 1.1.3, and nearly 10x on
Apache Doris 0.15.0 RC04.
-
+
On the standard SSB test SQL, Apache Doris 1.2.0-rc01 delivers an overall
performance improvement of nearly 2X over Apache Doris 1.1.3 and nearly 31X
over Apache Doris 0.15.0 RC04.
-
+
## 1. Hardware Environment
diff --git a/docs/benchmark/tpch.md b/docs/benchmark/tpch.md
index 117a4610862..9a751c40cf6 100644
--- a/docs/benchmark/tpch.md
+++ b/docs/benchmark/tpch.md
@@ -36,7 +36,7 @@ This document mainly introduces the performance of Doris on
the TPC-H test set.
On 22 queries on the TPC-H standard test dataset, we conducted pairwise tests
based on Apache Doris 1.2.0-rc01, Apache Doris 1.1.3 and Apache Doris 0.15.0
RC04 versions. The overall performance of Apache Doris 1.2.0-rc01 is nearly 3
times better than that of Apache Doris 1.1.3 and nearly 11 times better than
that of Apache Doris 0.15.0 RC04.
-
+
## 1. Hardware Environment
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/benchmark/ssb.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/benchmark/ssb.md
index a9c328ab53e..b775d756901 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/benchmark/ssb.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/benchmark/ssb.md
@@ -38,11 +38,11 @@ under the License.
在 SSB FlAT 宽表上， Apache Doris 1.2.0-rc01上相对 Apache Doris 1.1.3 整体性能提升了将近4倍，相对于
Apache Doris 0.15.0 RC04 ,性能提升了将近10倍 。
-
+
在标准的 SSB 测试SQL上， Apache Doris 1.2.0-rc01 上相对 Apache Doris 1.1.3
整体性能提升了将近2倍，相对于 Apache Doris 0.15.0 RC04 ,性能提升了将近 31 倍 。
-
+
## 1. 硬件环境
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/benchmark/tpch.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/benchmark/tpch.md
index 33d4ca138b5..7f9d12f8552 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/benchmark/tpch.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/benchmark/tpch.md
@@ -36,7 +36,7 @@ TPC-H是一个决策支持基准（Decision Support Benchmark），它由一套
在 TPC-H 标准测试数据集上的 22 个查询上，我们基于 Apache Doris 1.2.0-rc01， Apache Doris 1.1.3 及
Apache Doris 0.15.0 RC04 版本进行了对别测试， Apache Doris 1.2.0-rc01上相对 Apache Doris
1.1.3 整体性能提升了将近 3 倍，相对于 Apache Doris 0.15.0 RC04 ,性能提升了将近 11 倍 。
-
+
## 1. 硬件环境
diff --git a/static/images/ssb_12_11_015.png b/static/images/ssb.png
similarity index 100%
rename from static/images/ssb_12_11_015.png
rename to static/images/ssb.png
diff --git a/static/images/ssb_v11_v015_compare.png b/static/images/ssb_flat.png
similarity index 100%
rename from static/images/ssb_v11_v015_compare.png
rename to static/images/ssb_flat.png
diff --git a/static/images/image-20220614114351241.png b/static/images/tpch.png
similarity index 100%
rename from static/images/image-20220614114351241.png
rename to static/images/tpch.png
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14147: [typo](docs)Performance test documentation update
github-actions[bot] commented on PR #14147: URL: https://github.com/apache/doris/pull/14147#issuecomment-1320884886 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on pull request #14410: [chore](macOS) Fix linker errors
adonis0147 commented on PR #14410: URL: https://github.com/apache/doris/pull/14410#issuecomment-1320900276 > @adonis0147 In `mem_info.cpp:193`, when it is __APPLE_, there is an empty `refresh_proc_meminfo`, does this not work? Hi @xinyiZzz , the error was caused by [be/src/service/doris_main.cpp:505](https://github.com/apache/doris/blob/master/be/src/service/doris_main.cpp#L505). This line references a function which doesn't have a definition. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on pull request #14410: [chore](macOS) Fix linker errors
xinyiZzz commented on PR #14410: URL: https://github.com/apache/doris/pull/14410#issuecomment-1320904440 > > @adonis0147 In `mem_info.cpp:193`, when it is __APPLE_, there is an empty `refresh_proc_meminfo`, does this not work? > > Hi @xinyiZzz , the error was caused by [be/src/service/doris_main.cpp:505](https://github.com/apache/doris/blob/master/be/src/service/doris_main.cpp#L505). This line references a function which doesn't have a definition. Thks reply @adonis0147 , I mean `refresh_proc_meminfo` defines an empty function in `mem_info.cpp:193`. Because don't want mac to call the `refresh_allocator_mem` method. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on pull request #14410: [chore](macOS) Fix linker errors
adonis0147 commented on PR #14410: URL: https://github.com/apache/doris/pull/14410#issuecomment-1320907944 > > > @adonis0147 In `mem_info.cpp:193`, when it is __APPLE_, there is an empty `refresh_proc_meminfo`, does this not work? > > > > > > Hi @xinyiZzz , the error was caused by [be/src/service/doris_main.cpp:505](https://github.com/apache/doris/blob/master/be/src/service/doris_main.cpp#L505). This line references a function which doesn't have a definition. > > Thks reply @adonis0147 , I mean `refresh_proc_meminfo` defines an empty function in `mem_info.cpp:193`. Because don't want mac to call the `refresh_allocator_mem` method. It seems that `refresh_proc_meminfo` doesn't call `refresh_allocator_mem` and the implementation of `refresh_allocator_mem` just interacts with `jemalloc` or `tcmalloc` and no platform specific logic is introduced. I don't think we need to introduce a new special definition like `refresh_proc_meminfo`. More special handlings will make the maintenance job more difficult. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #14402: [Optimize](predicate) update inplace on VcompoundPred
HappenLee commented on code in PR #14402: URL: https://github.com/apache/doris/pull/14402#discussion_r1027105960 ## regression-test/conf/regression-conf.groovy: ## @@ -23,11 +23,11 @@ defaultDb = "regression_test" // add useLocalSessionState so that the jdbc will not send // init cmd like: select @@session.tx_read_only // at each time we connect. -jdbcUrl = "jdbc:mysql://127.0.0.1:9030/?useLocalSessionState=true" +jdbcUrl = "jdbc:mysql://127.0.0.1:9333/?useLocalSessionState=true" Review Comment: do not change the default port -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #14402: [Optimize](predicate) update inplace on VcompoundPred
HappenLee commented on code in PR #14402:
URL: https://github.com/apache/doris/pull/14402#discussion_r1027106252
##
be/src/vec/exprs/vcompound_pred.h:
##
@@ -16,38 +16,174 @@
// under the License.
#pragma once
-#include "runtime/runtime_state.h"
+#include
+
+#include "common/status.h"
+#include "util/simd/bits.h"
+#include "vec/columns/column.h"
+#include "vec/columns/columns_number.h"
+#include "vec/common/assert_cast.h"
#include "vec/exprs/vectorized_fn_call.h"
#include "vec/exprs/vexpr.h"
-#include "vec/functions/function.h"
namespace doris::vectorized {
-class VcompoundPred final : public VectorizedFnCall {
+inline std::string compound_operator_to_string(TExprOpcode::type op) {
+if (op == TExprOpcode::COMPOUND_AND) {
+return "and";
+} else if (op == TExprOpcode::COMPOUND_OR) {
+return "or";
+} else {
+return "not";
+}
+}
+
+class VcompoundPred : public VectorizedFnCall {
public:
VcompoundPred(const TExprNode& node) : VectorizedFnCall(node) {
-switch (node.opcode) {
-case TExprOpcode::COMPOUND_AND:
-_fn.name.function_name = "and";
-break;
-case TExprOpcode::COMPOUND_OR:
-_fn.name.function_name = "or";
-break;
-default:
-_fn.name.function_name = "not";
-break;
+_op = node.opcode;
+_fn.name.function_name = compound_operator_to_string(_op);
+_expr_name = "CompoundPredicate (" + _fn.name.function_name + ")";
+}
+
+VExpr* clone(ObjectPool* pool) const override { return pool->add(new
VcompoundPred(*this)); }
+
+const std::string& expr_name() const override { return _expr_name; }
+
+Status execute(VExprContext* context, doris::vectorized::Block* block,
+ int* result_column_id) override {
+if (_have_invalid_child()) {
+return VectorizedFnCall::execute(context, block, result_column_id);
+}
+
+int lhs_id = -1;
+int rhs_id = -1;
+RETURN_IF_ERROR(_children[0]->execute(context, block, &lhs_id));
+ColumnPtr lhs_column = block->get_by_position(lhs_id).column;
+
+ColumnPtr rhs_column = nullptr;
+
+size_t size = lhs_column->size();
+uint8* __restrict data = _get_raw_data(lhs_column);
+int filted = simd::count_zero_num((int8_t*)data, size);
+bool full = filted == 0;
+bool empty = filted == size;
+
+const uint8* __restrict data_rhs = nullptr;
+bool full_rhs = false;
+bool empty_rhs = false;
+
+auto get_rhs_colum = [&]() {
+if (rhs_id == -1) {
+RETURN_IF_ERROR(_children[1]->execute(context, block,
&rhs_id));
+rhs_column = block->get_by_position(rhs_id).column;
+data_rhs = _get_raw_data(rhs_column);
+if (!empty) {
+if (const uint8* null_map =
+
_get_null_map(block->get_by_position(rhs_id).column);
+null_map != nullptr) {
+for (size_t i = 0; i < size; i++) {
+data[i] &= !null_map[i];
+}
+}
+}
+int filted = simd::count_zero_num((int8_t*)data_rhs, size);
+full_rhs = filted == 0;
+empty_rhs = filted == size;
+}
+return Status::OK();
+};
+
+if (_op == TExprOpcode::COMPOUND_AND) {
+if (!empty) { // empty and any = empty, so lhs should not empty
+RETURN_IF_ERROR(get_rhs_colum());
+if (empty_rhs) { // any and empty = empty
+*result_column_id = rhs_id;
+return Status::OK();
+} else if (!full_rhs) { // any and full = any, so rhs should
not full.
+for (size_t i = 0; i < size; i++) {
+data[i] &= data_rhs[i];
+}
+}
+}
+} else if (_op == TExprOpcode::COMPOUND_OR) {
+if (!full) { // full or any = full, so lhs should not full
+RETURN_IF_ERROR(get_rhs_colum());
+if (full_rhs) { // any or full = full
+*result_column_id = rhs_id;
+return Status::OK();
+} else if (!empty_rhs) { // any or empty = any, so rhs should
not empty
+for (size_t i = 0; i < size; i++) {
+data[i] |= data_rhs[i];
+}
+}
+}
+} else {
+for (size_t i = 0; i < size; i++) {
+data[i] = !data[i];
+}
}
+
+*result_column_id = lhs_id;
+return Status::OK();
}
std::string debug_string() const override {
std::stringstream out;
-out << "CompoundPredicate {" << _fn.name.function_na
[GitHub] [doris] HappenLee commented on pull request #14417: [bug](test) fix regression test of jdbc postgresql table core
HappenLee commented on PR #14417: URL: https://github.com/apache/doris/pull/14417#issuecomment-1320911152 why the fix just fix the case ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee commented on pull request #14393: [Vectorized](udaf) fix java-udaf couldn't get jar core dump
HappenLee commented on PR #14393: URL: https://github.com/apache/doris/pull/14393#issuecomment-1320914956 Should do the check and return error Status in create function be called ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #14418: [fix](exec) fix thread token shutdown
morningman merged PR #14418: URL: https://github.com/apache/doris/pull/14418 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #14414: [fix](test) sync conf used in pipeline and in repository
morningman merged PR #14414: URL: https://github.com/apache/doris/pull/14414 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (3e1e8db173 -> 3489f4826c)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 3e1e8db173 [fix](exec) fix thread token shutdown (#14418) add 3489f4826c [fix](test) sync conf used in pipeline and in repository (#14414) No new revisions were added by this update. Summary of changes: regression-test/pipeline/p0/conf/fe.conf | 8 ++-- regression-test/pipeline/p1/conf/fe.conf | 8 ++-- 2 files changed, 4 insertions(+), 12 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](exec) fix thread token shutdown (#14418)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 3e1e8db173 [fix](exec) fix thread token shutdown (#14418)
3e1e8db173 is described below
commit 3e1e8db173bc1eb36a4ffa84d2d011ff8e1f8721
Author: zhannngchen <48427519+zhannngc...@users.noreply.github.com>
AuthorDate: Sun Nov 20 00:04:48 2022 +0800
[fix](exec) fix thread token shutdown (#14418)
Fix Thread pool token was shut down error.
This is because when there are more than 1 fragment of a query on one BE,
the thread token maybe
reset incorrectly, causing thread token shutdown earlier.
cherry-pick from master
Introduced from #13021
---
be/src/runtime/fragment_mgr.cpp | 11 ++-
be/src/runtime/plan_fragment_executor.cpp | 4 +++-
fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java | 5 +++--
3 files changed, 12 insertions(+), 8 deletions(-)
diff --git a/be/src/runtime/fragment_mgr.cpp b/be/src/runtime/fragment_mgr.cpp
index 57d2a05f1e..29cdd17320 100644
--- a/be/src/runtime/fragment_mgr.cpp
+++ b/be/src/runtime/fragment_mgr.cpp
@@ -255,8 +255,9 @@ Status FragmentExecState::execute() {
CgroupsMgr::apply_system_cgroup();
opentelemetry::trace::Tracer::GetCurrentSpan()->AddEvent("start
executing Fragment");
Status status = _executor.open();
-WARN_IF_ERROR(status, strings::Substitute("Got error while opening
fragment $0",
-
print_id(_fragment_instance_id)));
+WARN_IF_ERROR(status,
+ strings::Substitute("Got error while opening fragment
$0, query id: $1",
+ print_id(_fragment_instance_id),
print_id(_query_id)));
_executor.close();
if (!status.ok()) {
@@ -403,7 +404,8 @@ void FragmentExecState::coordinator_callback(const Status&
status, RuntimeProfil
<< apache::thrift::ThriftDebugString(params).c_str();
if (!exec_status.ok()) {
LOG(WARNING) << "report error status: " << exec_status.to_string()
- << " to coordinator: " << _coord_addr;
+ << " to coordinator: " << _coord_addr << ", query id: "
<< print_id(_query_id)
+ << ", instance id: " << print_id(_fragment_instance_id);
}
try {
try {
@@ -630,7 +632,6 @@ Status FragmentMgr::exec_plan_fragment(const
TExecPlanFragmentParams& params, Fi
BackendOptions::get_localhost());
}
fragments_ctx = search->second;
-_set_scan_concurrency(params, fragments_ctx.get());
} else {
// This may be a first fragment request of the query.
// Create the query fragments context.
@@ -689,7 +690,7 @@ Status FragmentMgr::exec_plan_fragment(const
TExecPlanFragmentParams& params, Fi
<< print_id(fragments_ctx->query_id)
<< " limit: " << PrettyPrinter::print(bytes_limit,
TUnit::BYTES);
} else {
-// Already has a query fragmentscontext, use it
+// Already has a query fragments context, use it
fragments_ctx = search->second;
}
}
diff --git a/be/src/runtime/plan_fragment_executor.cpp
b/be/src/runtime/plan_fragment_executor.cpp
index 8209c82e83..3a9206f1fa 100644
--- a/be/src/runtime/plan_fragment_executor.cpp
+++ b/be/src/runtime/plan_fragment_executor.cpp
@@ -626,7 +626,9 @@ void PlanFragmentExecutor::update_status(const Status&
new_status) {
void PlanFragmentExecutor::cancel(const PPlanFragmentCancelReason& reason,
const std::string& msg) {
LOG_INFO("PlanFragmentExecutor::cancel")
.tag("query_id", _query_id)
-.tag("instance_id", _runtime_state->fragment_instance_id());
+.tag("instance_id", _runtime_state->fragment_instance_id())
+.tag("reason", reason)
+.tag("error message", msg);
DCHECK(_prepared);
_cancel_reason = reason;
_cancel_msg = msg;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
index 035ed03cfe..a89aa3ef42 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
@@ -1695,8 +1695,9 @@ public class Coordinator {
// and returned_all_results_ is true.
// (UpdateStatus() initiates cancellation, if it hasn't already been
initiated)
if (!(returnedAllResults && status.isCancelled()) && !status.ok()) {
-LOG.warn("one instance report fail, query_id={} instance_id={}",
-DebugUtil.printId(queryId),
DebugUtil.printId(params.getFragmentInst
[GitHub] [doris] morningman merged pull request #14406: (improvement)[doc] add document version tag instruction
morningman merged PR #14406: URL: https://github.com/apache/doris/pull/14406 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (3489f4826c -> 2ccb5209a0)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 3489f4826c [fix](test) sync conf used in pipeline and in repository (#14414) add 2ccb5209a0 (improvement)[doc] add document version tag instruction (#14406) No new revisions were added by this update. Summary of changes: .../community/how-to-contribute/contribute-doc.md | 121 - .../community/how-to-contribute/contribute-doc.md | 121 - 2 files changed, 240 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #8789: [fix] update RewriteBinaryPredicatesRule and consider special decimalLiteral (0.0 is equal to 0)
github-actions[bot] commented on PR #8789: URL: https://github.com/apache/doris/pull/8789#issuecomment-1320997188 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hf200012 merged pull request #14147: [typo](docs)Performance test documentation update
hf200012 merged PR #14147: URL: https://github.com/apache/doris/pull/14147 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [typo](docs)Performance test documentation update (#14147)
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 71e80e8957 [typo](docs)Performance test documentation update (#14147) 71e80e8957 is described below commit 71e80e8957855095e291bd92bd29d6dca1bd2d89 Author: jiafeng.zhang AuthorDate: Sun Nov 20 09:40:57 2022 +0800 [typo](docs)Performance test documentation update (#14147) * Performance test documentation update --- docs/en/docs/benchmark/ssb.md | 390 docs/en/docs/benchmark/tpch.md | 750 +-- docs/images/image-20220614114351241.png | Bin 92508 -> 0 bytes docs/images/ssb.png | Bin 0 -> 84078 bytes docs/images/ssb_flat.png| Bin 0 -> 83556 bytes docs/images/ssb_v11_v015_compare.png| Bin 86762 -> 0 bytes docs/images/tpch.png| Bin 0 -> 81771 bytes docs/sidebars.json | 8 + docs/zh-CN/docs/benchmark/ssb.md| 409 + docs/zh-CN/docs/benchmark/tpch.md | 757 +--- 10 files changed, 2055 insertions(+), 259 deletions(-) diff --git a/docs/en/docs/benchmark/ssb.md b/docs/en/docs/benchmark/ssb.md index 900909485e..098a3a0a27 100644 --- a/docs/en/docs/benchmark/ssb.md +++ b/docs/en/docs/benchmark/ssb.md @@ -28,15 +28,21 @@ under the License. [Star Schema Benchmark(SSB)](https://www.cs.umb.edu/~poneil/StarSchemaB.PDF) is a performance test set in a lightweight data warehouse scenario. Based on [TPC-H](http://www.tpc.org/tpch/), SSB provides a simplified version of the star schema dataset, which is mainly used to test the performance of multi-table association queries under the star schema. . In addition, the industry usually flattens SSB as a wide table model (hereinafter referred to as: SSB flat) to test the performance of t [...] -This document mainly introduces the performance of Doris on the SSB test set. +This document mainly introduces the performance of Doris on the SSB 100G test set. > Note 1: The standard test set including SSB is usually far from the actual > business scenario, and some tests will perform parameter tuning for the test > set. Therefore, the test results of the standard test set can only reflect > the performance of the database in specific scenarios. Users are advised to > conduct further testing with actual business data. > > Note 2: The operations involved in this document are all performed in the > Ubuntu Server 20.04 environment, and CentOS 7 can also be tested. -On 13 queries on the SSB standard test dataset, we tested the upcoming Doris 1.1 version and Doris 0.15.0 RC04 version peer-to-peer, and the overall performance improved by 2-3 times. +We conducted pairwise testing on 13 queries on the SSB standard test dataset based on Apache Doris 1.2.0-rc01, Apache Doris 1.1.3 and Apache Doris 0.15.0 RC04 versions. - +The overall performance improvement on SSB FlAT wide tables was nearly 4x on Apache Doris 1.2.0-rc01 compared to Apache Doris 1.1.3, and nearly 10x on Apache Doris 0.15.0 RC04. + + + +On the standard SSB test SQL, Apache Doris 1.2.0-rc01 delivers an overall performance improvement of nearly 2X over Apache Doris 1.1.3 and nearly 31X over Apache Doris 0.15.0 RC04. + + ## 1. Hardware Environment @@ -52,7 +58,7 @@ On 13 queries on the SSB standard test dataset, we tested the upcoming Doris 1.1 - Doris deploys 3BE 1FE; - Kernel version: Linux version 5.4.0-96-generic (buildd@lgw01-amd64-051) - OS version: Ubuntu Server 20.04 LTS 64 bit -- Doris software version: Apache Doris 1.1, Apache Doris 0.15.0 RC04 +- Doris software version: Apache Doris 1.2.0-rc01, Apache Doris 1.1.3 , Apache Doris 0.15.0 RC04 - JDK: openjdk version "11.0.14" 2022-01-18 ## 3. Test data volume @@ -68,111 +74,129 @@ On 13 queries on the SSB standard test dataset, we tested the upcoming Doris 1.1 ## 4. Test Results -Here we use the upcoming Doris-1.1 version and Doris-0.15.0 RC04 version for comparative testing. The test results are as follows: - -| Query | Doris-1.1(ms) | Doris-0.15.0 RC04(ms) | -| - | - | - | -| Q1.1 | 90| 250 | -| Q1.2 | 10| 30| -| Q1.3 | 70| 120 | -| Q2.1 | 360 | 900 | -| Q2.2 | 340 | 1020 | -| Q2.3 | 260 | 770 | -| Q3.1 | 550 | 1710 | -| Q3.2 | 290 | 670 | -| Q3.3 | 240 | 550 | -| Q3.4 | 20| 30| -| Q4.1
[GitHub] [doris] yinzhijian commented on a diff in pull request #14377: [improvement](planner) eliminating useless sort node
yinzhijian commented on code in PR #14377:
URL: https://github.com/apache/doris/pull/14377#discussion_r1027225986
##
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##
@@ -586,6 +588,9 @@ public class SessionVariable implements Serializable,
Writable {
@VariableMgr.VarAttr(name = ENABLE_NEREIDS_STATS_DERIVE_V2)
public boolean enableNereidsStatsDeriveV2 = false;
+@VariableMgr.VarAttr(name = ENABLE_ELIMINATE_SORT_NODE)
+public boolean enableEliminateSortNode = true;
Review Comment:
The main concern is that there may be an unknown problem that can be
recovered by turning it off
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz opened a new pull request, #14419: [Docs](memory) Admin-manual adds mem tracker, memory exceeds limit, OOM analysis
xinyiZzz opened a new pull request, #14419: URL: https://github.com/apache/doris/pull/14419 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly opened a new pull request, #14421: [opt](nereids) make fragment id in explain get inline with profile
englefly opened a new pull request, #14421: URL: https://github.com/apache/doris/pull/14421 # Proposed changes Nereids assign fragment ID in its own way. The fragment Id in explain is different from the fragment id in profile. This difference makes trouble to understand profile. Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] lsy3993 opened a new pull request, #14422: [test](jdbc table)add new mysql jdbc external table
lsy3993 opened a new pull request, #14422: URL: https://github.com/apache/doris/pull/14422 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [x] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [x] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen commented on pull request #14418: [fix](exec) fix thread token shutdown
zhannngchen commented on PR #14418: URL: https://github.com/apache/doris/pull/14418#issuecomment-1321060430 > Just modify some logs? removed `_set_scan_concurrency(params, fragments_ctx.get())`, this method call will set _thread_token duplicately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #14421: [opt](nereids) make fragment id in explain get inline with profile
hello-stephen commented on PR #14421: URL: https://github.com/apache/doris/pull/14421#issuecomment-1321062065 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.5 seconds load time: 436 seconds storage size: 17150514423 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221120074309_clickbench_pr_49273.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #14422: [test](jdbc table)add new mysql jdbc external table
hello-stephen commented on PR #14422: URL: https://github.com/apache/doris/pull/14422#issuecomment-1321062821 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 35.76 seconds load time: 435 seconds storage size: 17151083938 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221120074820_clickbench_pr_49276.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
