(doris) branch master updated (3493a10c3fa -> 8f2d075cb89)
This is an automated email from the ASF dual-hosted git repository. huajianlan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 3493a10c3fa [opt](log)password should not be output in log (#34324) add 8f2d075cb89 [refactor](nereids) Remove SlotBinder and FunctionBinder (#36872) No new revisions were added by this update. Summary of changes: .../doris/nereids/rules/analysis/BindSink.java | 57 ++-- .../nereids/rules/analysis/ExpressionAnalyzer.java | 107 -- .../doris/nereids/rules/analysis/SlotBinder.java | 320 -- .../nereids/rules/analysis/SubExprAnalyzer.java| 3 + .../rules/expression/rules/FunctionBinder.java | 365 - .../expressions/functions/udf/AliasUdfBuilder.java | 30 +- .../trees/plans/commands/UpdateCommand.java| 6 +- .../rules/analysis/FillUpMissingSlotsTest.java | 7 +- .../expression/ExpressionRewriteTestHelper.java| 4 +- .../nereids/rules/expression/FoldConstantTest.java | 6 +- .../expression/SimplifyArithmeticRuleTest.java | 8 +- .../rules/expression/SimplifyRangeTest.java| 4 +- 12 files changed, 141 insertions(+), 776 deletions(-) delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SlotBinder.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/FunctionBinder.java - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running notifications feature from refs/heads/master:.asf.yaml in doris-website!
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@foo.apache.org'. Must be a valid @doris.apache.org list! - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch master updated: [doc](bi)document for quicksight (#794)
This is an automated email from the ASF dual-hosted git repository.
luzhijing pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 2a8eea03bd6 [doc](bi)document for quicksight (#794)
2a8eea03bd6 is described below
commit 2a8eea03bd69282cb6786ab41e55ea581c7976cc
Author: wyx123654 <45102878+wyx123...@users.noreply.github.com>
AuthorDate: Thu Jun 27 15:02:54 2024 +0800
[doc](bi)document for quicksight (#794)
---
docs/ecosystem/bi/quicksight.md| 53
.../current/ecosystem/bi/quicksight.md | 55 +
.../version-2.0/ecosystem/bi/quicksight.md | 55 +
.../version-2.1/ecosystem/bi/quicksight.md | 55 +
sidebars.json | 3 +-
static/images/bi-quicksight-en-1.png | Bin 0 -> 18221 bytes
static/images/bi-quicksight-en-2.png | Bin 0 -> 347383 bytes
static/images/bi-quicksight-en-3.png | Bin 0 -> 112916 bytes
static/images/bi-quicksight-en-4.png | Bin 0 -> 284997 bytes
static/images/bi-quicksight-en-5.png | Bin 0 -> 355584 bytes
static/images/bi-quicksight-en-6.png | Bin 0 -> 226864 bytes
.../version-2.0/ecosystem/bi/quicksight.md | 53
.../version-2.1/ecosystem/bi/quicksight.md | 53
versioned_sidebars/version-2.0-sidebars.json | 3 +-
versioned_sidebars/version-2.1-sidebars.json | 3 +-
15 files changed, 330 insertions(+), 3 deletions(-)
diff --git a/docs/ecosystem/bi/quicksight.md b/docs/ecosystem/bi/quicksight.md
new file mode 100644
index 000..54349060559
--- /dev/null
+++ b/docs/ecosystem/bi/quicksight.md
@@ -0,0 +1,53 @@
+---
+{
+"title": "QuickSight",
+"language": "en"
+}
+---
+
+
+
+
+## Introduction
+QuickSight is a robust data visualization and analysis platform that
seamlessly integrates data computation with visually appealing charts. It does
not necessitate programming skills from users, as data insights can be quickly
obtained through simple drag-and-drop operations. QuickSight offers a diverse
range of viewing options for exploring data from various perspectives.
Furthermore, it facilitates easy integration of multiple data sources,
simplifying and streamlining the process of p [...]
+
+## Data configuration
+
+1. Upon completing the registration process, please proceed to sign in to
QuickSight and choose the desired dataset.
+
+ 
+
+2. Select MySQL on the dataset selection page.
+
+ 
+
+3. On the data connection page, fill in the corresponding IP, port, database
name, username, and password for Doris. After filling in the information, click
"Verify the Connection". If the test is successful, create a data source.
+
+ 
+
+4. Select the table for which you want to create the dashboard.
+
+ 
+
+5. QuickSight offers two methods for importing tables. You can choose the
appropriate import method based on your needs.
+
+ 
+
+6. After importing the table, you can create the desired dashboard.
+
+ 
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/ecosystem/bi/quicksight.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/ecosystem/bi/quicksight.md
new file mode 100644
index 000..95fab90735e
--- /dev/null
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/ecosystem/bi/quicksight.md
@@ -0,0 +1,55 @@
+---
+{
+"title": "QuickSight",
+"language": "zh-CN"
+}
+---
+
+
+
+
+## QuickSight 介绍
+QuickSight
是一个强大的数据可视化分析平台，致力于将数据计算与美观的图表完美融合。它不要求用户具备编程技能，仅需通过拖放操作，就能迅速获取数据洞察。QuickSight
提供了多种视图选项，方便用户从不同角度探索数据。此外，它还支持轻松整合多个数据源，使得数据展示、探索和分析变得更加简单高效。这个平台不仅让数据处理变得快捷，还使分析过程更加直观和易于理解，极大地提升了用户的工作效率和分析能力。
+
+## 数据配置
+
+1. 注册账号后登录 QuickSight，然后选择数据集。
+
+ 
+
+2. 在数据集选择的页面中选择 MySQL。
+
+ 
+
+3. 在数据连接页面中填写对应 Doris 的 IP、端口、数据库名称、登录的用户名和密码。填写后点击连接测试，测试成功后创建链接。
+
+ 
+
+4. 创建数据源之后选择需要产生看板的表。
+
+ 
+
+5. QuickSight 有两种导入表的方式，可以根据自己的需求选择对应的导入方式。
+
+ 
+
+6. 导入表之后，即可以创建所需要的看板。
+
+ 
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/ecosystem/bi/quicksight.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/ecosystem/bi/quicksight.md
new file mode 100644
index 000.
Error while running notifications feature from refs/heads/master:.asf.yaml in doris-website!
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@foo.apache.org'. Must be a valid @doris.apache.org list! - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch master updated (2a8eea03bd6 -> c34f042646d)
This is an automated email from the ASF dual-hosted git repository. luzhijing pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git from 2a8eea03bd6 [doc](bi)document for quicksight (#794) add c34f042646d [doc] optimize 2 Getting Started documents in dev & 2.1 version (#767) No new revisions were added by this update. Summary of changes: docs/get-starting/quick-start.md | 26 ++-- docs/get-starting/what-is-apache-doris.md | 26 ++-- .../current/get-starting/quick-start.md| 48 -- .../current/get-starting/what-is-apache-doris.md | 40 +- .../version-2.1/get-starting/quick-start.md| 38 - .../get-starting/what-is-apache-doris.md | 40 -- .../version-2.1/get-starting/quick-start.md| 26 ++-- .../get-starting/what-is-apache-doris.md | 20 - 8 files changed, 132 insertions(+), 132 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (8f2d075cb89 -> 1d2c42cfda7)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 8f2d075cb89 [refactor](nereids) Remove SlotBinder and FunctionBinder (#36872) add 1d2c42cfda7 [fix](loadcase) error message changes (#36903) No new revisions were added by this update. Summary of changes: .../suites/load_p2/broker_load/test_broker_load.groovy | 13 ++--- 1 file changed, 10 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (1d2c42cfda7 -> a89bb1da2f3)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 1d2c42cfda7 [fix](loadcase) error message changes (#36903) add a89bb1da2f3 [test](migrate) move 2 cases from p2 to p0 (#36787) No new revisions were added by this update. Summary of changes: .licenserc.yaml| 2 +- .../data/multi_catalog/datev2_csv/create_table.hql | 20 +++ .../data/multi_catalog/datev2_csv/data.tar.gz | Bin 0 -> 184 bytes .../scripts/data/multi_catalog/datev2_csv/run.sh | 13 ++ .../data/multi_catalog/datev2_orc/create_table.hql | 20 +++ .../data/multi_catalog/datev2_orc/data.tar.gz | Bin 0 -> 459 bytes .../scripts/data/multi_catalog/datev2_orc/run.sh | 13 ++ .../multi_catalog/datev2_parquet/create_table.hql | 20 +++ .../data/multi_catalog/datev2_parquet/data.tar.gz | Bin 0 -> 384 bytes .../data/multi_catalog/datev2_parquet/run.sh | 13 ++ .../hive_upper_case_orc/create_table.hql | 21 +++ .../multi_catalog/hive_upper_case_orc/data.tar.gz | Bin 0 -> 521 bytes .../data/multi_catalog/hive_upper_case_orc/run.sh | 13 ++ .../hive_upper_case_parquet/create_table.hql | 21 +++ .../hive_upper_case_parquet/data.tar.gz| Bin 0 -> 690 bytes .../multi_catalog/hive_upper_case_parquet/run.sh | 13 ++ .../parquet_lz4_compression/create_table.hql | 31 .../parquet_lz4_compression/data.tar.gz| Bin 0 -> 11375 bytes .../multi_catalog/parquet_lz4_compression/run.sh | 13 ++ .../parquet_lzo_compression/create_table.hql | 31 .../parquet_lzo_compression/data.tar.gz| Bin 0 -> 28433 bytes .../multi_catalog/parquet_lzo_compression/run.sh | 13 ++ .../test_compress_partitioned}/create_table.hql| 56 +-- .../multi_catalog/test_compress_partitioned/run.sh | 22 +++ .../data/regression/crdmm_data/create_table.hql| 162 + .../scripts/data/regression/crdmm_data/data.tar.gz | Bin 0 -> 2035 bytes .../hive/scripts/data/regression/crdmm_data/run.sh | 14 ++ .../docker-compose/hive/scripts/hive-metastore.sh | 13 +- .../scripts/suites/test_hive_compress_type/run.sh | 22 --- .../suites/test_hive_to_date/create_table.hql | 52 --- .../scripts/suites/test_hive_to_date/data.tar.gz | Bin 745 -> 0 bytes .../hive/scripts/suites/test_hive_to_date/run.sh | 12 -- docker/thirdparties/run-thirdparties-docker.sh | 4 +- .../hive/test_external_credit_data.out | 12 ++ .../hive/test_upper_case_column_name.out | 89 +++ .../hive/hive_tpch_sf1_orc.groovy | 4 +- .../hive/hive_tpch_sf1_parquet.groovy | 4 +- .../hive/test_external_credit_data.groovy | 29 ++-- .../hive/test_upper_case_column_name.groovy| 103 + 39 files changed, 685 insertions(+), 170 deletions(-) create mode 100644 docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_csv/create_table.hql create mode 100644 docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_csv/data.tar.gz create mode 100755 docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_csv/run.sh create mode 100644 docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_orc/create_table.hql create mode 100644 docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_orc/data.tar.gz create mode 100755 docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_orc/run.sh create mode 100644 docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_parquet/create_table.hql create mode 100644 docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_parquet/data.tar.gz create mode 100755 docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_parquet/run.sh create mode 100644 docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_upper_case_orc/create_table.hql create mode 100644 docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_upper_case_orc/data.tar.gz create mode 100755 docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_upper_case_orc/run.sh create mode 100644 docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_upper_case_parquet/create_table.hql create mode 100644 docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_upper_case_parquet/data.tar.gz create mode 100755 docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_upper_case_parquet/run.sh create mode 100644 docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/parquet_lz4_compression/create_table.hql create mode 100644 docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/parquet_lz4_compression/data.tar.gz create mode
(doris) branch branch-1.2-lts updated: [fix](fe) partitionInfo may throw NPE cause fe can not start (#36875)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new 9087f1e48ac [fix](fe) partitionInfo may throw NPE cause fe can not
start (#36875)
9087f1e48ac is described below
commit 9087f1e48acb2fe00a7493c505bfef8dc74bb23d
Author: htyoung
AuthorDate: Thu Jun 27 15:27:09 2024 +0800
[fix](fe) partitionInfo may throw NPE cause fe can not start (#36875)
## Proposed changes
Issue Number: close #32690
Similar PR：https://github.com/apache/doris/pull/33108
Co-authored-by: tongyang.han
---
fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java | 1 +
1 file changed, 1 insertion(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java
index 98033041621..1d3c551a588 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java
@@ -523,6 +523,7 @@ public class CatalogRecycleBin extends MasterDaemon
implements Writable {
if (partitionInfo == null) {
LOG.error("replayErasePartition: partitionInfo is null for
partitionId[{}]", partitionId);
+return;
}
Partition partition = partitionInfo.getPartition();
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris-thirdparty) branch clucene-2.0 updated: add error code FileNotFound (#228)
This is an automated email from the ASF dual-hosted git repository. airborne pushed a commit to branch clucene-2.0 in repository https://gitbox.apache.org/repos/asf/doris-thirdparty.git The following commit(s) were added to refs/heads/clucene-2.0 by this push: new 51f15724f5 add error code FileNotFound (#228) 51f15724f5 is described below commit 51f15724f5bdb7ae5f6f5e5d7072d43a5bda63f8 Author: Sun Chenyang AuthorDate: Thu Jun 27 15:55:12 2024 +0800 add error code FileNotFound (#228) --- src/core/CLucene/debug/error.h | 1 + 1 file changed, 1 insertion(+) diff --git a/src/core/CLucene/debug/error.h b/src/core/CLucene/debug/error.h index 0c48db3d28..5c5c620352 100644 --- a/src/core/CLucene/debug/error.h +++ b/src/core/CLucene/debug/error.h @@ -34,6 +34,7 @@ #define CL_ERR_OutOfMemory 23 #define CL_ERR_FieldReader 24 #define CL_ERR_MaxBytesLength 25 +#define CL_ERR_FileNotFound 26 //error try/throw/catch definitions - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-streamloader) branch master updated: fix streamloader use too much memory (#19)
This is an automated email from the ASF dual-hosted git repository.
liaoxin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-streamloader.git
The following commit(s) were added to refs/heads/master by this push:
new 3a3664b fix streamloader use too much memory (#19)
3a3664b is described below
commit 3a3664b5fb62bbe7259b8084e41a147d770bddef
Author: hui lai <1353307...@qq.com>
AuthorDate: Thu Jun 27 15:57:15 2024 +0800
fix streamloader use too much memory (#19)
---
main.go | 10 +-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/main.go b/main.go
index 1677b4f..7d08378 100644
--- a/main.go
+++ b/main.go
@@ -38,7 +38,7 @@ import (
const (
fileBufferSize = 16 * 1024 * 1024 // 16MB
bufferSize = 1 * 1024 * 1024 // 1MB
- queueSize = 50 * 1024 * 1024 //50MB
+ defaultQueueSize= 100
defaultTimeout = 60 * 60 * 10
defaultBatchRows= 4096
defaultBatchBytes = 943718400
@@ -80,6 +80,7 @@ var (
retryIntervalint
retryInfomap[int]int
showVersion bool
+ queueSizeint
bufferPool = sync.Pool{
New: func() interface{} {
@@ -114,6 +115,7 @@ func initFlags() {
flag.IntVar(&retryInterval, "auto_retry_interval",
defaultRetryInterval, "retry failure")
flag.BoolVar(&debug, "debug", false, "enable debug")
flag.BoolVar(&showVersion, "version", false, "Display the version")
+ flag.IntVar(&queueSize, "queue_size", defaultQueueSize, "memory queue
size")
flag.Parse()
@@ -183,6 +185,7 @@ func initFlags() {
fmt.Println("retry_info: ", retry)
fmt.Println("retry_times: ", maxRetryTimes)
fmt.Println("retry_interval: ", retryInterval)
+ fmt.Println("queue_size: ", queueSize)
}
utils.InitLog(logLevel)
@@ -291,6 +294,11 @@ func paramCheck() {
log.Warnf("retryInterval invalid: %d, replace with default
value: %d", retryInterval, defaultRetryInterval)
retryInterval = defaultRetryInterval
}
+
+ if queueSize <= 0 {
+ log.Warnf("queueSize invalid: %d, replace with default value:
%d", queueSize, defaultQueueSize)
+ queueSize = defaultQueueSize
+ }
}
func calculateAndCheckWorkers(reader *file.FileReader, size int64) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [opt](s3) auto retry when meeting 429 error (#35396)
This is an automated email from the ASF dual-hosted git repository.
gavinchou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 493dc26d743 [opt](s3) auto retry when meeting 429 error (#35396)
493dc26d743 is described below
commit 493dc26d7435d9e78fe87a873b5174df9e64a886
Author: Mingyu Chen
AuthorDate: Thu Jun 27 16:01:23 2024 +0800
[opt](s3) auto retry when meeting 429 error (#35396)
Sometime the s3 sdk will return error like:
```
QpsLimitExceeded Unable to parse ExceptionName: QpsLimitExceeded Message:
Please reduce your request rate. code=429 type=100, request_id=66516C288EC49
```
We should slowdown the request rate by sleeping for a while.
- Add 2 new BE config
- `s3_read_base_wait_time_ms` and `s3_read_max_wait_time_ms`
When meet s3 429 error, the "get" request will
sleep `s3_read_base_wait_time_ms (*1, *2, *3, *4)` ms get
try again.
The max sleep time is s3_read_max_wait_time_ms
and the max retry time is max_s3_client_retry
- Add more metrics for s3 file reader
- `s3_file_reader_too_many_request`: counter of 429 error.
- `s3_file_reader_s3_get_request`: the QPS of s3 get request.
- `TotalGetRequest`: Get request counter in profile
- `TooManyRequestErr`: 429 error counter in profile
- `TooManyRequestSleepTime`: Sum of sleep time after 429 error in
profile
- `TotalBytesRead`: Total bytes read from s3 in profile
---
be/src/common/config.cpp| 2 +
be/src/common/config.h | 7 +++
be/src/io/file_factory.cpp | 2 +-
be/src/io/fs/s3_file_reader.cpp | 109
be/src/io/fs/s3_file_reader.h | 24 +++--
be/src/io/fs/s3_file_system.cpp | 2 +-
6 files changed, 120 insertions(+), 26 deletions(-)
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index 4460e477c8f..a08bb43db56 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -1222,6 +1222,8 @@ DEFINE_Int32(spill_io_thread_pool_queue_size, "102400");
DEFINE_mBool(check_segment_when_build_rowset_meta, "false");
DEFINE_mInt32(max_s3_client_retry, "10");
+DEFINE_mInt32(s3_read_base_wait_time_ms, "100");
+DEFINE_mInt32(s3_read_max_wait_time_ms, "800");
DEFINE_mBool(enable_s3_rate_limiter, "false");
diff --git a/be/src/common/config.h b/be/src/common/config.h
index dbf18002704..6942e316dcd 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -1304,6 +1304,13 @@ DECLARE_mBool(check_segment_when_build_rowset_meta);
DECLARE_mBool(enable_s3_rate_limiter);
// max s3 client retry times
DECLARE_mInt32(max_s3_client_retry);
+// When meet s3 429 error, the "get" request will
+// sleep s3_read_base_wait_time_ms (*1, *2, *3, *4) ms
+// get try again.
+// The max sleep time is s3_read_max_wait_time_ms
+// and the max retry time is max_s3_client_retry
+DECLARE_mInt32(s3_read_base_wait_time_ms);
+DECLARE_mInt32(s3_read_max_wait_time_ms);
// write as inverted index tmp directory
DECLARE_String(tmp_file_dir);
diff --git a/be/src/io/file_factory.cpp b/be/src/io/file_factory.cpp
index e61ed144486..0c84c2eb74c 100644
--- a/be/src/io/file_factory.cpp
+++ b/be/src/io/file_factory.cpp
@@ -156,7 +156,7 @@ Result FileFactory::create_file_reader(
auto client_holder =
std::make_shared(s3_conf.client_conf);
RETURN_IF_ERROR_RESULT(client_holder->init());
return io::S3FileReader::create(std::move(client_holder),
s3_conf.bucket, s3_uri.get_key(),
-file_description.file_size)
+file_description.file_size, profile)
.and_then([&](auto&& reader) {
return io::create_cached_file_reader(std::move(reader),
reader_options);
});
diff --git a/be/src/io/fs/s3_file_reader.cpp b/be/src/io/fs/s3_file_reader.cpp
index e7775803198..a5c6ec09162 100644
--- a/be/src/io/fs/s3_file_reader.cpp
+++ b/be/src/io/fs/s3_file_reader.cpp
@@ -37,6 +37,7 @@
#include "io/fs/s3_common.h"
#include "util/bvar_helper.h"
#include "util/doris_metrics.h"
+#include "util/runtime_profile.h"
#include "util/s3_util.h"
namespace doris::io {
@@ -45,13 +46,18 @@ bvar::Adder
s3_file_reader_read_counter("s3_file_reader", "read_at");
bvar::Adder s3_file_reader_total("s3_file_reader", "total_num");
bvar::Adder s3_bytes_read_total("s3_file_reader", "bytes_read");
bvar::Adder s3_file_being_read("s3_file_reader", "file_being_read");
+bvar::Adder
s3_file_reader_too_many_request_counter("s3_file_reader", "too_many_request");
bvar::LatencyRecorder s3_bytes_per_read("s3_file_reader",
(doris) branch master updated (493dc26d743 -> cc484136ee2)
This is an automated email from the ASF dual-hosted git repository. lijibing pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 493dc26d743 [opt](s3) auto retry when meeting 429 error (#35396) add cc484136ee2 [fix](statistics) External table partition stats improve and add test case. (#36884) No new revisions were added by this update. Summary of changes: .../apache/doris/analysis/ShowColumnStatsStmt.java | 4 +- .../apache/doris/statistics/HMSAnalysisTask.java | 4 +- .../apache/doris/statistics/OlapAnalysisTask.java | 2 +- .../statistics/test_external_partition.groovy | 101 + .../suites/statistics/test_partition_stats.groovy | 8 +- 5 files changed, 109 insertions(+), 10 deletions(-) create mode 100644 regression-test/suites/statistics/test_external_partition.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [Fix](nereids) fix NormalizeAgg, change the upper project projections rewrite logic (#36623)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 65255cde1ed [Fix](nereids) fix NormalizeAgg, change the upper project
projections rewrite logic (#36623)
65255cde1ed is described below
commit 65255cde1edb9e0b7c25a627a9cbf334f0689c55
Author: feiniaofeiafei <53502832+feiniaofeia...@users.noreply.github.com>
AuthorDate: Thu Jun 27 16:11:21 2024 +0800
[Fix](nereids) fix NormalizeAgg, change the upper project projections
rewrite logic (#36623)
cherry-pick #36161 to branch-2.0
NormalizeAggregate rewrite logic has a bug, for sql like this:
SELECT
CASE
1 WHEN CAST( NULL AS SIGNED ) THEN NULL
WHEN COUNT( DISTINCT CAST( NULL AS SIGNED ) ) THEN NULL
ELSE null
END ;
This is the plan after NormalizeAggregate, the LogicalAggregate only
output `count(DISTINCT cast(NULL as SIGNED))`#3, do not output cast(NULL
as SIGNED)#2, but the upper project use cast(NULL as SIGNED)#2, so Doris
report error "cast(NULL as SIGNED) not in aggregate's output".
LogicalResultSink[29] ( outputExprs=[__case_when_0#1] )
+--LogicalProject[26] ( distinct=false, projects=[CASE WHEN (1 = cast(NULL as
SIGNED)#2) THEN NULL WHEN (1 = count(DISTINCT cast(NULL as SIGNED))#3) THEN
NULL ELSE NULL END AS `CASE WHEN (1 = cast(NULL as SIGNED)) THEN NULL WHEN (1 =
count(DISTINCT cast(NULL as SIGNED))) THEN NULL ELSE NULL END`#1], excepts=[] )
+--LogicalAggregate[25] ( groupByExpr=[], outputExpr=[count(DISTINCT
cast(NULL as SIGNED)#2) AS `count(DISTINCT cast(NULL as SIGNED))`#3],
hasRepeat=false )
+--LogicalProject[24] ( distinct=false, projects=[cast(NULL as
SIGNED) AS `cast(NULL as SIGNED)`#2], excepts=[] )
+--LogicalOneRowRelation ( projects=[0 AS `0`#0] )
The problem is that the cast(NULL as SIGNED)#2 should not outputted by
LogicalAggregate, cast(NULL as SIGNED) should be computed in
LogicalProject.
This pr change the upper project projections rewrite logic:
aggregateOutputs is rewritten and become the upper-level LogicalProject
projections. During the rewriting process, the expressions inside the
agg function can be rewritten with expressions in aggregate function
arguments and group by expressions, but the ones outside the agg
function can only be rewritten with group by expressions.
-
Co-authored-by: moailing
---
.../nereids/rules/analysis/NormalizeAggregate.java | 61 ++
.../nereids/rules/rewrite/NormalizeToSlot.java | 11
.../normalize_aggregate_test.out | 3 ++
.../normalize_aggregate_test.groovy| 27 ++
4 files changed, 91 insertions(+), 11 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/NormalizeAggregate.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/NormalizeAggregate.java
index dc071544935..9503a41a9de 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/NormalizeAggregate.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/NormalizeAggregate.java
@@ -193,12 +193,16 @@ public class NormalizeAggregate implements
RewriteRuleFactory, NormalizeToSlot {
// push down 3 kinds of exprs, these pushed exprs will be used to
normalize agg output later
// 1. group by exprs
-// 2. trivalAgg children
-// 3. trivalAgg input slots
-Set allPushDownExprs =
-Sets.union(groupingByExprs, Sets.union(needPushSelf,
needPushInputSlots));
-NormalizeToSlotContext bottomSlotContext =
-NormalizeToSlotContext.buildContext(existsAlias,
allPushDownExprs);
+// 2. trivialAgg children
+// 3. trivialAgg input slots
+// We need to distinguish between expressions in aggregate function
arguments and group by expressions.
+NormalizeToSlotContext groupByExprContext =
NormalizeToSlotContext.buildContext(existsAlias, groupingByExprs);
+Set existsAliasAndGroupByAlias = getExistsAlias(existsAlias,
groupByExprContext.getNormalizeToSlotMap());
+Set argsOfAggFuncNeedPushDown = Sets.union(needPushSelf,
needPushInputSlots);
+NormalizeToSlotContext argsOfAggFuncNeedPushDownContext =
NormalizeToSlotContext
+.buildContext(existsAliasAndGroupByAlias,
argsOfAggFuncNeedPushDown);
+NormalizeToSlotContext bottomSlotContext =
argsOfAggFuncNeedPushDownContext.mergeContext(groupByExprContext);
+
Set pushedGroupByExprs =
bottomSlotContext.pushDownToNamedExpression(groupingByExprs);
Set pushedTrivalAggChildren =
@@ -258,8 +262,12 @@ public class Normal
(doris) branch branch-2.0 updated: [Fix](Nereids) fix leading with brace can not generate correct plan (#36193) (#36549)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 9bc5f9a39ea [Fix](Nereids) fix leading with brace can not generate
correct plan (#36193) (#36549)
9bc5f9a39ea is described below
commit 9bc5f9a39eafffab47d08db072efd71dd64f6178
Author: LiBinfeng <46676950+libinfeng...@users.noreply.github.com>
AuthorDate: Thu Jun 27 16:19:07 2024 +0800
[Fix](Nereids) fix leading with brace can not generate correct plan
(#36193) (#36549)
cherry-pick from master #36193
Problem:
when using leading like:
leading(t1 {t2 t3} {t4 t5} t6)
it would not generate correct plan because levellist can not express
enough message of braces
Solved:
remove levellist express of leading levels and use reverse polish
expression
Algorithm:
leading(t1 {t2 t3} {t4 t5} t6)
==>
stack top to down(t1 t2 t3 join join t4 t5 join t6 join) when generate
leading join, we can pop items in stack, when it's a table, make
logicalscan when it's a join
operator, make logical join and push back to stack
---
.../org/apache/doris/nereids/hint/LeadingHint.java | 235 ++---
.../nereids/trees/plans/logical/LogicalJoin.java | 14 --
.../data/nereids_p0/hint/fix_leading.out | 8 +-
.../data/nereids_p0/hint/test_leading.out | 92
4 files changed, 160 insertions(+), 189 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/hint/LeadingHint.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/hint/LeadingHint.java
index 92ff5c8bfc2..e139fa4e09c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/hint/LeadingHint.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/hint/LeadingHint.java
@@ -50,8 +50,10 @@ import java.util.Stack;
*/
public class LeadingHint extends Hint {
private String originalString = "";
+
+private List addJoinParameters;
+private List normalizedParameters;
private final List tablelist = new ArrayList<>();
-private final List levelList = new ArrayList<>();
private final Map relationIdToScanMap =
Maps.newLinkedHashMap();
@@ -81,68 +83,77 @@ public class LeadingHint extends Hint {
public LeadingHint(String hintName, List parameters, String
originalString) {
super(hintName);
this.originalString = originalString;
-int level = 0;
-Stack brace = new Stack<>();
-String lastParameter = "";
-for (String parameter : parameters) {
-if (parameter.equals("{")) {
-if (lastParameter.equals("}")) {
-level += 2;
-brace.push(true);
-} else {
-++level;
-brace.push(false);
-}
-} else if (parameter.equals("}")) {
-if (brace.pop().equals(true)) {
-level -= 2;
-} else {
-level--;
-}
-} else {
-tablelist.add(parameter);
-levelList.add(level);
-}
-lastParameter = parameter;
-}
-normalizeLevelList();
+addJoinParameters = insertJoinIntoParameters(parameters);
+normalizedParameters =
parseIntoReversePolishNotation(addJoinParameters);
}
-private void removeGap(int left, int right, int gap) {
-for (int i = left; i <= right; i++) {
-levelList.set(i, levelList.get(i) - (gap - 1));
+/**
+ * insert join string into leading string
+ * @param list of sql input leading string
+ * @return list of string adding joins into tables
+ */
+public static List insertJoinIntoParameters(List list) {
+List output = new ArrayList<>();
+
+for (String item : list) {
+if (item.equals("shuffle") || item.equals("broadcast")) {
+output.remove(output.size() - 1);
+output.add(item);
+continue;
+} else if (item.equals("{")) {
+output.add(item);
+continue;
+} else if (item.equals("}")) {
+output.remove(output.size() - 1);
+output.add(item);
+} else {
+output.add(item);
+}
+output.add("join");
}
+output.remove(output.size() - 1);
+return output;
}
-// when we write leading like: leading(t1 {{t2 t3} {t4 t5}} t6)
-// levelList would like 0 2 2 3 3 0, it could be reduced to 0 1 1 2 2 0
like leading(t1 {t2 t3 {t4 t5}} t6)
-// gap is like 0 to 2 or 3 to 0 in upper example, and this function is to
remove gap when we use a lot of braces
-private void normalizeLevelList() {
-int leftIndex = 0;
Error while running notifications feature from refs/heads/master:.asf.yaml in doris-website!
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@foo.apache.org'. Must be a valid @doris.apache.org list! - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch master updated: [download](2.0.12) Update 2.0.12 release (#798)
This is an automated email from the ASF dual-hosted git repository. luzhijing pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new 092b0c05a0a [download](2.0.12) Update 2.0.12 release (#798) 092b0c05a0a is described below commit 092b0c05a0af0389c543473397a36b8a3a110637 Author: KassieZ <139741991+kass...@users.noreply.github.com> AuthorDate: Thu Jun 27 15:55:03 2024 +0700 [download](2.0.12) Update 2.0.12 release (#798) --- blog/release-note-2.0.1.md| 20 ++--- blog/release-note-2.0.10.md | 2 +- blog/release-note-2.0.11.md | 4 +-- blog/release-note-2.0.12.md | 62 ++ blog/release-note-2.0.3.md| 32 ++-- blog/release-note-2.0.4.md| 4 +-- blog/release-note-2.0.5.md| 6 ++-- blog/release-note-2.0.7.md| 6 ++-- blog/release-note-2.0.8.md| 8 ++--- blog/release-note-2.0.9.md| 4 +-- blog/release-note-2.1.1.md| 8 ++--- blog/release-note-2.1.2.md| 10 +++ blog/release-note-2.1.3.md| 8 ++--- blog/release-note-2.1.4.md| 36 +++--- src/constant/download.data.ts | 70 --- 15 files changed, 188 insertions(+), 92 deletions(-) diff --git a/blog/release-note-2.0.1.md b/blog/release-note-2.0.1.md index 040b654b725..f7d741cdf35 100644 --- a/blog/release-note-2.0.1.md +++ b/blog/release-note-2.0.1.md @@ -31,13 +31,13 @@ under the License. Thanks to our community users and developers, 383 improvements and bug fixes have been made in Doris 2.0.1. -## Behavior Changes +## Behavior changes - [https://github.com/apache/doris/pull/21302](https://github.com/apache/doris/pull/21302) ## Improvements -### functionality and stability of array and map datatypes +### Functionality and stability of array and map datatypes - [https://github.com/apache/doris/pull/22793](https://github.com/apache/doris/pull/22793) - [https://github.com/apache/doris/pull/22927](https://github.com/apache/doris/pull/22927) - https://github.com/apache/doris/pull/22738 @@ -45,13 +45,13 @@ Thanks to our community users and developers, 383 improvements and bug fixes hav - https://github.com/apache/doris/pull/23250 - https://github.com/apache/doris/pull/22300 -### performance for inverted index query +### Performance for inverted index query - https://github.com/apache/doris/pull/22836 - https://github.com/apache/doris/pull/23381 - https://github.com/apache/doris/pull/23389 - https://github.com/apache/doris/pull/22570 -### performance for bitmap, like, scan, agg functions +### Performance for bitmap, like, scan, agg functions - https://github.com/apache/doris/pull/23172 - https://github.com/apache/doris/pull/23495 - https://github.com/apache/doris/pull/23476 @@ -59,20 +59,20 @@ Thanks to our community users and developers, 383 improvements and bug fixes hav - https://github.com/apache/doris/pull/23182 - https://github.com/apache/doris/pull/22216 -### functionality and stability of CCR +### Functionality and stability of CCR - https://github.com/apache/doris/pull/22447 - https://github.com/apache/doris/pull/22559 - https://github.com/apache/doris/pull/22173 - https://github.com/apache/doris/pull/22678 -### merge on write unique table +### Merge on write unique table - https://github.com/apache/doris/pull/22282 - https://github.com/apache/doris/pull/22984 - https://github.com/apache/doris/pull/21933 - https://github.com/apache/doris/pull/22874 -### optimizer table stats and analyze +### Optimizer table stats and analyze - https://github.com/apache/doris/pull/22658 - https://github.com/apache/doris/pull/22211 @@ -82,7 +82,7 @@ Thanks to our community users and developers, 383 improvements and bug fixes hav - https://github.com/apache/doris/pull/22882 - -### functionality and performance of multi catalog +### Functionality and performance of multi catalog - https://github.com/apache/doris/pull/22949 - https://github.com/apache/doris/pull/22923 @@ -93,7 +93,7 @@ Thanks to our community users and developers, 383 improvements and bug fixes hav - https://github.com/apache/doris/pull/23279 -## Important Bug fixes +## Bug fixes - https://github.com/apache/doris/pull/22673 - https://github.com/apache/doris/pull/22656 @@ -150,7 +150,7 @@ Thanks to our community users and developers, 383 improvements and bug fixes hav See the complete list of improvements and bug fixes on [github](https://github.com/apache/doris/issues?q=label%3Adev%2F2.0.1-merged+is%3Aclosed) . -## Big Thanks +## Credits Thanks all who contribute to this release: diff --git a/blog/release-note-2.0.10.md b/blog/release-note-2.0.10.md index 8f3b98a47da..d763c6136b3 100644 --- a/blog/release-note-2.0.10.md +++ b/blog/release-note-2.0.10.md @@ -34,7 +34,7 @@ Thanks to our community users and developers, about 83
(doris) branch branch-2.0 updated: [fix](Nereids) processCharacterLiteral even if both side are literal (#36729) (#36925)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 8f781032103 [fix](Nereids) processCharacterLiteral even if both side
are literal (#36729) (#36925)
8f781032103 is described below
commit 8f781032103dda8739b608c8a30be6c058676b0d
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Thu Jun 27 17:01:51 2024 +0800
[fix](Nereids) processCharacterLiteral even if both side are literal
(#36729) (#36925)
pick from master #36729
---
.../doris/nereids/util/TypeCoercionUtils.java | 4
.../test_string_literal_vs_other_literal.out | 11 ++
.../test_string_literal_vs_other_literal.groovy| 24 ++
3 files changed, 35 insertions(+), 4 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java
index 24da439aa27..f9badd36be7 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java
@@ -320,10 +320,6 @@ public class TypeCoercionUtils {
if (!(left instanceof Literal) && !(right instanceof Literal)) {
return (T) op.withChildren(left, right);
}
-if (left instanceof Literal && right instanceof Literal) {
-// process by constant folding
-return (T) op.withChildren(left, right);
-}
if (left instanceof Literal && ((Literal) left).isStringLikeLiteral()
&& !right.getDataType().isStringLikeType()) {
left = TypeCoercionUtils.characterLiteralTypeCoercion(
diff --git
a/regression-test/data/nereids_p0/type_inference/test_string_literal_vs_other_literal.out
b/regression-test/data/nereids_p0/type_inference/test_string_literal_vs_other_literal.out
new file mode 100644
index 000..788187beeba
--- /dev/null
+++
b/regression-test/data/nereids_p0/type_inference/test_string_literal_vs_other_literal.out
@@ -0,0 +1,11 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !numeric_eq --
+1
+
+-- !numeric_non_eq --
+
+-- !date_eq --
+1
+
+-- !date_eq --
+
diff --git
a/regression-test/suites/nereids_p0/type_inference/test_string_literal_vs_other_literal.groovy
b/regression-test/suites/nereids_p0/type_inference/test_string_literal_vs_other_literal.groovy
new file mode 100644
index 000..43ad242de07
--- /dev/null
+++
b/regression-test/suites/nereids_p0/type_inference/test_string_literal_vs_other_literal.groovy
@@ -0,0 +1,24 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_string_literal_vs_other_literal") {
+qt_numeric_eq """select 1 where "12345678901234567890" =
12345678901234567890;"""
+qt_numeric_non_eq """select 1 where "12345678901234567890" =
12345678901234567891;"""
+
+qt_date_eq """select 1 where "2024-06-24 00:01:02.345678" =
cast("2024-06-24 00:01:02.345678" as datetime(6)) """
+qt_date_eq """select 1 where "2024-06-24 00:01:02.345679" =
cast("2024-06-24 00:01:02.345678" as datetime(6)) """
+}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (cc484136ee2 -> 7fc2931cdb6)
This is an automated email from the ASF dual-hosted git repository. englefly pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from cc484136ee2 [fix](statistics) External table partition stats improve and add test case. (#36884) add 7fc2931cdb6 [feat](Nereids) push down predicates with multi columns through LogicalWindow and LogicalPartitionTopN (#36828) No new revisions were added by this update. Summary of changes: .../PushDownFilterThroughPartitionTopN.java| 21 +-- .../rules/rewrite/PushDownFilterThroughWindow.java | 15 ++-- .../push_down_filter_through_window.out| 31 +++ .../push_down_filter_through_window.groovy | 44 ++ 4 files changed, 88 insertions(+), 23 deletions(-) create mode 100644 regression-test/data/nereids_rules_p0/push_down_filter_through_window/push_down_filter_through_window.out create mode 100644 regression-test/suites/nereids_rules_p0/push_down_filter_through_window/push_down_filter_through_window.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [fix](protocol) only return multi result when CLIENT_MULTI_STATEMENTS been set (#36759) (#36924)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 6ac990bdaf2 [fix](protocol) only return multi result when
CLIENT_MULTI_STATEMENTS been set (#36759) (#36924)
6ac990bdaf2 is described below
commit 6ac990bdaf23a7cd39253b807bde167ec3eed356
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Thu Jun 27 17:10:06 2024 +0800
[fix](protocol) only return multi result when CLIENT_MULTI_STATEMENTS been
set (#36759) (#36924)
pick from master #36759
multi statement support by PR #3050.
But there is a minor issue in implementation.
as MySQL dev doc say in
https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_command_phase_sp.html#sect_protocol_command_phase_sp_multi_statement
server should only process multi statement
when client set CLIENT_MULTI_STATEMENTS.
When client not set CLIENT_MULTI_STATEMENTS, server should treat query
as single statement.
but Doris do slightly different with MySQL server. Doris always treat
query as multi statement, but only return multi result when client set
CLIENT_MULTI_STATEMENTS. When client do not set CLIENT_MULTI_STATEMENTS,
Doris will return the last statement result only.
-
Co-authored-by: Mingyu Chen
---
.../src/main/java/org/apache/doris/mysql/MysqlCapability.java | 6 +-
fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java | 6 +-
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlCapability.java
b/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlCapability.java
index 0701b6ac2f8..b3d4793eeb2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlCapability.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlCapability.java
@@ -161,9 +161,13 @@ public class MysqlCapability {
return (flags & Flag.CLIENT_DEPRECATE_EOF.getFlagBit()) != 0;
}
+public boolean isClientMultiStatements() {
+return (flags & Flag.CLIENT_MULTI_STATEMENTS.getFlagBit()) != 0;
+}
+
@Override
public boolean equals(Object obj) {
-if (obj == null || !(obj instanceof MysqlCapability)) {
+if (!(obj instanceof MysqlCapability)) {
return false;
}
if (flags != ((MysqlCapability) obj).flags) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
index 7c280566150..e608b68684b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
@@ -430,7 +430,11 @@ public class ConnectProcessor {
executor.execute();
if (i != stmts.size() - 1) {
ctx.getState().serverStatus |=
MysqlServerStatusFlag.SERVER_MORE_RESULTS_EXISTS;
-if (ctx.getState().getStateType() != MysqlStateType.ERR) {
+// here, doris do different with mysql.
+// when client not request CLIENT_MULTI_STATEMENTS, mysql
treat all query as
+// single statement. Doris treat it with multi statement,
but only return
+// the last statement result.
+if (ctx.getCapability().isClientMultiStatements()) {
finalizeCommand();
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [chore](memo) record every group expression est. rows (#36879)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 83cd4f0bf3f [chore](memo) record every group expression est. rows
(#36879)
83cd4f0bf3f is described below
commit 83cd4f0bf3fd5e838dcd5f9d2604a802c6c0e461
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Thu Jun 27 17:10:36 2024 +0800
[chore](memo) record every group expression est. rows (#36879)
---
.../src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java
index bf78fe2a0bf..5dc38a9cc57 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java
@@ -243,7 +243,6 @@ public class StatsCalculator extends
DefaultPlanVisitor {
.noneMatch(e ->
newStats.isInputSlotsUnknown(e.getInputSlots()));
groupExpression.getOwnerGroup().setStatsReliable(isReliable);
groupExpression.getOwnerGroup().setStatistics(newStats);
-groupExpression.setEstOutputRowCount(newStats.getRowCount());
} else {
// the reason why we update col stats here.
// consider join between 3 tables: A/B/C with join condition:
A.id=B.id=C.id and a filter: C.id=1
@@ -258,6 +257,7 @@ public class StatsCalculator extends
DefaultPlanVisitor {
// now we update OwnerGroup().getStatistics().A.id.ndv to 1
groupExpression.getOwnerGroup().getStatistics().updateNdv(newStats);
}
+groupExpression.setEstOutputRowCount(newStats.getRowCount());
groupExpression.setStatDerived(true);
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (83cd4f0bf3f -> dd77c840b46)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 83cd4f0bf3f [chore](memo) record every group expression est. rows (#36879) add dd77c840b46 [improvement](sqlserver catalog) Configurable whether to use encrypt when connecting to SQL Server using the catalog (#36659) No new revisions were added by this update. Summary of changes: fe/fe-common/src/main/java/org/apache/doris/common/Config.java | 4 fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java | 3 +++ 2 files changed, 7 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](protocol) only return multi result when CLIENT_MULTI_STATEMENTS been set (#36759) (#36919)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new bfd634f9c79 [fix](protocol) only return multi result when
CLIENT_MULTI_STATEMENTS been set (#36759) (#36919)
bfd634f9c79 is described below
commit bfd634f9c79bd30511a90cb3021df4802046b96e
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Thu Jun 27 18:35:40 2024 +0800
[fix](protocol) only return multi result when CLIENT_MULTI_STATEMENTS been
set (#36759) (#36919)
pick from master #36759
multi statement support by PR #3050.
But there is a minor issue in implementation.
as MySQL dev doc say in
https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_command_phase_sp.html#sect_protocol_command_phase_sp_multi_statement
server should only process multi statement
when client set CLIENT_MULTI_STATEMENTS.
When client not set CLIENT_MULTI_STATEMENTS, server should treat query
as single statement.
but Doris do slightly different with MySQL server. Doris always treat
query as multi statement, but only return multi result when client set
CLIENT_MULTI_STATEMENTS. When client do not set CLIENT_MULTI_STATEMENTS,
Doris will return the last statement result only.
Co-authored-by: Mingyu Chen
---
.../src/main/java/org/apache/doris/mysql/MysqlCapability.java | 6 +-
.../src/main/java/org/apache/doris/qe/ConnectProcessor.java | 8 +++-
2 files changed, 12 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlCapability.java
b/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlCapability.java
index 0701b6ac2f8..b3d4793eeb2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlCapability.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlCapability.java
@@ -161,9 +161,13 @@ public class MysqlCapability {
return (flags & Flag.CLIENT_DEPRECATE_EOF.getFlagBit()) != 0;
}
+public boolean isClientMultiStatements() {
+return (flags & Flag.CLIENT_MULTI_STATEMENTS.getFlagBit()) != 0;
+}
+
@Override
public boolean equals(Object obj) {
-if (obj == null || !(obj instanceof MysqlCapability)) {
+if (!(obj instanceof MysqlCapability)) {
return false;
}
if (flags != ((MysqlCapability) obj).flags) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
index 51911c0..358e5d1a078 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
@@ -315,7 +315,13 @@ public abstract class ConnectProcessor {
if (i != stmts.size() - 1) {
ctx.getState().serverStatus |=
MysqlServerStatusFlag.SERVER_MORE_RESULTS_EXISTS;
if (ctx.getState().getStateType() !=
MysqlStateType.ERR) {
-finalizeCommand();
+// here, doris do different with mysql.
+// when client not request
CLIENT_MULTI_STATEMENTS, mysql treat all query as
+// single statement. Doris treat it with multi
statement, but only return
+// the last statement result.
+if
(getConnectContext().getCapability().isClientMultiStatements()) {
+finalizeCommand();
+}
}
}
} else if
(connectType.equals(ConnectType.ARROW_FLIGHT_SQL)) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](regression-case) add wait for partition_col_stats case (#36899)
This is an automated email from the ASF dual-hosted git repository.
hellostephen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new b13c17de9b4 [fix](regression-case) add wait for partition_col_stats
case (#36899)
b13c17de9b4 is described below
commit b13c17de9b4b44a3ff3fd0dc78272b251de93b22
Author: minghong
AuthorDate: Thu Jun 27 18:50:16 2024 +0800
[fix](regression-case) add wait for partition_col_stats case (#36899)
## Proposed changes
Issue Number: close #xxx
---
.../suites/nereids_p0/stats/partition_col_stats.groovy | 12 ++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/regression-test/suites/nereids_p0/stats/partition_col_stats.groovy
b/regression-test/suites/nereids_p0/stats/partition_col_stats.groovy
index 3436b6dd86e..9aea3251326 100644
--- a/regression-test/suites/nereids_p0/stats/partition_col_stats.groovy
+++ b/regression-test/suites/nereids_p0/stats/partition_col_stats.groovy
@@ -36,10 +36,18 @@ suite("partition_col_stats") {
"""
//run this sql to make stats be cached
sql "select * from pt where k1<3;"
-sleep(1)
+def pt_data = sql "show data from pt;"
+def retry= 0;
+while (pt_data[0][4] != '7' && retry < 20) {
+pt_data = sql "show data from pt;"
+sleep(1);
+retry ++;
+print("wait partition row count, retry " + retry +" times\n");
+}
+
explain{
sql "physical plan select * from pt where k1<3;"
contains("stats=4")
}
-
+sql "set global enable_partition_analyze=false;"
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (b13c17de9b4 -> eb4d2a90f59)
This is an automated email from the ASF dual-hosted git repository. hellostephen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from b13c17de9b4 [fix](regression-case) add wait for partition_col_stats case (#36899) add eb4d2a90f59 [fix](script) do not prepare lakesoul data when --stop (#36181) No new revisions were added by this update. Summary of changes: docker/thirdparties/run-thirdparties-docker.sh | 92 +- 1 file changed, 46 insertions(+), 46 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](Azure) Use robust logic to check there is no multipart upload for Azure (#36904)
This is an automated email from the ASF dual-hosted git repository.
gavinchou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 44915f3513f [fix](Azure) Use robust logic to check there is no
multipart upload for Azure (#36904)
44915f3513f is described below
commit 44915f3513fbf5df3597b7d8cacd006f823fa27a
Author: AlexYue
AuthorDate: Thu Jun 27 19:31:18 2024 +0800
[fix](Azure) Use robust logic to check there is no multipart upload for
Azure (#36904)
Azure has no logic for multi part load's upload id. We should use one
robust way to check it.
---
be/src/io/fs/s3_file_writer.cpp | 8 +++-
1 file changed, 3 insertions(+), 5 deletions(-)
diff --git a/be/src/io/fs/s3_file_writer.cpp b/be/src/io/fs/s3_file_writer.cpp
index 20c616ef90a..86ee36bab18 100644
--- a/be/src/io/fs/s3_file_writer.cpp
+++ b/be/src/io/fs/s3_file_writer.cpp
@@ -152,9 +152,7 @@ Status S3FileWriter::close(bool non_block) {
Status S3FileWriter::_close_impl() {
VLOG_DEBUG << "S3FileWriter::close, path: " <<
_obj_storage_path_opts.path.native();
-const auto& upload_id =
-_obj_storage_path_opts.upload_id.has_value() ?
*_obj_storage_path_opts.upload_id : "";
-if (upload_id.empty() && _pending_buf) {
+if (_cur_part_num == 1 && _pending_buf) {
RETURN_IF_ERROR(_set_upload_to_remote_less_than_buffer_size());
}
@@ -339,9 +337,9 @@ Status S3FileWriter::_complete() {
if (_failed || _completed_parts.size() != _cur_part_num) {
_st = Status::InternalError(
"error status {}, have failed {}, complete parts {}, cur
part num {}, whole "
-"parts {}, file path {}",
+"parts {}, file path {}, file size {}, has left buffer {}",
_st, _failed, _completed_parts.size(), _cur_part_num,
_dump_completed_part(),
-_obj_storage_path_opts.path.native());
+_obj_storage_path_opts.path.native(), _bytes_appended,
_pending_buf != nullptr);
LOG(WARNING) << _st;
return _st;
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (44915f3513f -> b2b1cb43b45)
This is an automated email from the ASF dual-hosted git repository.
yangbowen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 44915f3513f [fix](Azure) Use robust logic to check there is no
multipart upload for Azure (#36904)
add b2b1cb43b45 [optimize](restore) update replica version after restore
(#36148)
No new revisions were added by this update.
Summary of changes:
.../java/org/apache/doris/backup/RestoreJob.java | 4 +---
.../java/org/apache/doris/catalog/Replica.java | 9
...y => test_backup_restore_version_revert.groovy} | 26 ++
3 files changed, 27 insertions(+), 12 deletions(-)
copy regression-test/suites/backup_restore/{test_backup_restore.groovy =>
test_backup_restore_version_revert.groovy} (72%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](restore) avoid NPE for restore job (#36395)
This is an automated email from the ASF dual-hosted git repository.
yangbowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 29d31a278c4 [fix](restore) avoid NPE for restore job (#36395)
29d31a278c4 is described below
commit 29d31a278c4365bd1cd7ec627c2f5ac98de95666
Author: Xujian Duan <50550370+darvend...@users.noreply.github.com>
AuthorDate: Thu Jun 27 20:15:40 2024 +0800
[fix](restore) avoid NPE for restore job (#36395)
## Proposed changes
Issue Number: close #36440
The indexes of OlapTable may not be assigned in method `readFields`:
```
@Override
public void readFields(DataInput in) throws IOException {
...
// read indexes
if (in.readBoolean()) {
this.indexes = TableIndexes.read(in);
}
...
}
```
So indexes may be null.
---
.../src/main/java/org/apache/doris/catalog/OlapTable.java | 14 --
1 file changed, 8 insertions(+), 6 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
index 490e8a7b3a5..b0103e0d68b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
@@ -743,12 +743,14 @@ public class OlapTable extends Table implements
MTMVRelatedTableIf, GsonPostProc
}
// reset the indexes and update the indexes in materialized index meta
too.
-List indexes = this.indexes.getIndexes();
-for (Index idx : indexes) {
-idx.setIndexId(env.getNextId());
-}
-for (Map.Entry entry :
indexIdToMeta.entrySet()) {
-entry.getValue().setIndexes(indexes);
+if (this.indexes != null) {
+List indexes = this.indexes.getIndexes();
+for (Index idx : indexes) {
+idx.setIndexId(env.getNextId());
+}
+for (Map.Entry entry :
indexIdToMeta.entrySet()) {
+entry.getValue().setIndexes(indexes);
+}
}
return Status.OK;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [feature](tvf) support max_filter_ratio (#35431) (#36911)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 5c1eef5f060 [feature](tvf) support max_filter_ratio (#35431) (#36911)
5c1eef5f060 is described below
commit 5c1eef5f060409371aa1befed92f0c8af54e27bf
Author: Mingyu Chen
AuthorDate: Thu Jun 27 20:58:53 2024 +0800
[feature](tvf) support max_filter_ratio (#35431) (#36911)
bp #35431
Co-authored-by: 苏小刚
---
.../commands/insert/AbstractInsertExecutor.java| 10 -
.../apache/doris/planner/GroupCommitPlanner.java | 3 +-
.../java/org/apache/doris/qe/SessionVariable.java | 12 ++
.../java/org/apache/doris/qe/StmtExecutor.java | 12 +-
.../doris/statistics/util/StatisticsUtil.java | 1 +
.../data/external_table_p0/tvf/test_hdfs_tvf.out | 1 -
.../external_table_p0/tvf/test_hdfs_tvf.groovy | 43 --
7 files changed, 73 insertions(+), 9 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/AbstractInsertExecutor.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/AbstractInsertExecutor.java
index 64bf625fef5..cb9073c77ec 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/AbstractInsertExecutor.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/AbstractInsertExecutor.java
@@ -162,13 +162,19 @@ public abstract class AbstractInsertExecutor {
}
}
-private void checkStrictMode() throws Exception {
+private void checkStrictModeAndFilterRatio() throws Exception {
// if in strict mode, insert will fail if there are filtered rows
if (ctx.getSessionVariable().getEnableInsertStrict()) {
if (filteredRows > 0) {
ErrorReport.reportDdlException("Insert has filtered data in
strict mode",
ErrorCode.ERR_FAILED_WHEN_INSERT);
}
+} else {
+if (filteredRows >
ctx.getSessionVariable().getInsertMaxFilterRatio() * (filteredRows +
loadedRows)) {
+ErrorReport.reportDdlException("Insert has too many filtered
data %d/%d insert_max_filter_ratio is %f",
+ErrorCode.ERR_FAILED_WHEN_INSERT, filteredRows,
filteredRows + loadedRows,
+ctx.getSessionVariable().getInsertMaxFilterRatio());
+}
}
}
@@ -179,7 +185,7 @@ public abstract class AbstractInsertExecutor {
beforeExec();
try {
execImpl(executor, jobId);
-checkStrictMode();
+checkStrictModeAndFilterRatio();
onComplete();
} catch (Throwable t) {
onFail(t);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/GroupCommitPlanner.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/GroupCommitPlanner.java
index 9ff06b29076..253d12ad864 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/GroupCommitPlanner.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/GroupCommitPlanner.java
@@ -98,7 +98,8 @@ public class GroupCommitPlanner {
}
streamLoadPutRequest
.setDb(db.getFullName())
-
.setMaxFilterRatio(ConnectContext.get().getSessionVariable().enableInsertStrict
? 0 : 1)
+
.setMaxFilterRatio(ConnectContext.get().getSessionVariable().enableInsertStrict
? 0
+:
ConnectContext.get().getSessionVariable().insertMaxFilterRatio)
.setTbl(table.getName())
.setFileType(TFileType.FILE_STREAM).setFormatType(TFileFormatType.FORMAT_CSV_PLAIN)
.setMergeType(TMergeType.APPEND).setThriftRpcTimeoutMs(5000).setLoadId(queryId)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
index 534b4f3386a..7c2257f6719 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
@@ -128,6 +128,7 @@ public class SessionVariable implements Serializable,
Writable {
public static final String PROFILE_LEVEL = "profile_level";
public static final String MAX_INSTANCE_NUM = "max_instance_num";
public static final String ENABLE_INSERT_STRICT = "enable_insert_strict";
+public static final String INSERT_MAX_FILTER_RATIO =
"insert_max_filter_ratio";
public static final String ENABLE_SPILLING = "enable_spilling";
public static final String ENABLE_SHORT_CIRCUIT_QUERY =
"enable_short_circuit_point_query";
public static final String ENABLE_EXCHANGE_NODE_PARALLEL_MERGE =
"enable_exchange_node_parallel_merge";
@@ -848,6 +849,9 @@ public class Sess
(doris) branch branch-2.1 updated: [test](migrate) move some cases from p2 to p0 (#36750)(#36787) (#36922)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-2.1 by this push: new fcc26cc6713 [test](migrate) move some cases from p2 to p0 (#36750)(#36787) (#36922) fcc26cc6713 is described below commit fcc26cc6713a1a0a1eb4c8533d7c7f7038dc4e77 Author: Mingyu Chen AuthorDate: Thu Jun 27 20:59:50 2024 +0800 [test](migrate) move some cases from p2 to p0 (#36750)(#36787) (#36922) bp #36750 and #36787 --- .licenserc.yaml| 1 + .../data/multi_catalog/datev2_csv/create_table.hql | 20 + .../data/multi_catalog/datev2_csv/data.tar.gz | Bin 0 -> 184 bytes .../scripts/data/multi_catalog/datev2_csv/run.sh | 13 + .../data/multi_catalog/datev2_orc/create_table.hql | 20 + .../data/multi_catalog/datev2_orc/data.tar.gz | Bin 0 -> 459 bytes .../scripts/data/multi_catalog/datev2_orc/run.sh | 13 + .../multi_catalog/datev2_parquet/create_table.hql | 20 + .../data/multi_catalog/datev2_parquet/data.tar.gz | Bin 0 -> 384 bytes .../data/multi_catalog/datev2_parquet/run.sh | 13 + .../hive_upper_case_orc/create_table.hql | 21 + .../multi_catalog/hive_upper_case_orc/data.tar.gz | Bin 0 -> 521 bytes .../data/multi_catalog/hive_upper_case_orc/run.sh | 13 + .../hive_upper_case_parquet/create_table.hql | 21 + .../hive_upper_case_parquet/data.tar.gz| Bin 0 -> 690 bytes .../multi_catalog/hive_upper_case_parquet/run.sh | 13 + .../parquet_lz4_compression/create_table.hql | 31 ++ .../parquet_lz4_compression/data.tar.gz| Bin 0 -> 11375 bytes .../multi_catalog/parquet_lz4_compression/run.sh | 13 + .../parquet_lzo_compression/create_table.hql | 31 ++ .../parquet_lzo_compression/data.tar.gz| Bin 0 -> 28433 bytes .../multi_catalog/parquet_lzo_compression/run.sh | 13 + .../test_compress_partitioned/create_table.hql | 152 ++ .../multi_catalog/test_compress_partitioned/run.sh | 22 + .../data/regression/crdmm_data/create_table.hql| 162 ++ .../scripts/data/regression/crdmm_data/data.tar.gz | Bin 0 -> 2035 bytes .../hive/scripts/data/regression/crdmm_data/run.sh | 14 + .../docker-compose/hive/scripts/hive-metastore.sh | 23 +- docker/thirdparties/run-thirdparties-docker.sh | 24 + .../hive/test_external_credit_data.out | 12 + .../hive/test_hive_compress_type.out} | 572 + .../hive/test_hive_to_date.out | 24 + .../hive/test_upper_case_column_name.out | 89 .../hive/hive_tpch_sf1_orc.groovy | 4 +- .../hive/hive_tpch_sf1_parquet.groovy | 4 +- .../hive/test_external_credit_data.groovy | 29 +- .../hive/test_hive_compress_type.groovy} | 39 +- .../hive/test_hive_to_date.groovy | 37 +- .../hive/test_upper_case_column_name.groovy| 103 39 files changed, 1497 insertions(+), 69 deletions(-) diff --git a/.licenserc.yaml b/.licenserc.yaml index 3f41200df2e..7412436a9c1 100644 --- a/.licenserc.yaml +++ b/.licenserc.yaml @@ -82,6 +82,7 @@ header: - "docker/thirdparties/docker-compose/hive/scripts/create_tpch1_orc.hql" - "docker/thirdparties/docker-compose/hive/scripts/create_tpch1_parquet.hql" - "docker/thirdparties/docker-compose/hive/scripts/preinstalled_data/" +- "docker/thirdparties/docker-compose/hive/scripts/data/**" - "docker/thirdparties/docker-compose/iceberg/spark-defaults.conf.tpl" - "conf/mysql_ssl_default_certificate/*" - "conf/mysql_ssl_default_certificate/client_certificate/ca.pem" diff --git a/docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_csv/create_table.hql b/docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_csv/create_table.hql new file mode 100644 index 000..2fb4b46dec4 --- /dev/null +++ b/docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_csv/create_table.hql @@ -0,0 +1,20 @@ +create database if not exists multi_catalog; + +use multi_catalog; + +CREATE external TABLE `datev2_csv`( + `id` int, + `day` date) +ROW FORMAT SERDE + 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' +STORED AS INPUTFORMAT + 'org.apache.hadoop.mapred.TextInputFormat' +OUTPUTFORMAT + 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' +LOCATION + '/user/doris/suites/multi_catalog/datev2_csv' +TBLPROPERTIES ( + 'transient_lastDdlTime'='1688118691'); + +msck repair table datev2_csv; + diff --git a/docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_csv/data.tar.gz b/docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_csv/data.tar.gz new file mode 100644 index 000..2c0bff2d8d1 Binary files /dev/null and b/docker/thirdparti
(doris) branch branch-2.1 updated (fcc26cc6713 -> 46eef9d948b)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git from fcc26cc6713 [test](migrate) move some cases from p2 to p0 (#36750)(#36787) (#36922) add 46eef9d948b [build](docker) add repo for new version of git (#35892) (#36909) No new revisions were added by this update. Summary of changes: docker/compilation/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (29d31a278c4 -> 7de03df521a)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 29d31a278c4 [fix](restore) avoid NPE for restore job (#36395) add 7de03df521a [feature](Azure) Implement azure table value function (#36917) No new revisions were added by this update. Summary of changes: .../property/constants/AzureProperties.java| 2 ++ .../doris/tablefunction/S3TableValuedFunction.java | 36 +- 2 files changed, 30 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](fe) CloudPartition returns cached version in the checkpoint thread (#36933)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d921f9ba408 [fix](fe) CloudPartition returns cached version in the
checkpoint thread (#36933)
d921f9ba408 is described below
commit d921f9ba408a5dc23bd4a06071ddf25a9e70fe8c
Author: walter
AuthorDate: Thu Jun 27 21:16:29 2024 +0800
[fix](fe) CloudPartition returns cached version in the checkpoint thread
(#36933)
---
.../main/java/org/apache/doris/cloud/catalog/CloudPartition.java| 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudPartition.java
b/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudPartition.java
index 9f75a33cfde..42327b6039d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudPartition.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudPartition.java
@@ -18,6 +18,7 @@
package org.apache.doris.cloud.catalog;
import org.apache.doris.catalog.DistributionInfo;
+import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.MaterializedIndex;
import org.apache.doris.catalog.Partition;
import org.apache.doris.cloud.proto.Cloud;
@@ -104,9 +105,8 @@ public class CloudPartition extends Partition {
@Override
public long getVisibleVersion() {
-if (Config.enable_check_compatibility_mode) {
-LOG.info("return 1 as the visible version in the compatibility
checking mode");
-return 1;
+if (Env.isCheckpointThread() ||
Config.enable_check_compatibility_mode) {
+return super.getVisibleVersion();
}
if (LOG.isDebugEnabled()) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [chore](restore) Log partition visible version (#36920)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 8fde0615789 [chore](restore) Log partition visible version (#36920)
8fde0615789 is described below
commit 8fde0615789e119b6a71d3bfe6b943f0bb7c5706
Author: walter
AuthorDate: Thu Jun 27 21:20:33 2024 +0800
[chore](restore) Log partition visible version (#36920)
---
fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java | 7 +--
.../java/org/apache/doris/transaction/DatabaseTransactionMgr.java | 2 +-
2 files changed, 6 insertions(+), 3 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java
b/fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java
index dec821424bf..098c473d32e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java
@@ -1148,6 +1148,7 @@ public class RestoreJob extends AbstractJob implements
GsonPostProcessable {
// generate new partition id
long newPartId = env.getNextId();
+long oldPartId = remotePart.getId();
remotePart.setIdForRestore(newPartId);
// indexes
@@ -1167,9 +1168,11 @@ public class RestoreJob extends AbstractJob implements
GsonPostProcessable {
// save version info for creating replicas
long visibleVersion = remotePart.getVisibleVersion();
+LOG.info("reset partition {} for restore, visible version: {}, old
partition id: {}",
+newPartId, visibleVersion, oldPartId);
// tablets
-Map nextIndexs = Maps.newHashMap();
+Map nextIndexes = Maps.newHashMap();
for (MaterializedIndex remoteIdx :
remotePart.getMaterializedIndices(IndexExtState.VISIBLE)) {
int schemaHash =
remoteTbl.getSchemaHashByIndexId(remoteIdx.getId());
int remotetabletSize = remoteIdx.getTablets().size();
@@ -1184,7 +1187,7 @@ public class RestoreJob extends AbstractJob implements
GsonPostProcessable {
// replicas
try {
Pair>, TStorageMedium> beIdsAndMedium
= Env.getCurrentSystemInfo()
-.selectBackendIdsForReplicaCreation(replicaAlloc,
nextIndexs, null, false, false);
+.selectBackendIdsForReplicaCreation(replicaAlloc,
nextIndexes, null, false, false);
Map> beIds = beIdsAndMedium.first;
for (Map.Entry> entry : beIds.entrySet()) {
for (Long beId : entry.getValue()) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
b/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
index 3850400c1b7..92249f8defc 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
@@ -1598,7 +1598,7 @@ public class DatabaseTransactionMgr {
protected void unprotectedCommitTransaction2PC(TransactionState
transactionState, Database db) {
// transaction state is modified during check if the transaction could
committed
if (transactionState.getTransactionStatus() !=
TransactionStatus.PRECOMMITTED) {
-LOG.warn("Unknow exception. state of transaction [{}] changed,
failed to commit transaction",
+LOG.warn("Unknown exception. state of transaction [{}] changed,
failed to commit transaction",
transactionState.getTransactionId());
return;
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](group commit) Group commit http stream should not begin txn (#36926)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 20d75efd26b [fix](group commit) Group commit http stream should not
begin txn (#36926)
20d75efd26b is described below
commit 20d75efd26b2d4d51808e59d48475b5361b33bb9
Author: meiyi
AuthorDate: Thu Jun 27 21:21:07 2024 +0800
[fix](group commit) Group commit http stream should not begin txn (#36926)
## Proposed changes
After nereids support group_commit, the http_stream in group commit will
also do some check like insert into values, and change the group_commit
flag to false.
Then will begin a txn. This is hang schema change.
---
.../trees/plans/commands/insert/GroupCommitInsertExecutor.java| 4
1 file changed, 4 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/GroupCommitInsertExecutor.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/GroupCommitInsertExecutor.java
index f4d640e2c64..08686561a41 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/GroupCommitInsertExecutor.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/GroupCommitInsertExecutor.java
@@ -81,6 +81,10 @@ public class GroupCommitInsertExecutor extends
AbstractInsertExecutor {
*/
public static boolean canGroupCommit(ConnectContext ctx, DataSink sink,
PhysicalSink physicalSink,
NereidsPlanner planner) {
+// The flag is set to false before execute sql, if it is true, this is
a http stream
+if (ctx.isGroupCommit()) {
+return false;
+}
PhysicalOlapTableSink olapSink = (PhysicalOlapTableSink)
physicalSink;
boolean can = analyzeGroupCommit(ctx, sink, olapSink, planner);
ctx.setGroupCommit(can);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [enhancement](compaction) optimizing memory usage for compaction (#36492)
This is an automated email from the ASF dual-hosted git repository.
gavinchou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 99901814d8b [enhancement](compaction) optimizing memory usage for
compaction (#36492)
99901814d8b is described below
commit 99901814d8b90887f54b1768b98b4f0b78fab376
Author: Luwei <814383...@qq.com>
AuthorDate: Thu Jun 27 21:41:20 2024 +0800
[enhancement](compaction) optimizing memory usage for compaction (#36492)
1 Limit the number of rowsets participating in base compaction.
2 Estimate the appropriate batch size by sampling and analyzing the
memory consumption of blocks.
---
be/src/cloud/cloud_base_compaction.cpp | 10 ++
be/src/cloud/cloud_cumulative_compaction.cpp | 13 +-
be/src/common/config.cpp | 6 +
be/src/common/config.h | 6 +
be/src/olap/base_compaction.cpp| 10 ++
be/src/olap/base_tablet.h | 5 +
be/src/olap/compaction.cpp | 15 ++-
be/src/olap/compaction.h | 2 +
be/src/olap/cumulative_compaction.cpp | 15 ++-
be/src/olap/iterators.h| 15 ++-
be/src/olap/merger.cpp | 67 +-
be/src/olap/merger.h | 6 +-
be/src/olap/rowset/rowset_meta.h | 15 +++
be/src/olap/rowset/segcompaction.cpp | 2 +-
be/src/olap/tablet_reader.h| 2 +
be/src/vec/olap/vertical_block_reader.cpp | 18 ++-
be/src/vec/olap/vertical_block_reader.h| 3 +-
be/src/vec/olap/vertical_merge_iterator.cpp| 29 +++--
be/src/vec/olap/vertical_merge_iterator.h | 25 +++-
be/test/olap/base_compaction_test.cpp | 84 +
be/test/olap/rowid_conversion_test.cpp | 6 +-
be/test/vec/olap/vertical_compaction_test.cpp | 14 ++-
.../compaction_width_array_column.groovy | 137 +
23 files changed, 463 insertions(+), 42 deletions(-)
diff --git a/be/src/cloud/cloud_base_compaction.cpp
b/be/src/cloud/cloud_base_compaction.cpp
index d4a86743a48..4ceab8eb6e3 100644
--- a/be/src/cloud/cloud_base_compaction.cpp
+++ b/be/src/cloud/cloud_base_compaction.cpp
@@ -163,6 +163,16 @@ Status CloudBaseCompaction::pick_rowsets_to_compact() {
return Status::Error("no suitable versions for
compaction");
}
+int score = 0;
+int rowset_cnt = 0;
+while (rowset_cnt < _input_rowsets.size()) {
+score +=
_input_rowsets[rowset_cnt++]->rowset_meta()->get_compaction_score();
+if (score > config::base_compaction_max_compaction_score) {
+break;
+}
+}
+_input_rowsets.resize(rowset_cnt);
+
// 1. cumulative rowset must reach base_compaction_min_rowset_num threshold
if (_input_rowsets.size() > config::base_compaction_min_rowset_num) {
VLOG_NOTICE << "satisfy the base compaction policy. tablet=" <<
_tablet->tablet_id()
diff --git a/be/src/cloud/cloud_cumulative_compaction.cpp
b/be/src/cloud/cloud_cumulative_compaction.cpp
index de318f979a5..2a26b1b294b 100644
--- a/be/src/cloud/cloud_cumulative_compaction.cpp
+++ b/be/src/cloud/cloud_cumulative_compaction.cpp
@@ -354,11 +354,20 @@ Status
CloudCumulativeCompaction::pick_rowsets_to_compact() {
return st;
}
+int64_t max_score = config::cumulative_compaction_max_deltas;
+auto process_memory_usage =
doris::GlobalMemoryArbitrator::process_memory_usage();
+bool memory_usage_high = process_memory_usage > MemInfo::soft_mem_limit()
* 0.8;
+if
(cloud_tablet()->last_compaction_status.is() ||
+memory_usage_high) {
+max_score = std::max(config::cumulative_compaction_max_deltas /
+
config::cumulative_compaction_max_deltas_factor,
+ config::cumulative_compaction_min_deltas + 1);
+}
+
size_t compaction_score = 0;
auto compaction_policy =
cloud_tablet()->tablet_meta()->compaction_policy();
_engine.cumu_compaction_policy(compaction_policy)
-->pick_input_rowsets(cloud_tablet(), candidate_rowsets,
- config::cumulative_compaction_max_deltas,
+->pick_input_rowsets(cloud_tablet(), candidate_rowsets, max_score,
config::cumulative_compaction_min_deltas,
&_input_rowsets,
&_last_delete_version, &compaction_score);
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index a08bb43db56..cd354ab28ed 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -385,6 +385,7 @@ DEFINE_mInt32(max_single_replica_compaction_threads, "-1");
DEFINE_Bool(enable_bas
(doris) branch branch-2.1 updated: [fix](oom) avoid oom when a lot of tablets fail on load (#36944)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new ee7f9a4f268 [fix](oom) avoid oom when a lot of tablets fail on load
(#36944)
ee7f9a4f268 is described below
commit ee7f9a4f2688c1800a8ed9b7e510cc5530b39274
Author: Yongqiang YANG <98214048+dataroar...@users.noreply.github.com>
AuthorDate: Thu Jun 27 22:12:42 2024 +0800
[fix](oom) avoid oom when a lot of tablets fail on load (#36944)
pick #36873
---
.../doris/transaction/DatabaseTransactionMgr.java | 62 +-
1 file changed, 49 insertions(+), 13 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
b/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
index 3996664708a..368415c3abd 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
@@ -1159,6 +1159,45 @@ public class DatabaseTransactionMgr {
return true;
}
+private class TabletsPublishResultLogs {
+public List quorumSuccLogs = Lists.newArrayList();
+public List timeoutSuccLogs = Lists.newArrayList();
+public List failedLogs = Lists.newArrayList();
+
+public void addQuorumSuccLog(String log) {
+if (quorumSuccLogs.size() < 16) {
+quorumSuccLogs.add(log);
+}
+}
+
+public void addTimeoutSuccLog(String log) {
+if (timeoutSuccLogs.size() < 16) {
+timeoutSuccLogs.add(log);
+}
+}
+
+public void addFailedLog(String log) {
+if (failedLogs.size() < 16) {
+failedLogs.add(log);
+}
+}
+
+public void log() {
+// log failed logs
+for (String log : failedLogs) {
+LOG.info(log);
+}
+// log timeout succ logs
+for (String log : timeoutSuccLogs) {
+LOG.info(log);
+}
+// log quorum succ logs
+for (String log : quorumSuccLogs) {
+LOG.info(log);
+}
+}
+}
+
private PublishResult finishCheckQuorumReplicas(TransactionState
transactionState,
List> relatedTblPartitions,
Set errorReplicaIds) {
@@ -1173,7 +1212,7 @@ public class DatabaseTransactionMgr {
List tabletSuccReplicas = Lists.newArrayList();
List tabletWriteFailedReplicas = Lists.newArrayList();
List tabletVersionFailedReplicas = Lists.newArrayList();
-List logs = Lists.newArrayList();
+TabletsPublishResultLogs logs = new TabletsPublishResultLogs();
Map publishTasks =
transactionState.getPublishVersionTasks();
PublishResult publishResult = PublishResult.QUORUM_SUCC;
@@ -1224,9 +1263,9 @@ public class DatabaseTransactionMgr {
if (hasFailedReplica) {
String writeDetail =
getTabletWriteDetail(tabletSuccReplicas,
tabletWriteFailedReplicas,
tabletVersionFailedReplicas);
-logs.add(String.format("publish version quorum
succ for transaction %s on tablet %s"
-+ " with version %s, and has failed
replicas, load require replica num %s. "
-+ "table %s, partition: [ id=%s, commit
version=%s ], tablet detail: %s",
+logs.addQuorumSuccLog(String.format("publish
version quorum succ for transaction %s "
++ "on tablet %s with version %s, and has
failed replicas, load require replica "
++ "num %s. table %s, partition: [ id=%s,
commit version=%s ], tablet detail: %s",
transactionState, tablet.getId(),
newVersion, loadRequiredReplicaNum, tableId,
partitionId,
partition.getCommittedVersion(), writeDetail));
}
@@ -1248,9 +1287,9 @@ public class DatabaseTransactionMgr {
// that are being publised exists on a few replicas we
should go
// ahead, otherwise data may be lost and thre
// publish task hangs forever.
-logs.add(String.format("publish version timeout succ
for transaction %s on tablet %s "
-+ "with version %s, and has failed replicas,
load require replica num %s. "
-+ "table %s, partition %s, tablet detail: %s",
+logs.addTimeoutSuccLog(String.format("publish version
timeout succ for
(doris) branch branch-2.0 updated: [chore](autobucket) add autobucket test and log #36874 (#36910)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new b4a19dd9adb [chore](autobucket) add autobucket test and log #36874
(#36910)
b4a19dd9adb is described below
commit b4a19dd9adbfb713109f3529c0b5a7cdee1c19fb
Author: yujun
AuthorDate: Thu Jun 27 22:29:50 2024 +0800
[chore](autobucket) add autobucket test and log #36874 (#36910)
cherry pick from #36874
---
.../doris/clone/DynamicPartitionScheduler.java | 48 +++---
.../doris/catalog/DynamicPartitionTableTest.java | 39 ++
2 files changed, 72 insertions(+), 15 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
b/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
index 770e1298808..506629de19d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
@@ -70,6 +70,7 @@ import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Set;
+import java.util.stream.Collectors;
/**
* This class is used to periodically add or drop partition on an olapTable
which specify dynamic partition properties
@@ -185,34 +186,49 @@ public class DynamicPartitionScheduler extends
MasterDaemon {
}
private static int getBucketsNum(DynamicPartitionProperty property,
OlapTable table,
-String nowPartitionName, boolean executeFirstTime) {
+String partitionName, String nowPartitionName, boolean
executeFirstTime) {
+// if execute first time, all partitions no contain data
if (!table.isAutoBucket() || executeFirstTime) {
return property.getBuckets();
}
// auto bucket
// get all history partitions
-ArrayList partitionSizeArray = Lists.newArrayList();
RangePartitionInfo info = (RangePartitionInfo)
(table.getPartitionInfo());
List> idToItems = new
ArrayList<>(info.getIdToItem(false).entrySet());
idToItems.sort(Comparator.comparing(o -> ((RangePartitionItem)
o.getValue()).getItems().upperEndpoint()));
-for (Map.Entry idToItem : idToItems) {
-Partition partition = table.getPartition(idToItem.getKey());
-// exclude current partition because its data isn't enough one
week/day/hour.
-if (partition != null &&
!partition.getName().equals(nowPartitionName)
-&& partition.getVisibleVersion() >= 2) {
-partitionSizeArray.add(partition.getAllDataSize(true));
-}
-}
+List partitions = idToItems.stream()
+.map(entry -> table.getPartition(entry.getKey()))
+.filter(partition -> partition != null &&
!partition.getName().equals(nowPartitionName)
+&& partition.getVisibleVersion() >= 2)
+.collect(Collectors.toList());
// no exist history partition data
-if (partitionSizeArray.isEmpty()) {
+if (partitions.isEmpty()) {
+LOG.info("autobucket use property's buckets due to all partitions
no data, table: [{}-{}], "
++ "partition: {}, buckets num: {}",
+table.getName(), table.getId(), partitionName,
property.getBuckets());
return property.getBuckets();
}
+ArrayList partitionSizeArray = partitions.stream()
+.map(partition -> partition.getAllDataSize(true))
+.collect(Collectors.toCollection(ArrayList::new));
+long estimatePartitionSize = getNextPartitionSize(partitionSizeArray);
// plus 5 for uncompressed data
-long uncompressedPartitionSize =
getNextPartitionSize(partitionSizeArray) * 5;
-return AutoBucketUtils.getBucketsNum(uncompressedPartitionSize,
Config.autobucket_min_buckets);
+long uncompressedPartitionSize = estimatePartitionSize * 5;
+int bucketsNum =
AutoBucketUtils.getBucketsNum(uncompressedPartitionSize,
Config.autobucket_min_buckets);
+LOG.info("autobucket calc with {} history partitions, table: [{}-{}],
partition: {}, buckets num: {}, "
++ " estimate partition size: {}, last partitions(partition
name, local size, remote size): {}",
+partitions.size(), table.getName(), table.getId(),
partitionName, bucketsNum,
+estimatePartitionSize,
+partitions.stream()
+.skip(Math.max(0, partitions.size() - 7))
+.map(partition -> "(" + partition.getName() + ", " +
partition.getDataSize(true)
++ ", " + partition.getRemoteDataSize() + ")")
+.col
(doris) branch branch-2.1 updated: [chore](autobucket) add autobucket test and log #36874 (#36907)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 474295cf315 [chore](autobucket) add autobucket test and log #36874
(#36907)
474295cf315 is described below
commit 474295cf31513775fc08bde748b07d2f8cef691a
Author: yujun
AuthorDate: Thu Jun 27 22:30:13 2024 +0800
[chore](autobucket) add autobucket test and log #36874 (#36907)
cherry pick from #36874
---
.../doris/clone/DynamicPartitionScheduler.java | 47 +++---
.../doris/catalog/DynamicPartitionTableTest.java | 38 +
2 files changed, 70 insertions(+), 15 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
b/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
index 8c5f4f669c5..38d68c320ec 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
@@ -71,6 +71,7 @@ import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Set;
+import java.util.stream.Collectors;
/**
* This class is used to periodically add or drop partition on an olapTable
which specify dynamic partition properties
@@ -186,7 +187,7 @@ public class DynamicPartitionScheduler extends MasterDaemon
{
}
private static int getBucketsNum(DynamicPartitionProperty property,
OlapTable table,
-String nowPartitionName, boolean executeFirstTime) {
+String partitionName, String nowPartitionName, boolean
executeFirstTime) {
// if execute first time, all partitions no contain data
if (!table.isAutoBucket() || executeFirstTime) {
return property.getBuckets();
@@ -194,27 +195,41 @@ public class DynamicPartitionScheduler extends
MasterDaemon {
// auto bucket
// get all history partitions
-ArrayList partitionSizeArray = Lists.newArrayList();
RangePartitionInfo info = (RangePartitionInfo)
(table.getPartitionInfo());
List> idToItems = new
ArrayList<>(info.getIdToItem(false).entrySet());
idToItems.sort(Comparator.comparing(o -> ((RangePartitionItem)
o.getValue()).getItems().upperEndpoint()));
-for (Map.Entry idToItem : idToItems) {
-Partition partition = table.getPartition(idToItem.getKey());
-// exclude current partition because its data isn't enough one
week/day/hour.
-if (partition != null &&
!partition.getName().equals(nowPartitionName)
-&& partition.getVisibleVersion() >= 2) {
-partitionSizeArray.add(partition.getAllDataSize(true));
-}
-}
+List partitions = idToItems.stream()
+.map(entry -> table.getPartition(entry.getKey()))
+.filter(partition -> partition != null &&
!partition.getName().equals(nowPartitionName)
+&& partition.getVisibleVersion() >= 2)
+.collect(Collectors.toList());
// no exist history partition data
-if (partitionSizeArray.isEmpty()) {
+if (partitions.isEmpty()) {
+LOG.info("autobucket use property's buckets due to all partitions
no data, table: [{}-{}], "
++ "partition: {}, buckets num: {}",
+table.getName(), table.getId(), partitionName,
property.getBuckets());
return property.getBuckets();
}
+ArrayList partitionSizeArray = partitions.stream()
+.map(partition -> partition.getAllDataSize(true))
+.collect(Collectors.toCollection(ArrayList::new));
+long estimatePartitionSize = getNextPartitionSize(partitionSizeArray);
// plus 5 for uncompressed data
-long uncompressedPartitionSize =
getNextPartitionSize(partitionSizeArray) * 5;
-return AutoBucketUtils.getBucketsNum(uncompressedPartitionSize,
Config.autobucket_min_buckets);
+long uncompressedPartitionSize = estimatePartitionSize * 5;
+int bucketsNum =
AutoBucketUtils.getBucketsNum(uncompressedPartitionSize,
Config.autobucket_min_buckets);
+LOG.info("autobucket calc with {} history partitions, table: [{}-{}],
partition: {}, buckets num: {}, "
++ " estimate partition size: {}, last partitions(partition
name, local size, remote size): {}",
+partitions.size(), table.getName(), table.getId(),
partitionName, bucketsNum,
+estimatePartitionSize,
+partitions.stream()
+.skip(Math.max(0, partitions.size() - 7))
+.map(partition -> "(" + partition.getName() + ", " +
partition.getDataSize(true)
++ ",
(doris) branch branch-2.0 updated: [improvement](clone) dead be will abort sched task #36795 (#36895)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 02b9d7a26d6 [improvement](clone) dead be will abort sched task #36795
(#36895)
02b9d7a26d6 is described below
commit 02b9d7a26d6cae300d09b05e6ad2d80b7d907121
Author: yujun
AuthorDate: Thu Jun 27 22:51:05 2024 +0800
[improvement](clone) dead be will abort sched task #36795 (#36895)
cherry pick from #36795
---
.../org/apache/doris/clone/TabletScheduler.java| 52 ++--
.../apache/doris/common/util/DebugPointUtil.java | 10 +-
.../java/org/apache/doris/system/HeartbeatMgr.java | 10 ++
.../apache/doris/clone/BeDownCancelCloneTest.java | 147 +
.../apache/doris/utframe/MockedBackendFactory.java | 8 ++
5 files changed, 213 insertions(+), 14 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/clone/TabletScheduler.java
b/fe/fe-core/src/main/java/org/apache/doris/clone/TabletScheduler.java
index 3341f5bb305..715739310bd 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/clone/TabletScheduler.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/clone/TabletScheduler.java
@@ -1822,36 +1822,64 @@ public class TabletScheduler extends MasterDaemon {
* If task is timeout, remove the tablet.
*/
public void handleRunningTablets() {
+Set aliveBeIds =
Sets.newHashSet(Env.getCurrentSystemInfo().getAllBackendIds(true));
// 1. remove the tablet ctx if timeout
-List timeoutTablets = Lists.newArrayList();
+List cancelTablets = Lists.newArrayList();
synchronized (this) {
-
runningTablets.values().stream().filter(TabletSchedCtx::isTimeout).forEach(timeoutTablets::add);
+for (TabletSchedCtx tabletCtx : runningTablets.values()) {
+long srcBeId = tabletCtx.getSrcBackendId();
+long destBeId = tabletCtx.getDestBackendId();
+if (Config.disable_tablet_scheduler) {
+tabletCtx.setErrMsg("tablet scheduler is disabled");
+cancelTablets.add(tabletCtx);
+} else if (Config.disable_balance && tabletCtx.getType() ==
Type.BALANCE) {
+tabletCtx.setErrMsg("balance is disabled");
+cancelTablets.add(tabletCtx);
+} else if (tabletCtx.isTimeout()) {
+tabletCtx.setErrMsg("timeout");
+cancelTablets.add(tabletCtx);
+stat.counterCloneTaskTimeout.incrementAndGet();
+} else if (destBeId > 0 && !aliveBeIds.contains(destBeId)) {
+tabletCtx.setErrMsg("dest be " + destBeId + " is dead");
+cancelTablets.add(tabletCtx);
+} else if (srcBeId > 0 && !aliveBeIds.contains(srcBeId)) {
+tabletCtx.setErrMsg("src be " + srcBeId + " is dead");
+cancelTablets.add(tabletCtx);
+}
+}
}
// 2. release ctx
-timeoutTablets.forEach(t -> {
+cancelTablets.forEach(t -> {
// Set "resetReplicaState" to true because
// the timeout task should also be considered as UNRECOVERABLE,
// so need to reset replica state.
-t.setErrMsg("timeout");
-finalizeTabletCtx(t, TabletSchedCtx.State.CANCELLED,
Status.UNRECOVERABLE, "timeout");
-stat.counterCloneTaskTimeout.incrementAndGet();
+finalizeTabletCtx(t, TabletSchedCtx.State.CANCELLED,
Status.UNRECOVERABLE, t.getErrMsg());
});
}
public List> getPendingTabletsInfo(int limit) {
-List tabletCtxs = getCopiedTablets(pendingTablets,
limit);
-return collectTabletCtx(tabletCtxs);
+return collectTabletCtx(getPendingTablets(limit));
+}
+
+public List getPendingTablets(int limit) {
+return getCopiedTablets(pendingTablets, limit);
}
public List> getRunningTabletsInfo(int limit) {
-List tabletCtxs =
getCopiedTablets(runningTablets.values(), limit);
-return collectTabletCtx(tabletCtxs);
+return collectTabletCtx(getRunningTablets(limit));
+}
+
+public List getRunningTablets(int limit) {
+return getCopiedTablets(runningTablets.values(), limit);
}
public List> getHistoryTabletsInfo(int limit) {
-List tabletCtxs = getCopiedTablets(schedHistory,
limit);
-return collectTabletCtx(tabletCtxs);
+return collectTabletCtx(getHistoryTablets(limit));
+}
+
+public List getHistoryTablets(int limit) {
+return getCopiedTablets(schedHistory, limit);
}
private List> collectTabletCtx(List
tabletCtxs) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/util/DebugPointUtil.java
b/fe/fe-core/src/main/jav
(doris) branch master updated: [fix](fe ut) fix unstable SystemInfoServiceTest (#36893)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 065d65fb768 [fix](fe ut) fix unstable SystemInfoServiceTest (#36893)
065d65fb768 is described below
commit 065d65fb768181961f7e15f9419cd72970c5884e
Author: yujun
AuthorDate: Thu Jun 27 22:54:50 2024 +0800
[fix](fe ut) fix unstable SystemInfoServiceTest (#36893)
For a random alg, (max -min) < 5% is too unstable. Make this diff
bigger, change it to 30%.
---
.../org/apache/doris/system/SystemInfoServiceTest.java| 15 +++
1 file changed, 11 insertions(+), 4 deletions(-)
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/system/SystemInfoServiceTest.java
b/fe/fe-core/src/test/java/org/apache/doris/system/SystemInfoServiceTest.java
index 0ac72df4a6d..b36f4b2a815 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/system/SystemInfoServiceTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/system/SystemInfoServiceTest.java
@@ -46,6 +46,8 @@ import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
public class SystemInfoServiceTest {
private SystemInfoService infoService;
@@ -403,7 +405,7 @@ public class SystemInfoServiceTest {
ReplicaAllocation replicaAlloc = ReplicaAllocation.DEFAULT_ALLOCATION;
// also check if the random selection logic can evenly distribute the
replica.
Map beCounterMap = Maps.newHashMap();
-for (int i = 0; i < 1; ++i) {
+for (int i = 0; i < 3; ++i) {
Pair>, TStorageMedium> ret =
infoService.selectBackendIdsForReplicaCreation(replicaAlloc,
Maps.newHashMap(), TStorageMedium.HDD, false, false);
Map> res = ret.first;
@@ -412,11 +414,16 @@ public class SystemInfoServiceTest {
beCounterMap.put(beId, beCounterMap.getOrDefault(beId, 0) + 1);
}
}
+Set expectBackendIds = infoService.getMixBackends().stream()
+.filter(Backend::isAlive).map(Backend::getId)
+.collect(Collectors.toSet());
+Assert.assertEquals(expectBackendIds,
beCounterMap.keySet().stream().collect(Collectors.toSet()));
List list = Lists.newArrayList(beCounterMap.values());
Collections.sort(list);
-int diff = list.get(list.size() - 1) - list.get(0);
-// The max replica num and min replica num's diff is less than 5%.
-Assert.assertTrue((diff * 1.0 / list.get(0)) < 0.05);
+int max = list.get(list.size() - 1);
+int diff = max - list.get(0);
+// The max replica num and min replica num's diff is less than 30%.
+Assert.assertTrue((diff * 1.0 / max) < 0.3);
}
private void addDisk(Backend be, String path, TStorageMedium medium, long
totalB, long availB) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [improvement](partition rebalance) improve partition rebalance choose candidate speed (#36509)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new c92e090cd04 [improvement](partition rebalance) improve partition
rebalance choose candidate speed (#36509)
c92e090cd04 is described below
commit c92e090cd04a2530630b310b1a7aecbc50dc1407
Author: yujun
AuthorDate: Thu Jun 27 22:56:24 2024 +0800
[improvement](partition rebalance) improve partition rebalance choose
candidate speed (#36509)
When partition reblancer choose candidate tablets, it will call
tabletListOfA.removeAll(tabletListOfB), but list.removeAll(list)'s
runtime is O(n^2). Then if each BE contains 10w+ tablets, it's rather
slow. And we found a online case the tablet scheduler thread is busy at
it.
So need improve this search.
---
.../apache/doris/clone/PartitionRebalancer.java| 79 +++---
.../java/org/apache/doris/clone/Rebalancer.java| 3 +
.../org/apache/doris/clone/TabletScheduler.java| 13 ++--
.../java/org/apache/doris/clone/PathSlotTest.java | 5 +-
.../doris/cluster/DecommissionBackendTest.java | 1 -
5 files changed, 66 insertions(+), 35 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/clone/PartitionRebalancer.java
b/fe/fe-core/src/main/java/org/apache/doris/clone/PartitionRebalancer.java
index 23e13e9161b..7095ad8dc54 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/clone/PartitionRebalancer.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/clone/PartitionRebalancer.java
@@ -30,8 +30,8 @@ import org.apache.doris.thrift.TStorageMedium;
import com.google.common.base.Preconditions;
import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
import com.google.common.collect.Ordering;
+import com.google.common.collect.Sets;
import com.google.common.collect.TreeMultimap;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -41,7 +41,9 @@ import java.util.List;
import java.util.Map;
import java.util.NavigableSet;
import java.util.Random;
+import java.util.Set;
import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.BiPredicate;
import java.util.stream.Collectors;
/*
@@ -121,44 +123,64 @@ public class PartitionRebalancer extends Rebalancer {
= algo.getNextMoves(clusterBalanceInfo,
Config.partition_rebalance_max_moves_num_per_selection);
List alternativeTablets = Lists.newArrayList();
-List inProgressIds = movesInProgressList.stream().map(m ->
m.tabletId).collect(Collectors.toList());
+Set inProgressIds = movesInProgressList.stream().map(m ->
m.tabletId).collect(Collectors.toSet());
+Random rand = new SecureRandom();
for (TwoDimensionalGreedyRebalanceAlgo.PartitionMove move : moves) {
// Find all tablets of the specified partition that would have a
replica at the source be,
// but would not have a replica at the destination be. That is to
satisfy the restriction
// of having no more than one replica of the same tablet per be.
List tabletIds =
invertedIndex.getTabletIdsByBackendIdAndStorageMedium(move.fromBe, medium);
-List invalidIds =
invertedIndex.getTabletIdsByBackendIdAndStorageMedium(move.toBe, medium);
-tabletIds.removeAll(invalidIds);
-// In-progress tablets can't be the candidate too.
-tabletIds.removeAll(inProgressIds);
+if (tabletIds.isEmpty()) {
+continue;
+}
+
+Set invalidIds = Sets.newHashSet(
+
invertedIndex.getTabletIdsByBackendIdAndStorageMedium(move.toBe, medium));
-Map tabletCandidates = Maps.newHashMap();
-for (long tabletId : tabletIds) {
+BiPredicate canMoveTablet = (Long tabletId,
TabletMeta tabletMeta) -> {
+return tabletMeta != null
+&& tabletMeta.getPartitionId() == move.partitionId
+&& tabletMeta.getIndexId() == move.indexId
+&& !invalidIds.contains(tabletId)
+&& !inProgressIds.contains(tabletId);
+};
+
+// Random pick one candidate to create tabletSchedCtx
+int startIdx = rand.nextInt(tabletIds.size());
+long pickedTabletId = -1L;
+TabletMeta pickedTabletMeta = null;
+for (int i = startIdx; i < tabletIds.size(); i++) {
+long tabletId = tabletIds.get(i);
TabletMeta tabletMeta = invertedIndex.getTabletMeta(tabletId);
-if (tabletMeta != null && tabletMeta.getPartitionId() ==
move.partitionId
-&& tabletMeta.getIndexId() == move.indexId) {
-tabletCandidates.put(tabletId, tabletMeta)
(doris) branch branch-2.1 updated: [fix](statistics)Use ConcurrentHashMap to avoid ConcurrentModificationException (#36452) (#36950)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 0051832f91c [fix](statistics)Use ConcurrentHashMap to avoid
ConcurrentModificationException (#36452) (#36950)
0051832f91c is described below
commit 0051832f91c1a42eae2096d94b2c28ead2b773ac
Author: Jibing-Li <64681310+jibing...@users.noreply.github.com>
AuthorDate: Thu Jun 27 23:06:03 2024 +0800
[fix](statistics)Use ConcurrentHashMap to avoid
ConcurrentModificationException (#36452) (#36950)
OlapTable idToPartition Map should use ConcurrentHashMap to avoid
ConcurrentModificationException.
backport: https://github.com/apache/doris/pull/36452
---
fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
index fa6425c72af..0164c311974 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
@@ -99,6 +99,7 @@ import java.util.Map.Entry;
import java.util.Objects;
import java.util.Optional;
import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
import java.util.stream.Collectors;
/**
@@ -140,7 +141,7 @@ public class OlapTable extends Table implements
MTMVRelatedTableIf {
@SerializedName("partitionInfo")
private PartitionInfo partitionInfo;
@SerializedName("idToPartition")
-private Map idToPartition = new HashMap<>();
+private ConcurrentHashMap idToPartition = new
ConcurrentHashMap<>();
private Map nameToPartition = Maps.newTreeMap();
@SerializedName(value = "distributionInfo")
@@ -611,7 +612,7 @@ public class OlapTable extends Table implements
MTMVRelatedTableIf {
// reset partition info and idToPartition map
Map partitionMap = Maps.newHashMap();
Map origIdToPartition = idToPartition;
-idToPartition = Maps.newHashMap();
+idToPartition = new ConcurrentHashMap<>();
for (Map.Entry entry : origPartNameToId.entrySet()) {
long newPartId = env.getNextId();
idToPartition.put(newPartId,
origIdToPartition.get(entry.getValue()));
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (55d81a4ba7c -> e6baf22bee5)
This is an automated email from the ASF dual-hosted git repository. gavinchou pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 55d81a4ba7c [fix](regression) fix regression test case failure (#36870) add e6baf22bee5 [fix](cloud) Fix loss some tag info in cloud (#36866) No new revisions were added by this update. Summary of changes: .../apache/doris/cloud/system/CloudSystemInfoService.java| 12 ++-- .../src/main/java/org/apache/doris/system/Backend.java | 3 ++- 2 files changed, 12 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [case](udf) Only one backend, skip scp udf file (#36810) (#36965)
This is an automated email from the ASF dual-hosted git repository.
hellostephen pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 0ee6e7f0040 [case](udf) Only one backend, skip scp udf file (#36810)
(#36965)
0ee6e7f0040 is described below
commit 0ee6e7f00408ec7b0bd0013bca33256e071f98d6
Author: Dongyang Li
AuthorDate: Fri Jun 28 00:26:00 2024 +0800
[case](udf) Only one backend, skip scp udf file (#36810) (#36965)
pick #36810
---
.../groovy/org/apache/doris/regression/suite/Suite.groovy | 15 +++
1 file changed, 11 insertions(+), 4 deletions(-)
diff --git
a/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy
b/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy
index f771edade1b..9c37ddb0542 100644
---
a/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy
+++
b/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy
@@ -558,7 +558,7 @@ class Suite implements GroovyInterceptable {
Assert.assertEquals(0, code)
}
-void sshExec(String username, String host, String cmd) {
+void sshExec(String username, String host, String cmd, boolean alert=true)
{
String command = "ssh ${username}@${host} '${cmd}'"
def cmds = ["/bin/bash", "-c", command]
logger.info("Execute: ${cmds}".toString())
@@ -566,8 +566,10 @@ class Suite implements GroovyInterceptable {
def errMsg = new StringBuilder()
def msg = new StringBuilder()
p.waitForProcessOutput(msg, errMsg)
-assert errMsg.length() == 0: "error occurred!" + errMsg
-assert p.exitValue() == 0
+if (alert) {
+assert errMsg.length() == 0: "error occurred!\n" + errMsg
+assert p.exitValue() == 0
+}
}
List getFrontendIpHttpPort() {
@@ -872,10 +874,15 @@ class Suite implements GroovyInterceptable {
def backendId_to_backendIP = [:]
def backendId_to_backendHttpPort = [:]
getBackendIpHttpPort(backendId_to_backendIP,
backendId_to_backendHttpPort)
+if(backendId_to_backendIP.size() == 1) {
+logger.info("Only one backend, skip scp udf file")
+return
+}
def udf_file_dir = new File(udf_file_path).parent
backendId_to_backendIP.values().each { be_ip ->
-sshExec ("root", be_ip, "ssh -o StrictHostKeyChecking=no
root@${be_ip} \"mkdir -p ${udf_file_dir}\"")
+sshExec("root", be_ip, "ssh-keygen -f '/root/.ssh/known_hosts' -R
\"${be_ip}\"", false)
+sshExec("root", be_ip, "ssh -o StrictHostKeyChecking=no
root@${be_ip} \"mkdir -p ${udf_file_dir}\"", false)
scpFiles("root", be_ip, udf_file_path, udf_file_path, false)
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch asf-site updated (22ae23c1884 -> bd419c1de4f)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/doris-website.git discard 22ae23c1884 Automated deployment with doris branch @ 092b0c05a0af0389c543473397a36b8a3a110637 new bd419c1de4f Automated deployment with doris branch @ 092b0c05a0af0389c543473397a36b8a3a110637 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (22ae23c1884) \ N -- N -- N refs/heads/asf-site (bd419c1de4f) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/1.2/search-index.json | 2 +- docs/2.0/search-index.json | 2 +- docs/dev/search-index.json | 2 +- search-index.json| 2 +- zh-CN/docs/1.2/search-index.json | 2 +- zh-CN/docs/2.0/search-index.json | 2 +- zh-CN/docs/dev/search-index.json | 2 +- zh-CN/search-index.json | 2 +- 8 files changed, 8 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (e6baf22bee5 -> 1b1029ff72e)
This is an automated email from the ASF dual-hosted git repository. cambyzju pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from e6baf22bee5 [fix](cloud) Fix loss some tag info in cloud (#36866) add 1b1029ff72e [opt](mow) reduce memory usage for mow table compaction (#36865) No new revisions were added by this update. Summary of changes: be/src/common/config.cpp | 2 + be/src/common/config.h | 2 + be/src/olap/base_tablet.cpp| 8 +++- be/src/olap/compaction.cpp | 53 +++--- be/src/olap/utils.h| 2 + regression-test/pipeline/external/conf/be.conf | 1 + regression-test/pipeline/p0/conf/be.conf | 1 + regression-test/pipeline/p1/conf/be.conf | 1 + 8 files changed, 46 insertions(+), 24 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (1b1029ff72e -> 3f3a6eb629f)
This is an automated email from the ASF dual-hosted git repository. lijibing pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 1b1029ff72e [opt](mow) reduce memory usage for mow table compaction (#36865) add 3f3a6eb629f [improvement](statistics)Use real base index id to fetch stats cache. (#36914) No new revisions were added by this update. Summary of changes: .../doris/nereids/stats/StatsCalculator.java | 6 ++--- .../nereids/trees/plans/algebra/OlapScan.java | 6 - .../logical/LogicalDeferMaterializeOlapScan.java | 5 .../trees/plans/logical/LogicalOlapScan.java | 8 -- .../physical/PhysicalDeferMaterializeOlapScan.java | 5 .../trees/plans/physical/PhysicalOlapScan.java | 8 -- .../apache/doris/statistics/StatisticsCache.java | 29 ++ 7 files changed, 32 insertions(+), 35 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (3f3a6eb629f -> d601d10e5b2)
This is an automated email from the ASF dual-hosted git repository. gabriellee pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 3f3a6eb629f [improvement](statistics)Use real base index id to fetch stats cache. (#36914) add d601d10e5b2 [refactor](minor) Delete non-pipeline profile reporting (#36952) No new revisions were added by this update. Summary of changes: be/src/pipeline/pipeline_fragment_context.cpp | 3 +- be/src/runtime/fragment_mgr.cpp | 70 +++ be/src/runtime/query_context.h| 1 - 3 files changed, 29 insertions(+), 45 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](autoinc) avoid duplicated auto inc when role of fe changes (#36960)
This is an automated email from the ASF dual-hosted git repository.
zhangchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 8ad8cac3d78 [fix](autoinc) avoid duplicated auto inc when role of fe
changes (#36960)
8ad8cac3d78 is described below
commit 8ad8cac3d7856e71e8f6ad9798f5515d49a7e2a7
Author: Yongqiang YANG <98214048+dataroar...@users.noreply.github.com>
AuthorDate: Fri Jun 28 10:00:17 2024 +0800
[fix](autoinc) avoid duplicated auto inc when role of fe changes (#36960)
avoid duplicated auto inc when role of fe changes
---
.../java/org/apache/doris/catalog/AutoIncrementGenerator.java| 9 -
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/AutoIncrementGenerator.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/AutoIncrementGenerator.java
index 9528f07e0ca..be110360850 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/catalog/AutoIncrementGenerator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/catalog/AutoIncrementGenerator.java
@@ -23,6 +23,7 @@ import org.apache.doris.common.io.Text;
import org.apache.doris.common.io.Writable;
import org.apache.doris.persist.AutoIncrementIdUpdateLog;
import org.apache.doris.persist.EditLog;
+import org.apache.doris.persist.gson.GsonPostProcessable;
import org.apache.doris.persist.gson.GsonUtils;
import com.google.common.base.Preconditions;
@@ -34,7 +35,7 @@ import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
-public class AutoIncrementGenerator implements Writable {
+public class AutoIncrementGenerator implements Writable, GsonPostProcessable {
private static final Logger LOG =
LogManager.getLogger(AutoIncrementGenerator.class);
public static final long NEXT_ID_INIT_VALUE = 1;
@@ -102,4 +103,10 @@ public class AutoIncrementGenerator implements Writable {
public static AutoIncrementGenerator read(DataInput in) throws IOException
{
return GsonUtils.GSON.fromJson(Text.readString(in),
AutoIncrementGenerator.class);
}
+
+@Override
+public void gsonPostProcess() throws IOException {
+nextId = batchEndId;
+}
+
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch master updated: [doc] Add release note 2.0.12 (#799)
This is an automated email from the ASF dual-hosted git repository.
luzhijing pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new d8451e1a814 [doc] Add release note 2.0.12 (#799)
d8451e1a814 is described below
commit d8451e1a8147314c0dbceb3aa2969a570b1e2a22
Author: lishiqi_amy
AuthorDate: Fri Jun 28 10:14:44 2024 +0800
[doc] Add release note 2.0.12 (#799)
---
docs/releasenotes/release-2.0.12.md| 58 ++
.../current/releasenotes/release-2.0.12.md | 52 +++
.../version-2.0/releasenotes/release-2.0.12.md | 52 +++
sidebars.json | 1 +
.../version-2.0/releasenotes/release-2.0.12.md | 58 ++
versioned_sidebars/version-2.0-sidebars.json | 1 +
6 files changed, 222 insertions(+)
diff --git a/docs/releasenotes/release-2.0.12.md
b/docs/releasenotes/release-2.0.12.md
new file mode 100644
index 000..0bc289c91a8
--- /dev/null
+++ b/docs/releasenotes/release-2.0.12.md
@@ -0,0 +1,58 @@
+---
+{
+"title": "Release 2.0.12",
+"language": "en"
+}
+---
+
+
+
+Thanks to our community developers and users for their contributions. Doris
version 2.0.12 will bring 99 improvements and bug fixes.
+
+**Quick Download:** https://doris.apache.org/download/
+
+**GitHub Release:** https://github.com/apache/doris/releases
+
+## Behavior changes
+
+- No longer set the default table comment to the table type. Instead, set it
to be empty by default, for example, change COMMENT 'OLAP' to COMMENT ' '.
This new behavior is more friendly for BI software that relies on table
comments. [#35855](https://github.com/apache/doris/pull/35855)
+
+- Change the type of the `@@autocommit` variable from `BOOLEAN` to `BIGINT` to
prevent errors from certain MySQL clients (such as .NET MySQL.Data).
[#33282](https://github.com/apache/doris/pull/33282)
+
+
+## Improvements
+
+- Remove the `disable_nested_complex_type` parameter and allow the creation of
nested `ARRAY`, `MAP`, and `STRUCT` types by default.
[#36255](https://github.com/apache/doris/pull/36255)
+
+- The HMS catalog supports the `SHOW CREATE DATABASE` command.
[#28145](https://github.com/apache/doris/pull/28145)
+
+- Add more inverted index metrics to the query profile.
[#36545](https://github.com/apache/doris/pull/36545)
+
+- Cross-Cluster Replication (CCR) supports inverted indices.
[#31743](https://github.com/apache/doris/pull/31743)
+
+You can access the full list through the GitHub
[link](https://github.com/apache/doris/compare/2.0.11...2.0.12) , with the key
features and improvements highlighted below.
+
+
+
+## Credits
+
+Thanks all who contribute to this release:
+
+@airborne12, D14@amorynan, D14@BiteThet, D14@cambyzju, D14@caoliang-web,
D14@dataroaring, D14@eldenmoon, D14@feiniaofeiafei, D14@felixwluo,
D14@gavinchou, D14@HappenLee, D14@hello-stephen, D14@jacktengg, D14@Jibing-Li,
D14@Johnnyssc, D14@liaoxin01, D14@LiBinfeng-01, D14@luwei16, D14@mongo360,
D14@morningman, D14@morrySnow, D14@mrhhsg, D14@Mryange, D14@mymeiyi,
D14@qidaye, D14@qzsee, D14@starocean999, D14@w41ter, D14@wangbo, D14@wsjz,
D14@wuwenchi, D14@xiaokang, D14@XuPengfei-1020, D1 [...]
\ No newline at end of file
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/releasenotes/release-2.0.12.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/releasenotes/release-2.0.12.md
new file mode 100644
index 000..68ad100f1f1
--- /dev/null
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/releasenotes/release-2.0.12.md
@@ -0,0 +1,52 @@
+---
+{
+"title": "Release 2.0.12",
+"language": "zh-CN"
+}
+---
+
+
+
+亲爱的社区小伙伴们，[Apache Doris 2.0.12](https://doris.apache.org/download/) 版本已于 2024
年 6 月 27 日正式与大家见面，该版本提交了 99 个改进项以及问题修复，欢迎大家下载体验。
+
+**官网下载：** https://doris.apache.org/download/
+
+**GitHub 下载：** https://github.com/apache/doris/releases
+
+## 行为变更
+
+- 不再将建表的默认注释设置为表的类型，而是改成默认为空，比如 COMMENT 'OLAP' 变成 COMMENT ''，这样对于依赖注释的 BI
软件更加友好。 [#35855](https://github.com/apache/doris/pull/35855)
+
+- 将 `@@autocommit` 变量的类型从 `BOOLEAN` 改成 `BIGINT`，以免有些 MySQL 客户端（比如.NET
MySQL.Data）报错。 [#33282](https://github.com/apache/doris/pull/33282)
+
+
+## 改进优化
+
+- 删除 `disable_nested_complex_type` 参数，默认允许创建嵌套的 `ARRAY` `MAP` `STRUCT`
类型。[#36255](https://github.com/apache/doris/pull/36255)
+
+- HMS Catalog 支持 `SHOW CREATE DATABASE` 命令。[
#28145](https://github.com/apache/doris/pull/28145)
+
+- 在 Query Profile
中增加更多倒排索引的指标。[#36545](https://github.com/apache/doris/pull/36545)
+
+- 跨集群数据复制（CCR）支持倒排索引 [#31743](https://github.com/apache/doris/pull/31743)
+
+## 致谢
+
+@amorynan、@BiteThet、@cambyzju、@caoliang-web、@dataroaring、@eldenmoon、@feiniaofeiafei、@felixwluo、@gavinchou、@HappenLee、@hello-stephen、@jacktengg、@Jibing-Li、@Johnnyssc、@liaoxin01、@LiBinfeng-01、@luwei16、@mongo360、@morningman、@morrySnow、@mrhh
Error while running notifications feature from refs/heads/master:.asf.yaml in doris-website!
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@foo.apache.org'. Must be a valid @doris.apache.org list! - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running notifications feature from refs/heads/master:.asf.yaml in doris-website!
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@foo.apache.org'. Must be a valid @doris.apache.org list! - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch master updated: Update release note 2.1.4 (#795)
This is an automated email from the ASF dual-hosted git repository. luzhijing pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new c3290b50e3e Update release note 2.1.4 (#795) c3290b50e3e is described below commit c3290b50e3e9373149721ad42901e4dff1fc185e Author: lishiqi_amy AuthorDate: Fri Jun 28 10:19:34 2024 +0800 Update release note 2.1.4 (#795) --- docs/releasenotes/release-2.1.4.md | 74 +++- .../current/releasenotes/release-2.1.4.md | 80 +- .../version-2.1/releasenotes/release-2.1.4.md | 80 +- .../version-2.1/releasenotes/release-2.1.4.md | 74 +++- 4 files changed, 174 insertions(+), 134 deletions(-) diff --git a/docs/releasenotes/release-2.1.4.md b/docs/releasenotes/release-2.1.4.md index bff23d629b5..d04a8be0cf6 100644 --- a/docs/releasenotes/release-2.1.4.md +++ b/docs/releasenotes/release-2.1.4.md @@ -24,7 +24,7 @@ specific language governing permissions and limitations under the License. --> -Apache Doris version 2.1.4 was officially released on June 26, 2024. In this update, we have optimized various functional experiences for data lakehouse scenarios, with a focus on resolving the abnormal memory usage issue in the previous version. Additionally, we have implemented several improvemnents and bug fixes to enhance the stability. Welcome to download and use it. +**Apache Doris version 2.1.4 was officially released on June 26, 2024.** In this update, we have optimized various functional experiences for data lakehouse scenarios, with a focus on resolving the abnormal memory usage issue in the previous version. Additionally, we have implemented several improvemnents and bug fixes to enhance the stability. Welcome to download and use it. **Quick Download:** https://doris.apache.org/download/ @@ -32,7 +32,7 @@ Apache Doris version 2.1.4 was officially released on June 26, 2024. In this upd **GitHub Release:** https://github.com/apache/doris/releases -## Behavior Changed +## Behavior changes - Non-existent files will be ignored when querying external tables such as Hive. [#35319](https://github.com/apache/doris/pull/35319) @@ -48,21 +48,21 @@ Apache Doris version 2.1.4 was officially released on June 26, 2024. In this upd For more infomation, please see the documentations: - - [Log Management - FE Log](https://doris.apache.org/docs/admin-manual/log-management/fe-log) + - [Log Management - FE Log](../admin-manual/log-management/fe-log.md) - - [Log Management - FE Log](https://doris.apache.org/docs/admin-manual/log-management/be-log) + - [Log Management - FE Log](../admin-manual/log-management/be-log.md) - If no table comment is provided when creating a table, the default comment will be empty instead of using the table type as the default comment. [#36025](https://github.com/apache/doris/pull/36025) - The default precision of DECIMALV3 has been adjusted from (9, 0) to (38, 9) to maintain compatibility with the version in which this feature was initially released. [#36316](https://github.com/apache/doris/pull/36316) -## New Features +## New features -### 01 Query Optimizer +### Query optimizer - Support FE flame graph tool - For more information, see the [documentation](https://doris.apache.org/community/developer-guide/fe-profiler) + For more information, see the [documentation](/community/developer-guide/fe-profiler.md) - Support `SELECT DISTINCT` to be used with aggregation. @@ -70,7 +70,7 @@ Apache Doris version 2.1.4 was officially released on June 26, 2024. In this upd - The new optimizer fully supports point query functionality [#36205](https://github.com/apache/doris/pull/36205). -### 02 Lakehouse +### Lakehouse - Support native reader of Apache Paimon deletion vector [#35241](https://github.com/apache/doris/pull/35241) @@ -78,33 +78,29 @@ Apache Doris version 2.1.4 was officially released on June 26, 2024. In this upd - Access controller with Hive Ranger plugin supports Data Mask -### 03 Asynchronous Materialized Views +### Asynchronous materialized views -- Support partition roll-up during construction. [#31812](https://github.com/apache/doris/pull/31812) +- Build support for internal table triggered updates, where if a materialized view uses an internal table and the data in the internal table changes, it can trigger a refresh of the materialized view, specifying REFRESH ON COMMIT when creating the materialized view. -- Support triggered updates during construction. [#34548](https://github.com/apache/doris/pull/34548) +- Support transparent rewriting for single tables. For more information, see [Querying Async Materialized View](../query/view-materialized-view/query-async-materialized-view.md). -- S
Error while running notifications feature from refs/heads/master:.asf.yaml in doris-website!
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@foo.apache.org'. Must be a valid @doris.apache.org list! - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch master updated: [opt](docs) Opt documents for datatypes and compaction (#792)
This is an automated email from the ASF dual-hosted git repository. luzhijing pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new 0b0e8103f9f [opt](docs) Opt documents for datatypes and compaction (#792) 0b0e8103f9f is described below commit 0b0e8103f9f5b9ee9fb6d801e3ba2cea016ba67e Author: Kang AuthorDate: Fri Jun 28 10:19:51 2024 +0800 [opt](docs) Opt documents for datatypes and compaction (#792) 1. reorg datetypes order 2. opt JSON type docs 3. add single replica and time_series compaction docs - Co-authored-by: Luzhijing <82810928+luzhij...@users.noreply.github.com> --- docs/admin-manual/compaction.md| 58 ++-- docs/sql-manual/sql-types/Data-Types/JSON.md | 57 --- docs/table-design/data-type.md | 24 +++ .../current/admin-manual/compaction.md | 51 +- .../sql-manual/sql-types/Data-Types/JSON.md| 81 -- .../current/table-design/data-type.md | 30 .../version-2.0/admin-manual/compaction.md | 51 +- .../sql-manual/sql-reference/Data-Types/JSON.md| 81 -- .../version-2.0/table-design/data-type.md | 29 .../version-2.1/admin-manual/compaction.md | 51 +- .../sql-manual/sql-types/Data-Types/JSON.md| 61 .../version-2.1/table-design/data-type.md | 30 sidebars.json | 12 ++-- .../version-2.0/admin-manual/compaction.md | 55 ++- .../sql-manual/sql-reference/Data-Types/JSON.md| 57 --- .../version-2.0/table-design/data-type.md | 22 +++--- .../version-2.1/admin-manual/compaction.md | 55 ++- .../sql-manual/sql-types/Data-Types/JSON.md| 57 --- .../version-2.1/table-design/data-type.md | 24 +++ versioned_sidebars/version-2.0-sidebars.json | 6 +- versioned_sidebars/version-2.1-sidebars.json | 12 ++-- 21 files changed, 708 insertions(+), 196 deletions(-) diff --git a/docs/admin-manual/compaction.md b/docs/admin-manual/compaction.md index e48756b57e2..b953cb5c137 100644 --- a/docs/admin-manual/compaction.md +++ b/docs/admin-manual/compaction.md @@ -29,13 +29,10 @@ under the License. Doris writes data through a structure similar to LSM-Tree, and continuously merges small files into large ordered files through compaction in the background. Compaction handles operations such as deletion and updating. -Appropriately adjusting the compaction strategy can greatly improve load and query efficiency. Doris provides the following two compaction strategies for tuning: +Appropriately adjusting the compaction strategy can greatly improve load and query efficiency. Doris provides the following compaction strategies for tuning: -## Vertical Compaction - - - +## Vertical compaction Vertical compaction is a new compaction algorithm implemented in Doris 1.2.2, which is used to optimize compaction execution efficiency and resource overhead in large-scale and wide table scenarios. It can effectively reduce the memory overhead of compaction and improve the execution speed of compaction. The test results show that the memory consumption by vertical compaction is only 1/10 of the original compaction algorithm, and the compaction rate is increased by 15%. @@ -47,7 +44,7 @@ BE configuration： - `vertical_compaction_max_segment_size` is used to configure the size of the disk file after vertical compaction, the default value is 268435456 (bytes) -## Segment Compaction +## Segment compaction Segment compaction mainly deals with the large-scale data load. Segment compaction operates during the load process and compact segments inside the job, which is different from normal compaction and vertical compaction. This mechanism can effectively reduce the number of generated segments and avoid the -238 (OLAP_ERR_TOO_MANY_SEGMENTS) errors. @@ -72,3 +69,52 @@ Situations where segment compaction is not recommended: - When the load operation itself has exhausted memory resources, it is not recommended to use the segment compaction to avoid further increasing memory pressure and causing the load job to fail. Refer to this [link](https://github.com/apache/doris/pull/12866) for more information about implementation and test results. + +## Single replica compaction + +By default, compaction for multiple replicas is performed independently, with each replica consuming CPU and IO resources. When single replica compaction is enabled, only one replica performs the compaction. Afterward, the other replicas pull the compacted files from this replica, resulting in CPU resources being consumed only once, saving N - 1 ti
(doris) branch master updated: [fix](doap) Fix DOAP syntax (#36956)
This is an automated email from the ASF dual-hosted git repository. kirs pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 17a17cd1034 [fix](doap) Fix DOAP syntax (#36956) 17a17cd1034 is described below commit 17a17cd10344bf06e63fde04a67458edc443fde2 Author: Arnout Engelen AuthorDate: Fri Jun 28 04:29:19 2024 +0200 [fix](doap) Fix DOAP syntax (#36956) xmlns strings need to match exactly, and these vocabularies are defined with `http` namespace strings, so we need to follow that. --- doap_Doris.rdf | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doap_Doris.rdf b/doap_Doris.rdf index 8be1d6d2e2f..37513252086 100644 --- a/doap_Doris.rdf +++ b/doap_Doris.rdf @@ -2,9 +2,9 @@ http://usefulinc.com/ns/doap#"; - xmlns:rdf="https://www.w3.org/1999/02/22-rdf-syntax-ns#"; - xmlns:asfext="https://projects.apache.org/ns/asfext#"; - xmlns:foaf="https://xmlns.com/foaf/0.1/";> + xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"; + xmlns:asfext="http://projects.apache.org/ns/asfext#"; + xmlns:foaf="http://xmlns.com/foaf/0.1/";>
(doris) branch master updated (17a17cd1034 -> 1c4b54240b1)
This is an automated email from the ASF dual-hosted git repository. lijibing pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 17a17cd1034 [fix](doap) Fix DOAP syntax (#36956) add 1c4b54240b1 [test](statisitics)Add partition stats test for mtmv. (#36849) No new revisions were added by this update. Summary of changes: .../suites/statistics/test_analyze_mtmv.groovy | 701 ++--- 1 file changed, 329 insertions(+), 372 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (1c4b54240b1 -> 875ad5d91d6)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 1c4b54240b1 [test](statisitics)Add partition stats test for mtmv.
(#36849)
add 875ad5d91d6 [test](tvf) move p2 tvf tests from p2 to p0 (#36871)
No new revisions were added by this update.
Summary of changes:
.../scripts/data/tvf/test_hdfs_tvf_compression/run.sh | 19 +++
.../hive/scripts/data/tvf/test_tvf/run.sh | 19 +++
.../tvf/test_hdfs_tvf_compression.out | 0
.../tvf/test_path_partition_keys.out | 0
.../tvf/test_path_partition_keys/dt1=cyw/a.csv| 0
.../tvf/test_path_partition_keys/dt1=cyw/b.csv| 0
.../tvf/test_path_partition_keys/dt1=hello/c.csv | 0
.../test_path_partition_keys/dt2=two/dt1=cyw/a.csv| 0
.../test_path_partition_keys/dt2=two/dt1=cyw/b.csv| 0
.../test_path_partition_keys/dt2=two/dt1=hello/c.csv | 0
.../tvf/test_tvf_p2.out | 0
.../tvf/test_hdfs_tvf_compression.groovy | 10 +-
.../tvf/test_path_partition_keys.groovy | 10 +-
.../tvf/test_tvf_p2.groovy| 8
14 files changed, 52 insertions(+), 14 deletions(-)
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/tvf/test_hdfs_tvf_compression/run.sh
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/data/tvf/test_tvf/run.sh
rename regression-test/data/{external_table_p2 =>
external_table_p0}/tvf/test_hdfs_tvf_compression.out (100%)
rename regression-test/data/{external_table_p2 =>
external_table_p0}/tvf/test_path_partition_keys.out (100%)
rename regression-test/data/{external_table_p2 =>
external_table_p0}/tvf/test_path_partition_keys/dt1=cyw/a.csv (100%)
rename regression-test/data/{external_table_p2 =>
external_table_p0}/tvf/test_path_partition_keys/dt1=cyw/b.csv (100%)
rename regression-test/data/{external_table_p2 =>
external_table_p0}/tvf/test_path_partition_keys/dt1=hello/c.csv (100%)
rename regression-test/data/{external_table_p2 =>
external_table_p0}/tvf/test_path_partition_keys/dt2=two/dt1=cyw/a.csv (100%)
rename regression-test/data/{external_table_p2 =>
external_table_p0}/tvf/test_path_partition_keys/dt2=two/dt1=cyw/b.csv (100%)
rename regression-test/data/{external_table_p2 =>
external_table_p0}/tvf/test_path_partition_keys/dt2=two/dt1=hello/c.csv (100%)
rename regression-test/data/{external_table_p2 =>
external_table_p0}/tvf/test_tvf_p2.out (100%)
rename regression-test/suites/{external_table_p2 =>
external_table_p0}/tvf/test_hdfs_tvf_compression.groovy (92%)
rename regression-test/suites/{external_table_p2 =>
external_table_p0}/tvf/test_path_partition_keys.groovy (95%)
rename regression-test/suites/{external_table_p2 =>
external_table_p0}/tvf/test_tvf_p2.groovy (93%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [fix](statistics)Fix AnalysisTaskExecutor unit test timeout. (#36456) (#36943)
This is an automated email from the ASF dual-hosted git repository.
lijibing pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 2a99862c6b6 [fix](statistics)Fix AnalysisTaskExecutor unit test
timeout. (#36456) (#36943)
2a99862c6b6 is described below
commit 2a99862c6b650721cd7a01228709501df74c2a7a
Author: Jibing-Li <64681310+jibing...@users.noreply.github.com>
AuthorDate: Fri Jun 28 11:00:19 2024 +0800
[fix](statistics)Fix AnalysisTaskExecutor unit test timeout. (#36456)
(#36943)
Deencapsulation.getField sometime doesn't work as expected, which may
cause FE unit test timeout. Add get function in source code to make it
easier.
backport: https://github.com/apache/doris/pull/36456
---
.../java/org/apache/doris/statistics/AnalysisTaskExecutor.java | 5 +
.../java/org/apache/doris/statistics/AnalysisTaskExecutorTest.java | 7 ++-
2 files changed, 7 insertions(+), 5 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisTaskExecutor.java
b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisTaskExecutor.java
index 3bdccaca047..5277d8025fc 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisTaskExecutor.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisTaskExecutor.java
@@ -105,4 +105,9 @@ public class AnalysisTaskExecutor {
executors.getQueue().clear();
taskQueue.clear();
}
+
+// For unit test only.
+public BlockingQueue getTaskQueue() {
+return taskQueue;
+}
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisTaskExecutorTest.java
b/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisTaskExecutorTest.java
index 8cfcfeabd28..d73dc45509a 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisTaskExecutorTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisTaskExecutorTest.java
@@ -45,7 +45,6 @@ import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
-import java.util.concurrent.BlockingQueue;
import java.util.concurrent.atomic.AtomicBoolean;
public class AnalysisTaskExecutorTest extends TestWithFeService {
@@ -105,14 +104,12 @@ public class AnalysisTaskExecutorTest extends
TestWithFeService {
OlapAnalysisTask analysisJob = new OlapAnalysisTask(analysisJobInfo);
AnalysisTaskExecutor analysisTaskExecutor = new
AnalysisTaskExecutor(1);
-BlockingQueue b =
Deencapsulation.getField(analysisTaskExecutor, "taskQueue");
AnalysisTaskWrapper analysisTaskWrapper = new
AnalysisTaskWrapper(analysisTaskExecutor, analysisJob);
Deencapsulation.setField(analysisTaskWrapper, "startTime", 5);
-b.put(analysisTaskWrapper);
+analysisTaskExecutor.putJob(analysisTaskWrapper);
analysisTaskExecutor.tryToCancel();
Assertions.assertTrue(cancelled.get());
-Assertions.assertTrue(b.isEmpty());
-
+Assertions.assertEquals(0, analysisTaskExecutor.getTaskQueue().size());
}
@Test
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](Nereids) processCharacterLiteral even if both side are literal (#36729) (#36921)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 557da5b24a8 [fix](Nereids) processCharacterLiteral even if both side
are literal (#36729) (#36921)
557da5b24a8 is described below
commit 557da5b24a8794479db259b333cdaa2b824a7e21
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Fri Jun 28 11:01:01 2024 +0800
[fix](Nereids) processCharacterLiteral even if both side are literal
(#36729) (#36921)
pick from master #36729
---
.../doris/nereids/util/TypeCoercionUtils.java | 4
.../test_string_literal_vs_other_literal.out | 11 ++
.../test_string_literal_vs_other_literal.groovy| 24 ++
3 files changed, 35 insertions(+), 4 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java
index e6dcca83a8b..0bbd11007a0 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java
@@ -505,10 +505,6 @@ public class TypeCoercionUtils {
if (!(left instanceof Literal) && !(right instanceof Literal)) {
return (T) op.withChildren(left, right);
}
-if (left instanceof Literal && right instanceof Literal) {
-// process by constant folding
-return (T) op.withChildren(left, right);
-}
if (left instanceof Literal && ((Literal) left).isStringLikeLiteral()
&& !right.getDataType().isStringLikeType()) {
left = TypeCoercionUtils.characterLiteralTypeCoercion(
diff --git
a/regression-test/data/nereids_p0/type_inference/test_string_literal_vs_other_literal.out
b/regression-test/data/nereids_p0/type_inference/test_string_literal_vs_other_literal.out
new file mode 100644
index 000..788187beeba
--- /dev/null
+++
b/regression-test/data/nereids_p0/type_inference/test_string_literal_vs_other_literal.out
@@ -0,0 +1,11 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !numeric_eq --
+1
+
+-- !numeric_non_eq --
+
+-- !date_eq --
+1
+
+-- !date_eq --
+
diff --git
a/regression-test/suites/nereids_p0/type_inference/test_string_literal_vs_other_literal.groovy
b/regression-test/suites/nereids_p0/type_inference/test_string_literal_vs_other_literal.groovy
new file mode 100644
index 000..43ad242de07
--- /dev/null
+++
b/regression-test/suites/nereids_p0/type_inference/test_string_literal_vs_other_literal.groovy
@@ -0,0 +1,24 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_string_literal_vs_other_literal") {
+qt_numeric_eq """select 1 where "12345678901234567890" =
12345678901234567890;"""
+qt_numeric_non_eq """select 1 where "12345678901234567890" =
12345678901234567891;"""
+
+qt_date_eq """select 1 where "2024-06-24 00:01:02.345678" =
cast("2024-06-24 00:01:02.345678" as datetime(6)) """
+qt_date_eq """select 1 where "2024-06-24 00:01:02.345679" =
cast("2024-06-24 00:01:02.345678" as datetime(6)) """
+}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (875ad5d91d6 -> 850d79b311f)
This is an automated email from the ASF dual-hosted git repository. jianliangqi pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 875ad5d91d6 [test](tvf) move p2 tvf tests from p2 to p0 (#36871) add 850d79b311f [enhancement](index compaction) Enable index compaction by default (#36812) No new revisions were added by this update. Summary of changes: be/src/common/config.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](inverted index)Make build index operation only affect base index (#36869)
This is an automated email from the ASF dual-hosted git repository.
jianliangqi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 783d4b8fbf1 [fix](inverted index)Make build index operation only
affect base index (#36869)
783d4b8fbf1 is described below
commit 783d4b8fbf1fb317abaf08c7654a8f33f43382a4
Author: qiye
AuthorDate: Fri Jun 28 11:07:13 2024 +0800
[fix](inverted index)Make build index operation only affect base index
(#36869)
When building index on table with mv, we should only send index build
jobs to base index.
Otherwise the build process may be stuck.
This PR make the build index operation only be allowed in base index and
add a test case.
---
.../apache/doris/alter/SchemaChangeHandler.java| 2 +
.../test_add_drop_index_on_table_with_mv.groovy| 135 +
2 files changed, 137 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java
b/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java
index 15f8c88bb64..6b42916a824 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java
@@ -3045,6 +3045,8 @@ public class SchemaChangeHandler extends AlterHandler {
// for now table's state can only be NORMAL
Preconditions.checkState(olapTable.getState() ==
OlapTableState.NORMAL, olapTable.getState().name());
+// remove the index which is not the base index, only base index can
be built inverted index
+indexSchemaMap.entrySet().removeIf(entry ->
!entry.getKey().equals(olapTable.getBaseIndexId()));
// begin checking each table
Map> changedIndexIdToSchema = Maps.newHashMap();
try {
diff --git
a/regression-test/suites/inverted_index_p0/index_change/test_add_drop_index_on_table_with_mv.groovy
b/regression-test/suites/inverted_index_p0/index_change/test_add_drop_index_on_table_with_mv.groovy
new file mode 100644
index 000..7b63521178e
--- /dev/null
+++
b/regression-test/suites/inverted_index_p0/index_change/test_add_drop_index_on_table_with_mv.groovy
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_add_drop_index_on_table_with_mv") {
+def tableName = "test_add_drop_index_on_table_with_mv"
+
+def timeout = 6
+def delta_time = 1000
+def alter_res = "null"
+def useTime = 0
+def wait_for_latest_op_on_table_finish = { table_name, OpTimeout ->
+for(int t = delta_time; t <= OpTimeout; t += delta_time){
+alter_res = sql """SHOW ALTER TABLE COLUMN WHERE TableName =
"${table_name}" ORDER BY CreateTime DESC LIMIT 1;"""
+alter_res = alter_res.toString()
+if(alter_res.contains("FINISHED")) {
+sleep(1) // wait change table state to normal
+logger.info(table_name + " latest alter job finished, detail:
" + alter_res)
+break
+}
+useTime = t
+sleep(delta_time)
+}
+assertTrue(useTime <= OpTimeout, "wait_for_latest_op_on_table_finish
timeout")
+}
+
+def wait_for_build_index_on_partition_finish = { table_name, OpTimeout ->
+for(int t = delta_time; t <= OpTimeout; t += delta_time){
+alter_res = sql """SHOW BUILD INDEX WHERE TableName =
"${table_name}";"""
+def expected_finished_num = alter_res.size();
+logger.info("expected_finished_num: " + expected_finished_num)
+// check only base table build index job
+assertEquals(1, expected_finished_num)
+def finished_num = 0;
+for (int i = 0; i < expected_finished_num; i++) {
+logger.info(table_name + " build index job state: " +
alter_res[i][7] + i)
+if (alter_res[i][7] == "FINISHED") {
+++finished_num;
+}
+}
+if (finished_num == expected_finished_num) {
+logger.info(table_name + " all build index job
(doris) branch master updated (783d4b8fbf1 -> 7deb82284c5)
This is an automated email from the ASF dual-hosted git repository.
jianliangqi pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 783d4b8fbf1 [fix](inverted index)Make build index operation only
affect base index (#36869)
add 7deb82284c5 [tests](inverted index)Add and improve add/drop index
cases (#36811)
No new revisions were added by this update.
Summary of changes:
.../java/org/apache/doris/alter/AlterJobV2.java| 2 +-
.../apache/doris/alter/SchemaChangeHandler.java| 1 +
.../test_index_change_on_new_column.out| 4 ++
.../test_index_change_on_renamed_column.out} | 35 ++---
...st_index_change_with_cumulative_compaction.out} | 0
.../test_index_change_with_full_compaction.out}| 0
...d_index_with_direct_schema_change_fault.groovy} | 35 +
...ld_index_with_light_schema_change_fault.groovy} | 30
.../{ => index_change}/test_add_drop_index.groovy | 0
.../test_add_drop_index_ignore_case_column.groovy | 0
.../test_add_drop_index_repeatly.groovy} | 57 --
.../test_add_drop_index_with_data.groovy | 0
.../test_add_drop_index_with_delete_data.groovy| 0
.../{ => index_change}/test_build_index.groovy | 0
.../test_build_index_with_clone_by_docker.groovy | 0
.../test_index_change_on_new_column.groovy | 8 ++-
... => test_index_change_on_renamed_column.groovy} | 37 +-
...index_change_with_cumulative_compaction.groovy} | 6 +--
... test_index_change_with_full_compaction.groovy} | 8 +--
19 files changed, 89 insertions(+), 134 deletions(-)
copy
regression-test/data/{correctness_p0/test_partial_const_args_for_function.out
=> inverted_index_p0/index_change/test_index_change_on_renamed_column.out} (59%)
rename
regression-test/data/inverted_index_p0/index_change/{test_index_change_with_compaction.out
=> test_index_change_with_cumulative_compaction.out} (100%)
copy regression-test/data/{compaction/test_compaction_dup_keys.out =>
inverted_index_p0/index_change/test_index_change_with_full_compaction.out}
(100%)
copy
regression-test/suites/fault_injection_p0/{test_build_index_with_clone_fault.groovy
=> test_build_index_with_direct_schema_change_fault.groovy} (69%)
copy
regression-test/suites/fault_injection_p0/{test_build_index_with_clone_fault.groovy
=> test_build_index_with_light_schema_change_fault.groovy} (71%)
rename regression-test/suites/inverted_index_p0/{ =>
index_change}/test_add_drop_index.groovy (100%)
rename regression-test/suites/inverted_index_p0/{ =>
index_change}/test_add_drop_index_ignore_case_column.groovy (100%)
copy
regression-test/suites/{fault_injection_p0/test_build_index_with_clone_fault.groovy
=> inverted_index_p0/index_change/test_add_drop_index_repeatly.groovy} (60%)
rename regression-test/suites/inverted_index_p0/{ =>
index_change}/test_add_drop_index_with_data.groovy (100%)
rename regression-test/suites/inverted_index_p0/{ =>
index_change}/test_add_drop_index_with_delete_data.groovy (100%)
rename regression-test/suites/inverted_index_p0/{ =>
index_change}/test_build_index.groovy (100%)
rename regression-test/suites/inverted_index_p0/{ =>
index_change}/test_build_index_with_clone_by_docker.groovy (100%)
copy
regression-test/suites/inverted_index_p0/index_change/{test_index_change_on_new_column.groovy
=> test_index_change_on_renamed_column.groovy} (75%)
copy
regression-test/suites/inverted_index_p0/index_change/{test_index_change_with_compaction.groovy
=> test_index_change_with_cumulative_compaction.groovy} (98%)
rename
regression-test/suites/inverted_index_p0/index_change/{test_index_change_with_compaction.groovy
=> test_index_change_with_full_compaction.groovy} (98%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (7deb82284c5 -> 6b300ff6af6)
This is an automated email from the ASF dual-hosted git repository. lihaopeng pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 7deb82284c5 [tests](inverted index)Add and improve add/drop index cases (#36811) add 6b300ff6af6 [feat](function) Function to encode/decode varchar (#36649) No new revisions were added by this update. Summary of changes: be/src/util/simd/reverse_copy_bytes.h | 37 be/src/vec/functions/function_decode_varchar.cpp | 123 +++ be/src/vec/functions/function_encode_varchar.cpp | 115 +++ be/src/vec/functions/simple_function_factory.h | 4 + .../function_compressed_materialization_test.cpp | 225 + .../doris/catalog/BuiltinScalarFunctions.java | 10 + .../functions/scalar/DecodeAsVarchar.java | 75 +++ .../functions/scalar/EncodeAsBigInt.java | 69 +++ .../expressions/functions/scalar/EncodeAsInt.java | 69 +++ .../functions/scalar/EncodeAsLargeInt.java | 68 +++ .../functions/scalar/EncodeAsSmallInt.java | 69 +++ .../expressions/visitor/ScalarFunctionVisitor.java | 25 +++ gensrc/script/doris_builtins_functions.py | 8 + .../string_functions/test_code_varchar.out | 85 .../string_functions/test_code_varchar.groovy | 104 ++ 15 files changed, 1086 insertions(+) create mode 100644 be/src/util/simd/reverse_copy_bytes.h create mode 100644 be/src/vec/functions/function_decode_varchar.cpp create mode 100644 be/src/vec/functions/function_encode_varchar.cpp create mode 100644 be/test/vec/function/function_compressed_materialization_test.cpp create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/DecodeAsVarchar.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/EncodeAsBigInt.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/EncodeAsInt.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/EncodeAsLargeInt.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/EncodeAsSmallInt.java create mode 100644 regression-test/data/query_p0/sql_functions/string_functions/test_code_varchar.out create mode 100644 regression-test/suites/query_p0/sql_functions/string_functions/test_code_varchar.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (6b300ff6af6 -> 4012bceb3c8)
This is an automated email from the ASF dual-hosted git repository. lihaopeng pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 6b300ff6af6 [feat](function) Function to encode/decode varchar (#36649) add 4012bceb3c8 [Bug](cast) fix cast string to int return wrong result (#36788) No new revisions were added by this update. Summary of changes: be/src/util/string_parser.hpp| 5 + regression-test/data/datatype_p0/json/json_cast.out | 12 regression-test/suites/datatype_p0/json/json_cast.groovy | 6 +- 3 files changed, 22 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated (557da5b24a8 -> 69487c47205)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git from 557da5b24a8 [fix](Nereids) processCharacterLiteral even if both side are literal (#36729) (#36921) add 69487c47205 [refactor](nereids) Remove SlotBinder and FunctionBinder (#36872) (#36930) No new revisions were added by this update. Summary of changes: .../doris/nereids/rules/analysis/BindSink.java | 50 +-- .../nereids/rules/analysis/ExpressionAnalyzer.java | 107 -- .../doris/nereids/rules/analysis/SlotBinder.java | 320 -- .../nereids/rules/analysis/SubExprAnalyzer.java| 3 + .../rules/expression/rules/FunctionBinder.java | 365 - .../expressions/functions/udf/AliasUdfBuilder.java | 30 +- .../trees/plans/commands/UpdateCommand.java| 6 +- .../rules/analysis/FillUpMissingSlotsTest.java | 7 +- .../expression/ExpressionRewriteTestHelper.java| 4 +- .../nereids/rules/expression/FoldConstantTest.java | 6 +- .../expression/SimplifyArithmeticRuleTest.java | 8 +- .../rules/expression/SimplifyRangeTest.java| 4 +- 12 files changed, 138 insertions(+), 772 deletions(-) delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SlotBinder.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/FunctionBinder.java - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated (69487c47205 -> 328ef869ac9)
This is an automated email from the ASF dual-hosted git repository. lijibing pushed a change to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git from 69487c47205 [refactor](nereids) Remove SlotBinder and FunctionBinder (#36872) (#36930) add 328ef869ac9 [fix](regression)Disable auto analyze for related cases, do not need to enable it for pipeline test cases. (#36604) (#36946) No new revisions were added by this update. Summary of changes: .../suites/statistics/test_analyze_mtmv.groovy | 911 ++--- .../suites/statistics/test_update_rows_mv.groovy | 2 - 2 files changed, 453 insertions(+), 460 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](bitmap) incorrect type of BitmapValue with fastunion (#36834) (#36896)
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new f27ae8fa097 [fix](bitmap) incorrect type of BitmapValue with fastunion
(#36834) (#36896)
f27ae8fa097 is described below
commit f27ae8fa097902d2c7cfb7f0020ebeed5c7f95ac
Author: Jerry Hu
AuthorDate: Fri Jun 28 11:29:03 2024 +0800
[fix](bitmap) incorrect type of BitmapValue with fastunion (#36834) (#36896)
---
be/src/util/bitmap_value.h | 5 +++--
be/test/util/bitmap_value_test.cpp | 23 +++
2 files changed, 26 insertions(+), 2 deletions(-)
diff --git a/be/src/util/bitmap_value.h b/be/src/util/bitmap_value.h
index d8f68a227e7..ea99fa58baf 100644
--- a/be/src/util/bitmap_value.h
+++ b/be/src/util/bitmap_value.h
@@ -1652,7 +1652,6 @@ public:
case SINGLE: {
_set.insert(_sv);
_type = SET;
-_convert_to_bitmap_if_need();
break;
}
case BITMAP:
@@ -1663,10 +1662,12 @@ public:
_type = BITMAP;
break;
case SET: {
-_convert_to_bitmap_if_need();
break;
}
}
+if (_type == SET) {
+_convert_to_bitmap_if_need();
+}
}
if (_type == EMPTY && single_values.size() == 1) {
diff --git a/be/test/util/bitmap_value_test.cpp
b/be/test/util/bitmap_value_test.cpp
index e7652199ab0..d0ad3a82fda 100644
--- a/be/test/util/bitmap_value_test.cpp
+++ b/be/test/util/bitmap_value_test.cpp
@@ -1026,6 +1026,29 @@ TEST(BitmapValueTest, bitmap_union) {
EXPECT_EQ(3, bitmap3.cardinality());
bitmap3.fastunion({&bitmap});
EXPECT_EQ(5, bitmap3.cardinality());
+
+const auto old_config = config::enable_set_in_bitmap_value;
+config::enable_set_in_bitmap_value = true;
+BitmapValue bitmap4; // empty
+
+BitmapValue bitmap_set1;
+BitmapValue bitmap_set2;
+BitmapValue bitmap_set3;
+
+const int set_data1[] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
15};
+bitmap_set1.add_many(set_data1, 15);
+
+const int set_data2[] = {16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
28, 29, 30};
+bitmap_set2.add_many(set_data2, 15);
+
+const int set_data3[] = {31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42,
43, 44, 45};
+bitmap_set3.add_many(set_data3, 15);
+
+bitmap4.fastunion({&bitmap_set1, &bitmap_set2, &bitmap_set3});
+
+EXPECT_EQ(bitmap4.cardinality(), 45);
+EXPECT_EQ(bitmap4.get_type_code(), BitmapTypeCode::BITMAP32);
+config::enable_set_in_bitmap_value = old_config;
}
TEST(BitmapValueTest, bitmap_intersect) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (4012bceb3c8 -> d7b5411aeb5)
This is an automated email from the ASF dual-hosted git repository. eldenmoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 4012bceb3c8 [Bug](cast) fix cast string to int return wrong result (#36788) add d7b5411aeb5 [Fix](Prepared Statment) use fixed charset to init StringLiteral (#36860) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/doris/analysis/StringLiteral.java | 5 - .../apache/doris/nereids/trees/expressions/literal/Literal.java | 9 +++-- .../src/main/java/org/apache/doris/qe/MysqlConnectProcessor.java | 4 +++- 3 files changed, 14 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [Improvement](inverted index) Remove the check for inverted index file exists (#36929)
This is an automated email from the ASF dual-hosted git repository.
airborne pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 132df2d5876 [Improvement](inverted index) Remove the check for
inverted index file exists (#36929)
132df2d5876 is described below
commit 132df2d587692e2b4ff58a8837e57b84ddc9deec
Author: Sun Chenyang
AuthorDate: Fri Jun 28 12:10:48 2024 +0800
[Improvement](inverted index) Remove the check for inverted index file
exists (#36929)
## Proposed changes
backport #36945
---
be/src/clucene | 2 +-
be/src/olap/olap_common.h | 1 -
.../rowset/segment_v2/inverted_index_cache.cpp | 19
.../inverted_index_compound_directory.cpp | 45 +++
.../segment_v2/inverted_index_compound_reader.h| 2 +-
.../rowset/segment_v2/inverted_index_reader.cpp| 51 +-
be/src/vec/exec/scan/new_olap_scan_node.cpp| 2 -
be/src/vec/exec/scan/new_olap_scan_node.h | 1 -
be/src/vec/exec/scan/new_olap_scanner.cpp | 2 -
9 files changed, 48 insertions(+), 77 deletions(-)
diff --git a/be/src/clucene b/be/src/clucene
index a28adab869f..51f15724f5b 16
--- a/be/src/clucene
+++ b/be/src/clucene
@@ -1 +1 @@
-Subproject commit a28adab869f1397aefd7c3636d977c406613617d
+Subproject commit 51f15724f5bdb7ae5f6f5e5d7072d43a5bda63f8
diff --git a/be/src/olap/olap_common.h b/be/src/olap/olap_common.h
index e5f029d2a29..811e77590f9 100644
--- a/be/src/olap/olap_common.h
+++ b/be/src/olap/olap_common.h
@@ -357,7 +357,6 @@ struct OlapReaderStatistics {
int64_t inverted_index_query_timer = 0;
int64_t inverted_index_query_cache_hit = 0;
int64_t inverted_index_query_cache_miss = 0;
-int64_t inverted_index_query_file_exists_timer = 0;
int64_t inverted_index_query_null_bitmap_timer = 0;
int64_t inverted_index_query_bitmap_copy_timer = 0;
int64_t inverted_index_query_bitmap_op_timer = 0;
diff --git a/be/src/olap/rowset/segment_v2/inverted_index_cache.cpp
b/be/src/olap/rowset/segment_v2/inverted_index_cache.cpp
index 035e28efabd..41e3b134a32 100644
--- a/be/src/olap/rowset/segment_v2/inverted_index_cache.cpp
+++ b/be/src/olap/rowset/segment_v2/inverted_index_cache.cpp
@@ -136,18 +136,17 @@ Status
InvertedIndexSearcherCache::get_index_searcher(const io::FileSystemSPtr&
std::unique_ptr(new
MemTracker("InvertedIndexSearcherCacheWithRead"));
#ifndef BE_TEST
{
-bool exists = false;
-{
-SCOPED_RAW_TIMER(&stats->inverted_index_query_file_exists_timer);
-RETURN_IF_ERROR(fs->exists(file_path, &exists));
-}
-if (!exists) {
-return Status::Error(
-"inverted index path: {} not exist.", file_path);
-}
SCOPED_RAW_TIMER(&stats->inverted_index_searcher_open_timer);
SCOPED_CONSUME_MEM_TRACKER(mem_tracker.get());
-index_searcher = build_index_searcher(fs, index_dir, file_name);
+try {
+index_searcher = build_index_searcher(fs, index_dir, file_name);
+} catch (CLuceneError& err) {
+if (err.number() == CL_ERR_FileNotFound) {
+return Status::Error(
+"inverted index path: {} not exist.", file_path);
+}
+throw err;
+}
}
#endif
diff --git
a/be/src/olap/rowset/segment_v2/inverted_index_compound_directory.cpp
b/be/src/olap/rowset/segment_v2/inverted_index_compound_directory.cpp
index 0eb7e31a027..1c1857f2ac2 100644
--- a/be/src/olap/rowset/segment_v2/inverted_index_compound_directory.cpp
+++ b/be/src/olap/rowset/segment_v2/inverted_index_compound_directory.cpp
@@ -266,30 +266,26 @@ bool DorisCompoundDirectory::FSIndexInput::open(const
io::FileSystemSPtr& fs, co
io::FileBlockCachePathPolicy cache_policy;
auto type = config::enable_file_cache ? config::file_cache_type : "";
io::FileReaderOptions reader_options(io::cache_type_from_string(type),
cache_policy);
-if (!fs->open_file(fd, reader_options, &h->_reader).ok()) {
-error.set(CL_ERR_IO, "open file error");
+auto st = fs->open_file(fd, reader_options, &h->_reader);
+if (st.is_not_found()) {
+error.set(CL_ERR_FileNotFound, "File does not exist");
+} else if (st.is_io_error()) {
+error.set(CL_ERR_IO, "File open io error");
+} else if (st.code() == ErrorCode::PERMISSION_DENIED) {
+error.set(CL_ERR_IO, "File Access denied");
+} else {
+error.set(CL_ERR_IO, "Could not open file");
}
//Check if a valid handle was retrieved
-if (h->_reader) {
+if (st.ok() && h->_reader) {
//Store the file length
h->_length = h->_reader->size();
h->_fpos = 0;
ret = _CLNEW FSIndexInput(h, buffer_size);
(doris) branch branch-2.1 updated: [fix](statistics)Fix AnalysisTaskExecutor unit test timeout. (#36456) (#36948)
This is an automated email from the ASF dual-hosted git repository.
lijibing pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new da294906bdf [fix](statistics)Fix AnalysisTaskExecutor unit test
timeout. (#36456) (#36948)
da294906bdf is described below
commit da294906bdf19f83ac1ba6600aa36205e3d9b601
Author: Jibing-Li <64681310+jibing...@users.noreply.github.com>
AuthorDate: Fri Jun 28 12:11:53 2024 +0800
[fix](statistics)Fix AnalysisTaskExecutor unit test timeout. (#36456)
(#36948)
Deencapsulation.getField sometime doesn't work as expected, which may
cause FE unit test timeout. Add get function in source code to make it
easier.
backport: https://github.com/apache/doris/pull/36456
---
.../java/org/apache/doris/statistics/AnalysisTaskExecutor.java | 5 +
.../java/org/apache/doris/statistics/AnalysisTaskExecutorTest.java | 7 ++-
2 files changed, 7 insertions(+), 5 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisTaskExecutor.java
b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisTaskExecutor.java
index 3bdccaca047..5277d8025fc 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisTaskExecutor.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisTaskExecutor.java
@@ -105,4 +105,9 @@ public class AnalysisTaskExecutor {
executors.getQueue().clear();
taskQueue.clear();
}
+
+// For unit test only.
+public BlockingQueue getTaskQueue() {
+return taskQueue;
+}
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisTaskExecutorTest.java
b/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisTaskExecutorTest.java
index 5698f0e9b20..cef9c483f0c 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisTaskExecutorTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisTaskExecutorTest.java
@@ -44,7 +44,6 @@ import org.junit.jupiter.api.Test;
import java.util.Collections;
import java.util.List;
import java.util.Map;
-import java.util.concurrent.BlockingQueue;
import java.util.concurrent.atomic.AtomicBoolean;
public class AnalysisTaskExecutorTest extends TestWithFeService {
@@ -104,14 +103,12 @@ public class AnalysisTaskExecutorTest extends
TestWithFeService {
OlapAnalysisTask analysisJob = new OlapAnalysisTask(analysisJobInfo);
AnalysisTaskExecutor analysisTaskExecutor = new
AnalysisTaskExecutor(1);
-BlockingQueue b =
Deencapsulation.getField(analysisTaskExecutor, "taskQueue");
AnalysisTaskWrapper analysisTaskWrapper = new
AnalysisTaskWrapper(analysisTaskExecutor, analysisJob);
Deencapsulation.setField(analysisTaskWrapper, "startTime", 5);
-b.put(analysisTaskWrapper);
+analysisTaskExecutor.putJob(analysisTaskWrapper);
analysisTaskExecutor.tryToCancel();
Assertions.assertTrue(cancelled.get());
-Assertions.assertTrue(b.isEmpty());
-
+Assertions.assertEquals(0, analysisTaskExecutor.getTaskQueue().size());
}
@Test
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (d7b5411aeb5 -> e613b54b5ab)
This is an automated email from the ASF dual-hosted git repository. starocean999 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from d7b5411aeb5 [Fix](Prepared Statment) use fixed charset to init StringLiteral (#36860) add e613b54b5ab [mv](nereids) add expression cost to filter (#36789) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/doris/nereids/cost/CostModelV1.java | 8 +++- 1 file changed, 3 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running notifications feature from refs/heads/master:.asf.yaml in doris-website!
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@foo.apache.org'. Must be a valid @doris.apache.org list! - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch master updated (0b0e8103f9f -> 3f0691c7124)
This is an automated email from the ASF dual-hosted git repository. luzhijing pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git from 0b0e8103f9f [opt](docs) Opt documents for datatypes and compaction (#792) add 3f0691c7124 [docs](jdbc catalog) fix oracle and sap hana create examples and instructions (#791) No new revisions were added by this update. Summary of changes: docs/lakehouse/database/oracle.md | 10 +- docs/lakehouse/database/sap-hana.md| 6 +- .../current/lakehouse/database/oracle.md | 26 ++--- .../current/lakehouse/database/sap-hana.md | 106 - .../version-2.0/lakehouse/database/oracle.md | 28 +++--- .../version-2.0/lakehouse/database/sap-hana.md | 106 - .../version-2.1/lakehouse/database/oracle.md | 26 ++--- .../version-2.1/lakehouse/database/sap-hana.md | 6 +- .../version-2.0/lakehouse/database/oracle.md | 8 +- .../version-2.0/lakehouse/database/sap-hana.md | 6 +- .../version-2.1/lakehouse/database/oracle.md | 8 +- .../version-2.1/lakehouse/database/sap-hana.md | 6 +- 12 files changed, 211 insertions(+), 131 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch master updated: [doc] fix conf path in quick-start (#796)
This is an automated email from the ASF dual-hosted git repository. luzhijing pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new 7d5958495c [doc] fix conf path in quick-start (#796) 7d5958495c is described below commit 7d5958495c37dfbf24ee4f0293d07cef0354 Author: zjj AuthorDate: Fri Jun 28 12:47:36 2024 +0800 [doc] fix conf path in quick-start (#796) --- docs/get-starting/quick-start.md | 2 +- docs/install/cluster-deployment/standard-deployment.md| 2 +- .../current/get-starting/quick-start.md | 4 ++-- .../current/install/cluster-deployment/standard-deployment.md | 2 +- .../version-2.0/get-starting/quick-start.md | 4 ++-- .../version-2.0/install/cluster-deployment/standard-deployment.md | 2 +- .../version-2.1/get-starting/quick-start.md | 4 ++-- .../version-2.1/install/cluster-deployment/standard-deployment.md | 2 +- versioned_docs/version-2.0/get-starting/quick-start.md| 4 ++-- .../version-2.0/install/cluster-deployment/standard-deployment.md | 2 +- versioned_docs/version-2.1/get-starting/quick-start.md| 4 ++-- .../version-2.1/install/cluster-deployment/standard-deployment.md | 2 +- 12 files changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/get-starting/quick-start.md b/docs/get-starting/quick-start.md index a44bff6b16..9b4868215d 100644 --- a/docs/get-starting/quick-start.md +++ b/docs/get-starting/quick-start.md @@ -77,7 +77,7 @@ server1:apache-doris/fe doris$ ./bin/start_fe.sh --daemon ### Configure BE -Go to the `apache-doris/be/be.conf` file for BE configuration. Below are some key configurations to pay attention to. Add JAVA_HOME manually and point it to your JDK8 runtime environment. For other configurations, you can go with the default values for a quick single-machine experience. +Go to the `apache-doris/be/conf/be.conf` file for BE configuration. Below are some key configurations to pay attention to. Add JAVA_HOME manually and point it to your JDK8 runtime environment. For other configurations, you can go with the default values for a quick single-machine experience. ```Shell # Add JAVA_HOME and point it to your JDK8 runtime environment. Suppose your JDK8 is at /home/doris/jdk8, set it as follows: diff --git a/docs/install/cluster-deployment/standard-deployment.md b/docs/install/cluster-deployment/standard-deployment.md index 94c0bd2364..aa10c1f064 100644 --- a/docs/install/cluster-deployment/standard-deployment.md +++ b/docs/install/cluster-deployment/standard-deployment.md @@ -369,7 +369,7 @@ The configuration file for BE is in the "conf" directory under the BE deployment 1. Configure Java environment Starting from version 1.2, Doris supports Java UDF (User-Defined Function), and BE relies on the Java environment. You need to configure the `JAVA_HOME` environment variable in the operating system beforehand or specify the Java environment variable in the BE configuration file. ```SQL -## Modify Java environment variable in be/be.conf +## Modify Java environment variable in be/conf/be.conf JAVA_HOME = ``` diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/get-starting/quick-start.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/get-starting/quick-start.md index ce17bfc217..953d5bdf95 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/get-starting/quick-start.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/get-starting/quick-start.md @@ -54,7 +54,7 @@ server1:~ doris$ mv apache-doris-2.0.3-bin-x64 apache-doris ### 配置 FE -FE 的配置文件为 apache-doris/fe/fe.conf。下面是一些需要关注的核心配置。除了 JAVA_HOME, 需要手动增加，并且指向你的 JDK8 运行环境。其它配置，可以使用默认值，即可支持单机快速体验。 +FE 的配置文件为 apache-doris/fe/conf/fe.conf。下面是一些需要关注的核心配置。除了 JAVA_HOME, 需要手动增加，并且指向你的 JDK8 运行环境。其它配置，可以使用默认值，即可支持单机快速体验。 ```Plain # 增加 JAVA_HOME 配置，指向 JDK8 的运行环境。假如我们 JDK8 位于 /home/doris/jdk8, 则设置如下 @@ -78,7 +78,7 @@ server1:apache-doris/fe doris$ ./bin/start_fe.sh --daemon ### 配置 BE -BE 的配置文件为 apache-doris/be/be.conf。下面是一些需要关注的核心配置。除了 JAVA_HOME, 需要手动增加，并且指向你的 JDK8 运行环境。其它配置，可以使用默认值，即可支持我们的快速体验。 +BE 的配置文件为 apache-doris/be/conf/be.conf。下面是一些需要关注的核心配置。除了 JAVA_HOME, 需要手动增加，并且指向你的 JDK8 运行环境。其它配置，可以使用默认值，即可支持我们的快速体验。 ```Plain # 增加 JAVA_HOME 配置，指向 JDK8 的运行环境。假如我们 JDK8 位于 /home/doris/jdk8, 则设置如下 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/cluster-deployment/standard-deployment.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/cluster-deployment/standard-deployment.md index ad77bca143..709a928dc3 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/cluster-deployment/standard-deployment.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/cluster-deployment/
Error while running notifications feature from refs/heads/master:.asf.yaml in doris-website!
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@foo.apache.org'. Must be a valid @doris.apache.org list! - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running notifications feature from refs/heads/master:.asf.yaml in doris-website!
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@foo.apache.org'. Must be a valid @doris.apache.org list! - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch master updated (3f0691c712 -> 1c09aa1e44)
This is an automated email from the ASF dual-hosted git repository. luzhijing pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git from 3f0691c712 [docs](jdbc catalog) fix oracle and sap hana create examples and instructions (#791) add 1c09aa1e44 [doc](bi)document for tableau (#738) No new revisions were added by this update. Summary of changes: docs/ecosystem/bi/tableau.md | 49 .../current/ecosystem/bi/tableau.md| 51 + .../version-2.0/ecosystem/bi/tableau.md| 51 + .../version-2.1/ecosystem/bi/tableau.md| 51 + sidebars.json | 1 + static/images/bi-tableau-en-1.png | Bin 0 -> 252571 bytes static/images/bi-tableau-en-2.png | Bin 0 -> 56720 bytes static/images/bi-tableau-en-3.png | Bin 0 -> 269345 bytes versioned_docs/version-2.0/ecosystem/bi/tableau.md | 49 versioned_docs/version-2.1/ecosystem/bi/tableau.md | 49 versioned_sidebars/version-2.0-sidebars.json | 1 + versioned_sidebars/version-2.1-sidebars.json | 1 + 12 files changed, 303 insertions(+) create mode 100644 docs/ecosystem/bi/tableau.md create mode 100644 i18n/zh-CN/docusaurus-plugin-content-docs/current/ecosystem/bi/tableau.md create mode 100644 i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/ecosystem/bi/tableau.md create mode 100644 i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/ecosystem/bi/tableau.md create mode 100644 static/images/bi-tableau-en-1.png create mode 100644 static/images/bi-tableau-en-2.png create mode 100644 static/images/bi-tableau-en-3.png create mode 100644 versioned_docs/version-2.0/ecosystem/bi/tableau.md create mode 100644 versioned_docs/version-2.1/ecosystem/bi/tableau.md - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch master updated: [doc](mtmv)Supplement asynchronous materialized view document (#706)
This is an automated email from the ASF dual-hosted git repository.
luzhijing pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new ac7c3dbb8a [doc](mtmv)Supplement asynchronous materialized view
document (#706)
ac7c3dbb8a is described below
commit ac7c3dbb8a5922cc73b1624ecd0a73f35470cd1d
Author: zhangdong <493738...@qq.com>
AuthorDate: Fri Jun 28 12:47:20 2024 +0800
[doc](mtmv)Supplement asynchronous materialized view document (#706)
Co-authored-by: Luzhijing <82810928+luzhij...@users.noreply.github.com>
---
.../async-materialized-view.md | 459 -
.../Create/CREATE-ASYNC-MATERIALIZED-VIEW.md | 26 +-
.../async-materialized-view.md | 451 +++-
.../Create/CREATE-ASYNC-MATERIALIZED-VIEW.md | 30 +-
.../async-materialized-view.md | 457 ++--
.../Create/CREATE-ASYNC-MATERIALIZED-VIEW.md | 68 ++-
.../async-materialized-view.md | 413 --
.../Create/CREATE-ASYNC-MATERIALIZED-VIEW.md | 32 +-
8 files changed, 1835 insertions(+), 101 deletions(-)
diff --git a/docs/query/view-materialized-view/async-materialized-view.md
b/docs/query/view-materialized-view/async-materialized-view.md
index fd42802c6f..21bee69a22 100644
--- a/docs/query/view-materialized-view/async-materialized-view.md
+++ b/docs/query/view-materialized-view/async-materialized-view.md
@@ -131,7 +131,7 @@ REFRESH MATERIALIZED VIEW mv1
partitions(p_20231017_20231018);
Specific syntax can be viewed [REFRESH MATERIALIZED
VIEW](../../sql-manual/sql-statements/Utility-Statements/REFRESH-MATERIALIZED-VIEW.md)
-### task management
+### Task management
Each materialized view defaults to a job responsible for refreshing data,
which is used to describe the refresh strategy and other information of the
materialized view. Each time a refresh is triggered, a task is generated,
Task is used to describe specific refresh information, such as the time used
for refreshing, which partitions were refreshed, etc
@@ -154,7 +154,7 @@ Can pause the scheduled scheduling of materialized views
Specific syntax can be viewed [PAUSE MATERIALIZED VIEW
JOB](../../sql-manual/sql-statements/Utility-Statements/PAUSE-MATERIALIZED-VIEW.md)
- RESUME materialized view job scheduling
+ Resume materialized view job scheduling
```sql
RESUME MATERIALIZED VIEW JOB ON mv1;
@@ -164,7 +164,7 @@ Can RESUME scheduled scheduling of materialized views
Specific syntax can be viewed [RESUME MATERIALIZED VIEW
JOB](../../sql-manual/sql-statements/Utility-Statements/RESUME-MATERIALIZED-VIEW.md)
- Viewing tasks in materialized views
+ View tasks in materialized views
```sql
select * from tasks("type"="mv");
@@ -182,7 +182,7 @@ Can cancel the operation of this task
Specific syntax can be viewed [CANCEL MATERIALIZED VIEW
TASK](../../sql-manual/sql-statements/Utility-Statements/CANCEL-MATERIALIZED-VIEW-TASK.md)
-### Modifying materialized views
+### Modify materialized views
Modify the properties of materialized views
```sql
@@ -203,6 +203,457 @@ The materialized view has a dedicated deletion syntax and
cannot be deleted thro
Specific syntax can be viewed [DROP ASYNC MATERIALIZED
VIEW](../../sql-manual/sql-statements/Data-Definition-Statements/Drop/DROP-ASYNC-MATERIALIZED-VIEW.md)
+## Partition description
+There are two ways to partition materialized views:
+
+1. Custom Partitioning
+
+2. Automatically Create Partitions Based on Dependent Base Table Partitions
+
+### Custom partitioning
+When creating a materialized view without specifying partition information,
the materialized view will default to creating a single partition where all
data will be stored.
+
+### Partitioning based on dependent base tables
+A materialized view can be created by joining multiple base tables.
+
+A materialized view can be partitioned to follow one of the base tables (it is
recommended to choose the fact table).
+
+For example
+
+The table creation statement for t1 is as follows:
+
+```sql
+CREATE TABLE `t1` (
+ `user_id` LARGEINT NOT NULL,
+ `o_date` DATE NOT NULL,
+ `num` SMALLINT NOT NULL
+) ENGINE=OLAP
+COMMENT 'OLAP'
+PARTITION BY RANGE(`o_date`)
+(
+PARTITION p20170101 VALUES [('2017-01-01'), ('2017-01-02')),
+PARTITION p20170102 VALUES [('2017-01-02'), ('2017-01-03')),
+PARTITION p20170201 VALUES [('2017-02-01'), ('2017-02-02'))
+)
+DISTRIBUTED BY HASH(`user_id`) BUCKETS 2
+PROPERTIES ('replication_num' = '1') ;
+```
+
+The table creation statement for t2 is as follows:
+
+```sql
+CREATE TABLE `t2` (
+ `user_id` LARGEINT NOT NULL,
+ `age` SMALLINT NOT NULL
+) ENGINE=OLAP
+PARTITION BY LIST(`age`)
+(
+PARTITION `p1` VALUES IN ('1'),
+PARTITION `p2` VALUES IN ('2')
+)
+DISTRIBUTED BY HASH(`user_id`) BUCKETS 2
+PROPERTIES ('r
(doris-website) branch master updated (ac7c3dbb8a -> 7255580489)
This is an automated email from the ASF dual-hosted git repository. luzhijing pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git from ac7c3dbb8a [doc](mtmv)Supplement asynchronous materialized view document (#706) add 7255580489 Adding Job Description Documents (#784) No new revisions were added by this update. Summary of changes: docs/data-operate/scheduler/job-scheduler.md | 181 + .../docusaurus-plugin-content-docs/current.json| 4 + .../data-operate/scheduler/job-scheduler.md| 174 .../version-2.1.json | 4 + .../data-operate/scheduler/job-scheduler.md| 174 sidebars.json | 7 + .../data-operate/scheduler/job-scheduler.md| 181 + versioned_sidebars/version-2.1-sidebars.json | 7 + 8 files changed, 732 insertions(+) create mode 100644 docs/data-operate/scheduler/job-scheduler.md create mode 100644 i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/scheduler/job-scheduler.md create mode 100644 i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/data-operate/scheduler/job-scheduler.md create mode 100644 versioned_docs/version-2.1/data-operate/scheduler/job-scheduler.md - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running notifications feature from refs/heads/master:.asf.yaml in doris-website!
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@foo.apache.org'. Must be a valid @doris.apache.org list! - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running notifications feature from refs/heads/master:.asf.yaml in doris-website!
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@foo.apache.org'. Must be a valid @doris.apache.org list! - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [feature](Azure) Consider delete non-existent file always successful on Azure (#36932)
This is an automated email from the ASF dual-hosted git repository.
gavinchou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 81aa4968b77 [feature](Azure) Consider delete non-existent file always
successful on Azure (#36932)
81aa4968b77 is described below
commit 81aa4968b77e671cdca7d82ff109333f5446a137
Author: AlexYue
AuthorDate: Fri Jun 28 12:58:50 2024 +0800
[feature](Azure) Consider delete non-existent file always successful on
Azure (#36932)
When deleting non-existent files on object storage like S3, the result
would be considered as always true. This pr makes azure client acts the
same way.
As for the added compare string, you can refer to this
[doc](https://github.com/microsoft/api-guidelines/blob/vNext/azure/ConsiderationsForServiceDesign.md)
---
be/src/io/fs/azure_obj_storage_client.cpp | 34 ++-
cloud/src/recycler/azure_obj_client.cpp | 32 +++--
2 files changed, 45 insertions(+), 21 deletions(-)
diff --git a/be/src/io/fs/azure_obj_storage_client.cpp
b/be/src/io/fs/azure_obj_storage_client.cpp
index b7ff0a7..043886672a2 100644
--- a/be/src/io/fs/azure_obj_storage_client.cpp
+++ b/be/src/io/fs/azure_obj_storage_client.cpp
@@ -58,6 +58,7 @@ auto base64_encode_part_num(int part_num) {
}
constexpr char SAS_TOKEN_URL_TEMPLATE[] =
"https://{}.blob.core.windows.net/{}/{}{}";;
+constexpr char BlobNotFound[] = "BlobNotFound";
} // namespace
namespace doris::io {
@@ -105,17 +106,30 @@ struct AzureBatchDeleter {
return resp;
}
-auto get_defer_response = [](const auto& defer) {
-// DeferredResponse might throw exception
-if (!defer.GetResponse().Value.Deleted) {
-throw Exception(Status::IOError("Batch delete failed"));
-}
-};
for (auto&& defer_response : deferred_resps) {
-auto response =
-do_azure_client_call([&]() {
get_defer_response(defer_response); }, _opts);
-if (response.status.code != ErrorCode::OK) {
-return response;
+try {
+auto r = defer_response.GetResponse();
+if (!r.Value.Deleted) {
+auto msg = fmt::format("Azure batch delete failed, path
msg {}",
+
wrap_object_storage_path_msg(_opts));
+LOG_WARNING(msg);
+return {.status = convert_to_obj_response(
+
Status::InternalError(std::move(msg)))};
+}
+} catch (Azure::Core::RequestFailedException& e) {
+if (Azure::Core::Http::HttpStatusCode::NotFound ==
e.StatusCode &&
+0 == strcmp(e.ErrorCode.c_str(), BlobNotFound)) {
+continue;
+}
+auto msg = fmt::format(
+"Azure request failed because {}, error msg {}, http
code {}, path msg {}",
+e.what(), e.Message, static_cast(e.StatusCode),
+wrap_object_storage_path_msg(_opts));
+LOG_WARNING(msg);
+return {.status = convert_to_obj_response(
+Status::InternalError(std::move(msg))),
+.http_code = static_cast(e.StatusCode),
+.request_id = std::move(e.RequestId)};
}
}
diff --git a/cloud/src/recycler/azure_obj_client.cpp
b/cloud/src/recycler/azure_obj_client.cpp
index b5cd3e76d8e..163c9b43559 100644
--- a/cloud/src/recycler/azure_obj_client.cpp
+++ b/cloud/src/recycler/azure_obj_client.cpp
@@ -39,6 +39,7 @@ using namespace Azure::Storage::Blobs;
namespace doris::cloud {
constexpr size_t BlobBatchMaxOperations = 256;
+constexpr char BlobNotFound[] = "BlobNotFound";
template
ObjectStorageResponse do_azure_client_call(Func f, const
ObjectStoragePathOptions& opts) {
@@ -78,18 +79,27 @@ struct AzureBatchDeleter {
return resp;
}
-auto get_defer_response = [](const auto& defer) {
-// DeferredResponse might throw exception
-if (!defer.GetResponse().Value.Deleted) {
-throw std::runtime_error("Batch delete blobs failed");
-}
-};
-
for (auto&& defer_response : deferred_resps) {
-auto response =
-do_azure_client_call([&]() {
get_defer_response(defer_response); }, _opts);
-if (response.ret != 0) {
-return response;
+try {
+auto r = defer_response.GetResponse();
+if (!r.Value.Deleted) {
+LOG_INFO("A
(doris) branch branch-2.1 updated (da294906bdf -> b226fe54e93)
This is an automated email from the ASF dual-hosted git repository. lijibing pushed a change to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git from da294906bdf [fix](statistics)Fix AnalysisTaskExecutor unit test timeout. (#36456) (#36948) add b226fe54e93 [fix](statistics)Fix select mv with specified partitions bug. (#36817) (#36982) No new revisions were added by this update. Summary of changes: .../org/apache/doris/nereids/rules/analysis/BindRelation.java| 9 ++--- .../nereids/rules/exploration/mv/MaterializedViewUtils.java | 2 ++ .../doris/nereids/trees/plans/logical/LogicalOlapScan.java | 8 3 files changed, 12 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (81aa4968b77 -> 89bbad9db79)
This is an automated email from the ASF dual-hosted git repository. lijibing pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 81aa4968b77 [feature](Azure) Consider delete non-existent file always successful on Azure (#36932) add 89bbad9db79 [improvement](statistics)Disable fetch stats for iceberg table through Iceberg api by default. (#36931) No new revisions were added by this update. Summary of changes: .../java/org/apache/doris/datasource/hive/HMSExternalTable.java | 8 ++-- fe/fe-core/src/main/java/org/apache/doris/qe/GlobalVariable.java | 9 + 2 files changed, 15 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
