[GitHub] [doris] zhannngchen commented on a diff in pull request #10706: [feature-wip](unique-key-merge-on-write) add bloom filter index for primary key, DSIP-018[1.2]

[GitBox]

zhannngchen commented on code in PR #10706:
URL: https://github.com/apache/doris/pull/10706#discussion_r917232884


##
be/src/olap/primary_key_index.cpp:
##
@@ -30,12 +30,17 @@ Status PrimaryKeyIndexBuilder::init() {
 options.encoding = 
segment_v2::EncodingInfo::get_default_encoding(type_info, true);
 // TODO(liaoxin) test to confirm whether it needs to be compressed
 options.compression = segment_v2::NO_COMPRESSION; // currently not 
compressed
-_index_builder.reset(new segment_v2::IndexedColumnWriter(options, 
type_info, _file_writer));
-return _index_builder->init();
+_primary_key_index_builder.reset(
+new segment_v2::IndexedColumnWriter(options, type_info, 
_file_writer));
+RETURN_IF_ERROR(_primary_key_index_builder->init());
+
+return 
segment_v2::BloomFilterIndexWriter::create(segment_v2::BloomFilterOptions(), 
type_info,
+  
&_bloom_filter_index_builder);
 }
 
 Status PrimaryKeyIndexBuilder::add_item(const Slice& key) {
-_index_builder->add(&key);
+_primary_key_index_builder->add(&key);

Review Comment:
   RETURN_IF_ERROR



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #10706: [feature-wip](unique-key-merge-on-write) add bloom filter index for primary key, DSIP-018[1.2]

[GitBox]

github-actions[bot] commented on PR #10706:
URL: https://github.com/apache/doris/pull/10706#issuecomment-1179494625

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #10533: [feature][tracing] Support query tracing to improve doris observabil…

[GitBox]

github-actions[bot] commented on PR #10533:
URL: https://github.com/apache/doris/pull/10533#issuecomment-1179498636

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #10533: [feature][tracing] Support query tracing to improve doris observabil…

[GitBox]

github-actions[bot] commented on PR #10533:
URL: https://github.com/apache/doris/pull/10533#issuecomment-1179498641

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morningman merged pull request #10533: [feature][tracing] Support query tracing to improve doris observabil…

[GitBox]

morningman merged PR #10533:
URL: https://github.com/apache/doris/pull/10533


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [feature](tracing) Support query tracing to improve doris observability by introducing OpenTelemetry. (#10533)

[morningman]

This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new d5ea677282 [feature](tracing) Support query tracing to improve doris 
observability by introducing OpenTelemetry. (#10533)
d5ea677282 is described below

commit d5ea6772829a6cfa8ee46c9c49103e8d036df7b9
Author: luozenglin <37725793+luozeng...@users.noreply.github.com>
AuthorDate: Sat Jul 9 15:50:40 2022 +0800

[feature](tracing) Support query tracing to improve doris observability by 
introducing OpenTelemetry. (#10533)

The collection of query traces is implemented in fe and be, and the spans 
are exported to zipkin.
DSIP: 
https://cwiki.apache.org/confluence/display/DORIS/DSIP-012%3A+Introduce+opentelemetry
---
 be/CMakeLists.txt  |  28 +
 be/src/common/config.h |  15 +++
 be/src/common/status.h |  13 +++
 be/src/exec/data_sink.h|   9 ++
 be/src/exec/exec_node.cpp  |   1 +
 be/src/exec/exec_node.h|  11 ++
 be/src/exec/olap_scanner.cpp   |   2 +
 be/src/exec/table_function_node.cpp|   2 +
 be/src/runtime/fragment_mgr.cpp|  23 +++-
 be/src/runtime/plan_fragment_executor.cpp  |  53 +
 be/src/runtime/runtime_state.h |   7 ++
 be/src/service/doris_main.cpp  |   3 +
 be/src/service/internal_service.cpp|   8 ++
 be/src/util/CMakeLists.txt |   2 +
 be/src/util/runtime_profile.cpp|   2 +-
 be/src/util/runtime_profile.h  |   8 +-
 .../telemetry/brpc_carrier.cpp}|  22 ++--
 be/src/util/telemetry/brpc_carrier.h   |  64 +++
 be/src/util/telemetry/telemetry.cpp|  70 
 be/src/util/telemetry/telemetry.h  | 100 +
 be/src/vec/exec/join/vhash_join_node.cpp   |  17 ++-
 be/src/vec/exec/vaggregation_node.cpp  |  10 +-
 be/src/vec/exec/vanalytic_eval_node.cpp|   7 +-
 be/src/vec/exec/vassert_num_rows_node.cpp  |   6 +-
 be/src/vec/exec/vblocking_join_node.cpp|   5 +-
 be/src/vec/exec/vbroker_scan_node.cpp  |  10 +-
 be/src/vec/exec/vcross_join_node.cpp   |   9 +-
 be/src/vec/exec/vempty_set_node.cpp|   1 +
 be/src/vec/exec/ves_http_scan_node.cpp |  11 +-
 be/src/vec/exec/vexcept_node.cpp   |   3 +
 be/src/vec/exec/vexchange_node.cpp |   3 +
 be/src/vec/exec/vintersect_node.cpp|   3 +
 be/src/vec/exec/vmysql_scan_node.cpp   |   3 +
 be/src/vec/exec/vodbc_scan_node.cpp|   3 +
 be/src/vec/exec/volap_scan_node.cpp|  23 +++-
 be/src/vec/exec/vrepeat_node.cpp   |   7 +-
 be/src/vec/exec/vschema_scan_node.cpp  |   5 +
 be/src/vec/exec/vselect_node.cpp   |   6 +-
 be/src/vec/exec/vset_operation_node.cpp|   8 +-
 be/src/vec/exec/vsort_node.cpp |   6 +-
 be/src/vec/exec/vtable_function_node.cpp   |   6 +-
 be/src/vec/exec/vunion_node.cpp|  10 +-
 be/src/vec/sink/vdata_stream_sender.cpp|   3 +
 be/src/vec/sink/vmysql_table_sink.cpp  |   3 +
 be/src/vec/sink/vresult_file_sink.cpp  |   3 +
 be/src/vec/sink/vresult_sink.cpp   |   3 +
 be/src/vec/sink/vtablet_sink.cpp   |   3 +
 docs/.vuepress/sidebar/en/docs.js  |   1 +
 docs/.vuepress/sidebar/zh-CN/docs.js   |   1 +
 docs/en/docs/admin-manual/tracing.md   |  80 +
 docs/zh-CN/docs/admin-manual/tracing.md|  82 ++
 fe/fe-core/pom.xml |  26 +
 .../src/main/java/org/apache/doris/PaloFe.java |   3 +
 .../main/java/org/apache/doris/common/Config.java  |   6 +
 .../apache/doris/common/telemetry/ScopedSpan.java  |  38 +--
 .../apache/doris/common/telemetry/Telemetry.java   | 101 +
 .../java/org/apache/doris/qe/ConnectContext.java   |  12 ++
 .../java/org/apache/doris/qe/ConnectProcessor.java |  13 ++-
 .../main/java/org/apache/doris/qe/Coordinator.java |  69 +---
 .../java/org/apache/doris/qe/StmtExecutor.java | 125 ++---
 .../org/apache/doris/rpc/BackendServiceClient.java |  32 +-
 61 files changed, 1079 insertions(+), 130 deletions(-)

diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index aa894e19a5..f638046a55 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -328,6 +328,27 @@ set_target_properties(minizip PROPERTIES IMPORTED_LOCATION 
${T

[GitHub] [doris] morningman merged pull request #10620: [Enhancement][multi-catalog]Impl parallel for file scanner to improve the scanner performance

[GitBox]

morningman merged PR #10620:
URL: https://github.com/apache/doris/pull/10620


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [improvement](multi-catalog) Impl parallel for file scanner to improve the scanner performance (#10620)

[morningman]

This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 24d824a783 [improvement](multi-catalog) Impl parallel for file scanner 
to improve the scanner performance (#10620)
24d824a783 is described below

commit 24d824a78390e88a63bb448e75c0c7fd1aa5c910
Author: huangzhaowei 
AuthorDate: Sat Jul 9 15:52:53 2022 +0800

[improvement](multi-catalog) Impl parallel for file scanner to improve the 
scanner performance (#10620)

Add multi-thread support in FileScanNode on be and impl the file spilt 
logic in fe.
---
 be/src/vec/exec/file_arrow_scanner.cpp |  2 +-
 be/src/vec/exec/file_scan_node.cpp | 31 ++--
 be/src/vec/exec/file_scan_node.h   | 12 +-
 .../doris/catalog/external/HMSExternalTable.java   |  4 +-
 .../main/java/org/apache/doris/common/Config.java  |  6 +++
 .../planner/external/ExternalFileScanNode.java | 43 +-
 .../planner/external/ExternalHudiScanProvider.java |  2 +-
 .../main/java/org/apache/doris/qe/Coordinator.java |  2 +-
 8 files changed, 81 insertions(+), 21 deletions(-)

diff --git a/be/src/vec/exec/file_arrow_scanner.cpp 
b/be/src/vec/exec/file_arrow_scanner.cpp
index 9adad2d71a..88ebca2bb0 100644
--- a/be/src/vec/exec/file_arrow_scanner.cpp
+++ b/be/src/vec/exec/file_arrow_scanner.cpp
@@ -37,7 +37,7 @@ FileArrowScanner::FileArrowScanner(RuntimeState* state, 
RuntimeProfile* profile,
   _arrow_batch_cur_idx(0) {}
 
 FileArrowScanner::~FileArrowScanner() {
-close();
+FileArrowScanner::close();
 }
 
 Status FileArrowScanner::_open_next_reader() {
diff --git a/be/src/vec/exec/file_scan_node.cpp 
b/be/src/vec/exec/file_scan_node.cpp
index 741b66dd81..ff4989a033 100644
--- a/be/src/vec/exec/file_scan_node.cpp
+++ b/be/src/vec/exec/file_scan_node.cpp
@@ -96,9 +96,14 @@ Status FileScanNode::open(RuntimeState* state) {
 Status FileScanNode::start_scanners() {
 {
 std::unique_lock l(_batch_queue_lock);
-_num_running_scanners = 1;
+_num_running_scanners = _scan_ranges.size();
+}
+
+_scanners_status.resize(_scan_ranges.size());
+for (int i = 0; i < _scan_ranges.size(); i++) {
+_scanner_threads.emplace_back(&FileScanNode::scanner_worker, this, i, 
_scan_ranges.size(),
+  std::ref(_scanners_status[i]));
 }
-_scanner_threads.emplace_back(&FileScanNode::scanner_worker, this, 0, 
_scan_ranges.size());
 return Status::OK();
 }
 
@@ -203,7 +208,10 @@ Status FileScanNode::close(RuntimeState* state) {
 for (int i = 0; i < _scanner_threads.size(); ++i) {
 _scanner_threads[i].join();
 }
-
+for (int i = 0; i < _scanners_status.size(); i++) {
+std::future f = _scanners_status[i].get_future();
+RETURN_IF_ERROR(f.get());
+}
 // Close
 _batch_queue.clear();
 return ExecNode::close(state);
@@ -264,18 +272,16 @@ Status FileScanNode::scanner_scan(const TFileScanRange& 
scan_range, ScannerCount
 return Status::OK();
 }
 
-void FileScanNode::scanner_worker(int start_idx, int length) {
+void FileScanNode::scanner_worker(int start_idx, int length, 
std::promise& p_status) {
 Thread::set_self_name("file_scanner");
 Status status = Status::OK();
 ScannerCounter counter;
-for (int i = 0; i < length && status.ok(); ++i) {
-const TFileScanRange& scan_range =
-_scan_ranges[start_idx + 
i].scan_range.ext_scan_range.file_scan_range;
-status = scanner_scan(scan_range, &counter);
-if (!status.ok()) {
-LOG(WARNING) << "Scanner[" << start_idx + i
- << "] process failed. status=" << 
status.get_error_msg();
-}
+const TFileScanRange& scan_range =
+_scan_ranges[start_idx].scan_range.ext_scan_range.file_scan_range;
+status = scanner_scan(scan_range, &counter);
+if (!status.ok()) {
+LOG(WARNING) << "Scanner[" << start_idx
+ << "] process failed. status=" << status.get_error_msg();
 }
 
 // Update stats
@@ -296,6 +302,7 @@ void FileScanNode::scanner_worker(int start_idx, int 
length) {
 if (!status.ok()) {
 _queue_writer_cond.notify_all();
 }
+p_status.set_value(status);
 }
 
 std::unique_ptr FileScanNode::create_scanner(const 
TFileScanRange& scan_range,
diff --git a/be/src/vec/exec/file_scan_node.h b/be/src/vec/exec/file_scan_node.h
index 5106d654cc..2d7deb9626 100644
--- a/be/src/vec/exec/file_scan_node.h
+++ b/be/src/vec/exec/file_scan_node.h
@@ -17,7 +17,14 @@
 
 #pragma once
 
-#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
 
 #include "common/status.h"
 #include "exec/base_scanner.h"
@@ -77,7 +84,7 @@ private:
 
 Status start_scanners();
 
-void sca

[GitHub] [doris] BiteTheDDDDt merged pull request #10467: [Bug][Vectorized] Support array function in where pre in volap_scan_node

[GitBox]

BiteThet merged PR #10467:
URL: https://github.com/apache/doris/pull/10467


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [Bug][Vectorized] Support array function in where pre in volap_scan_node (#10467)

[panxiaolei]

This is an automated email from the ASF dual-hosted git repository.

panxiaolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 1f08f2d144 [Bug][Vectorized] Support array function in where pre in 
volap_scan_node (#10467)
1f08f2d144 is described below

commit 1f08f2d14489d0fe6f39d2b14e7057ad1909041a
Author: HappenLee 
AuthorDate: Sat Jul 9 16:22:01 2022 +0800

[Bug][Vectorized] Support array function in where pre in volap_scan_node 
(#10467)

Co-authored-by: lihaopeng 
Support array function in where pre in volap_scan_node
---
 be/src/exprs/array_functions.cpp   | 4 
 be/src/exprs/array_functions.h | 2 ++
 fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java | 5 +
 3 files changed, 11 insertions(+)

diff --git a/be/src/exprs/array_functions.cpp b/be/src/exprs/array_functions.cpp
index 32a1639af7..1bb2a59042 100644
--- a/be/src/exprs/array_functions.cpp
+++ b/be/src/exprs/array_functions.cpp
@@ -41,4 +41,8 @@ void ArrayFunctions::init() {}
 ARRAY_FUNCTION(IntVal, TYPE_INT);
 ARRAY_FUNCTION(StringVal, TYPE_VARCHAR);
 
+doris_udf::AnyVal array_fake_function(FunctionContext* context) {
+return doris_udf::AnyVal(true);
+}
+
 } // namespace doris
diff --git a/be/src/exprs/array_functions.h b/be/src/exprs/array_functions.h
index 7bceebf429..292e984934 100644
--- a/be/src/exprs/array_functions.h
+++ b/be/src/exprs/array_functions.h
@@ -32,4 +32,6 @@ public:
 
 static CollectionVal array(FunctionContext* context, int num_children, 
const StringVal* values);
 };
+
+doris_udf::AnyVal array_fake_function(FunctionContext* context);
 } // namespace doris
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java 
b/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java
index c03722b0fa..13a4d938ea 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java
@@ -1265,6 +1265,11 @@ public class FunctionSet {
   boolean varArgs, Type ... args) {
 ArrayList argsType = new ArrayList();
 for (Type type : args) {
+// only to prevent olap scan node use array expr to find a fake 
symbol
+// TODO: delete the code after we remove origin exec engine
+if (type.isArrayType()) {
+symbol = 
"_ZN5doris19array_fake_functionEPN9doris_udf15FunctionContextE";
+}
 argsType.add(type);
 }
 addBuiltinBothScalaAndVectorized(ScalarFunction.createBuiltin(


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] BiteTheDDDDt closed pull request #10388: [feature-wip] (array-type) add the array_distinct function

[GitBox]

BiteThet closed pull request #10388: [feature-wip] (array-type) add the 
array_distinct function
URL: https://github.com/apache/doris/pull/10388


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] Gabriel39 opened a new issue, #10715: [Bug] Wrong results when using in runtime filter

[GitBox]

Gabriel39 opened a new issue, #10715:
URL: https://github.com/apache/doris/issues/10715

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Version
   
   master
   
   ### What's Wrong?
   
   When I set runtime_filter_type=1 which means IN runtime filter is forced to 
used, all data will be filtered in some cases
   
   ### What You Expected?
   
   get correct result
   
   ### How to Reproduce?
   
   TPCDS 1g
   
   WITH
 ws_wh AS (
  SELECT
ws1.ws_order_number
  , ws1.ws_warehouse_sk wh1
  , ws2.ws_warehouse_sk wh2
  FROM
web_sales ws1
  , web_sales ws2
  WHERE (ws1.ws_order_number = ws2.ws_order_number)
 AND (ws1.ws_warehouse_sk <> ws2.ws_warehouse_sk)
   )
   SELECT count(wr_order_number)
  FROM
web_returns
  , ws_wh
  WHERE (wr_order_number = ws_wh.ws_order_number);
   
   ### Anything Else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] Gabriel39 opened a new pull request, #10716: [BUG] runtime filter core dump

[GitBox]

Gabriel39 opened a new pull request, #10716:
URL: https://github.com/apache/doris/pull/10716

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem Summary:
   
   Describe the overview of changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] wangbo opened a new pull request, #10717: [refactor] remove PredicateColumn

[GitBox]

wangbo opened a new pull request, #10717:
URL: https://github.com/apache/doris/pull/10717

   # Proposed changes
   
   remove PredicateColumn
   
   ## Problem Summary:
   
   Describe the overview of changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] caiconghui opened a new issue, #10718: [Enhancement] (alter) Increase the success rate of alter job by ignoring some task failure

[GitBox]

caiconghui opened a new issue, #10718:
URL: https://github.com/apache/doris/issues/10718

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Description
   
   now, when we do some alter operation on big table, it is easy to see some 
task failure, but we shouldn't 
   simply mark the job as failed, we allow some task failure because of 
multiple replicas for one tablet.
   
   ### Solution
   
   change the logic of marking failure of alter job
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] caiconghui opened a new pull request, #10719: [enhancement](alter) Make alter job more robust by ignoring some task failure

[GitBox]

caiconghui opened a new pull request, #10719:
URL: https://github.com/apache/doris/pull/10719

   # Proposed changes
   
   Issue Number: close #10718 
   
   ## Problem Summary:
   
   Describe the overview of changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-spark-connector] caiconghui merged pull request #41: [Enhancement] Add param max_filter_ratio for sink of spark connector

[GitBox]

caiconghui merged PR #41:
URL: https://github.com/apache/doris-spark-connector/pull/41


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris-spark-connector] branch master updated: [Enhancement] Add param max_filter_ratio for sink of spark connector (#41)

[caiconghui]

This is an automated email from the ASF dual-hosted git repository.

caiconghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-spark-connector.git


The following commit(s) were added to refs/heads/master by this push:
 new 41b612d  [Enhancement] Add param max_filter_ratio for sink of spark 
connector (#41)
41b612d is described below

commit 41b612d9d5fb359f6ef55c092b8f6479973fd1d3
Author: zhenhb <894672...@qq.com>
AuthorDate: Sat Jul 9 18:07:20 2022 +0800

[Enhancement] Add param max_filter_ratio for sink of spark connector (#41)
---
 .../main/java/org/apache/doris/spark/DorisStreamLoad.java   | 13 +
 .../org/apache/doris/spark/cfg/ConfigurationOptions.java|  2 ++
 2 files changed, 15 insertions(+)

diff --git 
a/spark-doris-connector/src/main/java/org/apache/doris/spark/DorisStreamLoad.java
 
b/spark-doris-connector/src/main/java/org/apache/doris/spark/DorisStreamLoad.java
index db3ef7d..be3c938 100644
--- 
a/spark-doris-connector/src/main/java/org/apache/doris/spark/DorisStreamLoad.java
+++ 
b/spark-doris-connector/src/main/java/org/apache/doris/spark/DorisStreamLoad.java
@@ -68,6 +68,7 @@ public class DorisStreamLoad implements Serializable{
 private String authEncoding;
 private String columns;
 private String[] dfColumns;
+private String maxFilterRatio;
 
 public DorisStreamLoad(String hostPort, String db, String tbl, String 
user, String passwd) {
 this.hostPort = hostPort;
@@ -90,6 +91,9 @@ public class DorisStreamLoad implements Serializable{
 this.loadUrlStr = String.format(loadUrlPattern, hostPort, db, tbl);
 this.authEncoding = 
Base64.getEncoder().encodeToString(String.format("%s:%s", user, 
passwd).getBytes(StandardCharsets.UTF_8));
 this.columns = 
settings.getProperty(ConfigurationOptions.DORIS_WRITE_FIELDS);
+
+this.maxFilterRatio = 
settings.getProperty(ConfigurationOptions.DORIS_MAX_FILTER_RATIO);
+
 }
 
 public DorisStreamLoad(SparkSettings settings, String[] dfColumns) throws 
IOException, DorisException {
@@ -100,10 +104,14 @@ public class DorisStreamLoad implements Serializable{
 this.tbl = dbTable[1];
 this.user = 
settings.getProperty(ConfigurationOptions.DORIS_REQUEST_AUTH_USER);
 this.passwd = 
settings.getProperty(ConfigurationOptions.DORIS_REQUEST_AUTH_PASSWORD);
+
+
 this.loadUrlStr = String.format(loadUrlPattern, hostPort, db, tbl);
 this.authEncoding = 
Base64.getEncoder().encodeToString(String.format("%s:%s", user, 
passwd).getBytes(StandardCharsets.UTF_8));
 this.columns = 
settings.getProperty(ConfigurationOptions.DORIS_WRITE_FIELDS);
 this.dfColumns = dfColumns;
+
+this.maxFilterRatio = 
settings.getProperty(ConfigurationOptions.DORIS_MAX_FILTER_RATIO);
 }
 
 public String getLoadUrlStr() {
@@ -132,6 +140,11 @@ public class DorisStreamLoad implements Serializable{
 if (columns != null && !columns.equals("")) {
 conn.addRequestProperty("columns", columns);
 }
+
+if (maxFilterRatio != null && !maxFilterRatio.equals("")) {
+conn.addRequestProperty("max_filter_ratio", maxFilterRatio);
+}
+
 conn.setDoOutput(true);
 conn.setDoInput(true);
 conn.addRequestProperty("format", "json");
diff --git 
a/spark-doris-connector/src/main/java/org/apache/doris/spark/cfg/ConfigurationOptions.java
 
b/spark-doris-connector/src/main/java/org/apache/doris/spark/cfg/ConfigurationOptions.java
index 9a0cead..e3c55d6 100644
--- 
a/spark-doris-connector/src/main/java/org/apache/doris/spark/cfg/ConfigurationOptions.java
+++ 
b/spark-doris-connector/src/main/java/org/apache/doris/spark/cfg/ConfigurationOptions.java
@@ -71,4 +71,6 @@ public interface ConfigurationOptions {
 
 String DORIS_SINK_MAX_RETRIES = "doris.sink.max-retries";
 int SINK_MAX_RETRIES_DEFAULT = 1;
+
+String DORIS_MAX_FILTER_RATIO = "doris.max.filter.ratio";
 }


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] weizuo93 opened a new issue, #10720: [Bug] Segment files are removed as trash but tablet meta is normal

[GitBox]

weizuo93 opened a new issue, #10720:
URL: https://github.com/apache/doris/issues/10720

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Version
   
   trunk version. commit id : a5efda68829c0873800b62d7e2e2c3b1807d1734
   
   ### What's Wrong?
   
   We replaced the original disk on the BE node with a new disk by wrong 
operation when BE restarted. When we discovered the mistake and added the 
original disk. There is no unhealthy replica in the cluster after a period of 
time, and we removed the wrong disk. When query comes, exception information is 
thrown and the error code is `-3109` which means `failed to open segment`. We 
found that segment files for some tablets in this BE node had been removed as 
trash but tablet meta is normal in original disk. These abnormal tablets can 
not be detected and repaired by FE.
   
   ### What You Expected?
   
   Tablet metadata should be consistent with data files for a tablet. When 
segment files removed as trash, the tablet should be droped on the BE node so 
that FE node could detecte and repaire the error replica.
   
   ### How to Reproduce?
   
   Cluster: 1 FE + 3 BE (BE01, BE02 and BE03, there is one disk called `disk-1` 
on BE01.)
   
   STEP 1: create a table on the cluster and ensure there are 3 replica for 
each tablet.
   
   STEP 2: insert data into the table.
   
   STEP 3: remove the `disk-1` on BE01 and add a new disk called `disk-2`, then 
restart BE01.
   When the deamon start, we will find that there is no replica on 
BE01 because there is only one empty disk which is `disk-2`, and the replica 
repair task will clone some replica to the `disk-2` on BE01.
   
   STEP 4: When there is no unhealthy replica in the cluster after a period of 
time, add the `disk-1` and restart BE01(there is two disks which are `disk-1` 
and `disk-2`).
   When the deamon start, we will find tablets in `disk-2` would be 
load and tablets which hold the same id with that on `disk-2` will not be load. 
Data on different disks are loaded in parallel. If the later loaded tablet on 
`disk-1`(there is a tablet with same id on `disk-2` has been loaded before 
successfully), the tablet will not be loaded successfully and segment files 
would be removed as trash but metadata is normal.
   
   STEP 5: remove the `disk-2` on BE01, keep `disk-1` on BE01, then restart 
BE01.
  When the deamon start, we will find tablets in `disk-1` would be 
load. These tablets hold normal metadata but has no segment files.
   
   STEP 6: query the table. If the query falls on these replica on BE01, an 
exception will occur and the error code is `-3109` which means `failed to open 
segment`. These abnormal tablets can not be detected and repaired by FE.
   
   
   
   ### Anything Else?
   
   NO.
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] weizuo93 opened a new pull request, #10721: [Bug] Fix bug that segments are removed as trash but tablet meta is normal

[GitBox]

weizuo93 opened a new pull request, #10721:
URL: https://github.com/apache/doris/pull/10721

   # Proposed changes
   
   Issue Number: close #10720 
   
   ## Problem Summary:
   
   We replaced the original disk on the BE node with a new disk by wrong 
operation when BE restarted. When we discovered the mistake and added the 
original disk. There is no unhealthy replica in the cluster after a period of 
time, and we removed the wrong disk. When query comes, exception information is 
thrown and the error code is `-3109` which means `failed to open segment`. We 
found that segment files for some tablets had been removed as trash but tablet 
meta is normal in original disk. These abnormal tablets can not be detected and 
repaired by FE.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (No)
   2. Has unit tests been added: (No Need)
   3. Has document been added or modified: (No Need)
   4. Does it need to update dependencies: (No)
   5. Are there any changes that cannot be rolled back: (No)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yiguolei merged pull request #10716: [BUG] runtime filter core dump

[GitBox]

yiguolei merged PR #10716:
URL: https://github.com/apache/doris/pull/10716


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [BUG] runtime filter core dump (#10716)

[yiguolei]

This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 7f9eeb8fc3 [BUG] runtime filter core dump (#10716)
7f9eeb8fc3 is described below

commit 7f9eeb8fc31550e2406251fd9d5ba57bfad89fb5
Author: Gabriel 
AuthorDate: Sat Jul 9 21:36:22 2022 +0800

[BUG] runtime filter core dump (#10716)
---
 be/src/vec/exec/volap_scan_node.cpp | 4 
 1 file changed, 4 insertions(+)

diff --git a/be/src/vec/exec/volap_scan_node.cpp 
b/be/src/vec/exec/volap_scan_node.cpp
index fc3cb0f59f..fd77f50ff7 100644
--- a/be/src/vec/exec/volap_scan_node.cpp
+++ b/be/src/vec/exec/volap_scan_node.cpp
@@ -482,6 +482,10 @@ void VOlapScanNode::scanner_thread(VOlapScanner* scanner) {
 }
 
 if (!vexprs.empty()) {
+if (*scanner->vconjunct_ctx_ptr()) {
+(*scanner->vconjunct_ctx_ptr())->close(state);
+*scanner->vconjunct_ctx_ptr() = nullptr;
+}
 WARN_IF_ERROR((*_vconjunct_ctx_ptr)->clone(state, 
scanner->vconjunct_ctx_ptr()),
   "Something wrong for runtime filters: ");
 scanner->set_use_pushdown_conjuncts(true);


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] HappenLee opened a new pull request, #10722: [bug-fix] fix the bug join node insert null column to not null column

[GitBox]

HappenLee opened a new pull request, #10722:
URL: https://github.com/apache/doris/pull/10722

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem Summary:
   
   Describe the overview of changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch dev-1.0.1-v20220709 updated: [tmpfix] join node nullable to nonnull

[morningman]

This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch dev-1.0.1-v20220709
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/dev-1.0.1-v20220709 by this 
push:
 new ae1f42fad7 [tmpfix] join node nullable to nonnull
ae1f42fad7 is described below

commit ae1f42fad734f6987be5422edf15e284e1a82d6e
Author: morningman 
AuthorDate: Sat Jul 9 22:17:35 2022 +0800

[tmpfix] join node nullable to nonnull
---
 be/src/vec/exec/join/vhash_join_node.cpp | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/be/src/vec/exec/join/vhash_join_node.cpp 
b/be/src/vec/exec/join/vhash_join_node.cpp
index 25be6c80a9..72af7c29b3 100644
--- a/be/src/vec/exec/join/vhash_join_node.cpp
+++ b/be/src/vec/exec/join/vhash_join_node.cpp
@@ -1368,10 +1368,14 @@ Status HashJoinNode::_build_output_block(Block* 
origin_block, Block* output_bloc
 // we should repalce `insert_column_datas` by `insert_range_from`
 
 auto insert_column_datas = [](auto& to, const auto& from, size_t rows) {
-if (to->is_nullable() && !from.is_nullable()) {
+auto [to_null, from_null] = std::pair {to->is_nullable(), 
from.is_nullable()};
+if (to_null && !from_null) {
 auto& null_column = reinterpret_cast(*to);
 null_column.get_nested_column().insert_range_from(from, 0, rows);
 null_column.get_null_map_column().get_data().resize_fill(rows, 0);
+} else if (!to_null && from_null) {
+const auto& null_column = reinterpret_cast(from);
+to->insert_range_from(null_column.get_nested_column(), 0, rows);
 } else {
 to->insert_range_from(from, 0, rows);
 }


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] SaintBacchus opened a new issue, #10723: [Bug] [Multi-Catalog] Failed to use hms database when there are some presto views in hms.

[GitBox]

SaintBacchus opened a new issue, #10723:
URL: https://github.com/apache/doris/issues/10723

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Version
   
   master branch
   
   ### What's Wrong?
   
   When create some presto view in a hive datasource and then use doris hms 
table to show the tables, I will get these error
   ```
   2022-07-09 16:46:24,893 WARN (doris-mysql-nio-pool-2|261) 
[ReadListener.lambda$handleEvent$0():60] Exception happened in one 
session([remote ip: 10.248.181.219]).
   java.lang.NullPointerException: null
   at 
org.apache.doris.catalog.external.HMSExternalTable.init(HMSExternalTable.java:90)
 ~[doris-fe.jar:1.0-SNAPSHOT]
   at 
org.apache.doris.catalog.external.HMSExternalTable.makeSureInitialized(HMSExternalTable.java:72)
 ~[doris-fe.jar:1.0-SNAPSHOT]
   at 
org.apache.doris.catalog.external.HMSExternalTable.getFullSchema(HMSExternalTable.java:147)
 ~[doris-fe.jar:1.0-SNAPSHOT]
   at 
org.apache.doris.catalog.external.HMSExternalTable.getBaseSchema(HMSExternalTable.java:153)
 ~[doris-fe.jar:1.0-SNAPSHOT]
   at 
org.apache.doris.qe.ConnectProcessor.handleFieldList(ConnectProcessor.java:324) 
~[doris-fe.jar:1.0-SNAPSHOT]
   at 
org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:369) 
~[doris-fe.jar:1.0-SNAPSHOT]
   at 
org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:555) 
~[doris-fe.jar:1.0-SNAPSHOT]
   at 
org.apache.doris.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:52)
 ~[doris-fe.jar:1.0-SNAPSHOT]
   at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
~[?:?]
   at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
~[?:?]
   at java.lang.Thread.run(Thread.java:834) ~[?:?]
   ```
   
   The code 
[remoteTable.getSd().getInputFormat()](https://github.com/apache/doris/blob/7f9eeb8fc31550e2406251fd9d5ba57bfad89fb5/fe/fe-core/src/main/java/org/apache/doris/catalog/external/HMSExternalTable.java#L90)
 will be null when the hms table is a view. 
   
   
   ### What You Expected?
   
   the current datasource will be the dest hive database. 
   
   ### How to Reproduce?
   
   _No response_
   
   ### Anything Else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morningman commented on issue #9594: [Bug] PAUSE SYNC JOB 卡死

[GitBox]

morningman commented on issue #9594:
URL: https://github.com/apache/doris/issues/9594#issuecomment-1179559887

   > 要是如你说，这么简单的问题，我还有必要来这里提issues?
   
   请注意你的言辞！开源社区没有任何人有义务解决你的问题！


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morningman closed issue #9594: [Bug] PAUSE SYNC JOB 卡死

[GitBox]

morningman closed issue #9594: [Bug] PAUSE SYNC JOB   卡死
URL: https://github.com/apache/doris/issues/9594


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] SaintBacchus opened a new pull request, #10724: [MultiCatalog]Add strong checker for hms table

[GitBox]

SaintBacchus opened a new pull request, #10724:
URL: https://github.com/apache/doris/pull/10724

   # Proposed changes
   
   Issue Number: close #10723 
   
   ## Problem Summary:
   Add a strong checker for the hms table. Now we only support a small subset 
of the hive table, so we need intercept the execution of this nonsupport table 
in the analyse phase.
   
   Also this modify will fix issue #10723 
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] SaintBacchus commented on pull request #10724: [enhancement][MultiCatalog]Add strong checker for hms table

[GitBox]

SaintBacchus commented on PR #10724:
URL: https://github.com/apache/doris/pull/10724#issuecomment-1179560504

   Hudi and Iceberg also have some nonsupport case, I will add it tomorrow


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-flink-connector] morningman opened a new pull request, #46: [TLP] remove incubator

[GitBox]

morningman opened a new pull request, #46:
URL: https://github.com/apache/doris-flink-connector/pull/46

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem Summary:
   
   Describe the overview of changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-spark-connector] morningman opened a new pull request, #43: [TLP] remove incubator

[GitBox]

morningman opened a new pull request, #43:
URL: https://github.com/apache/doris-spark-connector/pull/43

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem Summary:
   
   Describe the overview of changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-manager] morningman opened a new pull request, #62: [TLP] remove incubator

[GitBox]

morningman opened a new pull request, #62:
URL: https://github.com/apache/doris-manager/pull/62

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem Summary:
   
   Describe the overview of changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris-website] branch master updated: [TLP] remove incubator

[morningman]

This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git


The following commit(s) were added to refs/heads/master by this push:
 new 7d35327161 [TLP] remove incubator
7d35327161 is described below

commit 7d353271615e506de1a3b552a0cfbd36633f2b4f
Author: morningman 
AuthorDate: Sun Jul 10 00:11:03 2022 +0800

[TLP] remove incubator
---
 .asf.yaml | 45 +
 README.md |  8 
 2 files changed, 49 insertions(+), 4 deletions(-)

diff --git a/.asf.yaml b/.asf.yaml
new file mode 100644
index 00..70d2e1d2e3
--- /dev/null
+++ b/.asf.yaml
@@ -0,0 +1,45 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+github:
+  description: Apache Doris Website
+  homepage: https://doris.apache.org/
+  labels:
+- data-warehousing
+- mpp
+- olap
+- dbms
+- apache
+- doris
+- database
+- distributed-system
+- big-data
+- iceberg
+- hudi
+- datalake
+- hive
+- hadoop
+- analytics
+- tpch
+- ssb
+- vectorized
+  enabled_merge_buttons:
+squash:  true
+merge:   false
+rebase:  false
+  notifications:
+pullrequests_status:  commits@doris.apache.org
diff --git a/README.md b/README.md
index 6f59267744..ac6c9ad9cf 100644
--- a/README.md
+++ b/README.md
@@ -19,9 +19,9 @@ under the License.
 
 # Doris document website
 
-This repo is for [Apache Doris(Incubating) website](https://doris.apache.org)
+This repo is for [Apache Doris Website](https://doris.apache.org)
 
-And it use Github Action to automatically sync content from [Apache 
Doris(Incubating) Code Repo](https://github.com/apache/incubator-doris)
+And it use Github Action to automatically sync content from [Apache Doris Code 
Repo](https://github.com/apache/doris)
 
 There are 2 Github Actions:
 
@@ -38,7 +38,7 @@ There are 2 Github Actions:
 To view the website, navigate to 
 [https://doris.apache.org](https://doris.apache.org) 
 or
-[https://doris.incubator.apache.org](https://doris.incubator.apache.org)
+[https://doris.apache.org](https://doris.apache.org)
 
 ## How to share a blog
 
@@ -167,6 +167,6 @@ Put the 1.0 version sidebar js files into the sidebar/en/ 
and sidebar/zh-CN/.
 
 ## About Doris
 
-For more details about Doris, refer to 
[incubator-doris](https://github.com/apache/incubator-doris/blob/master/docs/README.md).
+For more details about Doris, refer to 
[doris](https://github.com/apache/doris/blob/master/docs/README.md).
 
 


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] Kikyou1997 opened a new issue, #10725: [Bug] Get wrong results of TPC-DS SQL

[GitBox]

Kikyou1997 opened a new issue, #10725:
URL: https://github.com/apache/doris/issues/10725

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Version
   
   master
   
   ### What's Wrong?
   
   In TPC-DS 1G, we run q95, it would return 
   
   ```
   0, NULL, NULL
   ```
   
   
   
   ### What You Expected?
   
   Expected results is
   
   ```
   | order count | total shipping cost | total net profit |
   +-+-+--+
   |  73 |   120440.34 | 42133.12 |
   +-+-+--+
   ```
   
   ### How to Reproduce?
   
   Just  install the doris with master, make sure the `enable_projection` 
session variable is true. Import the TPC-DS data, and run the sql mentioned 
above.
   
   Then, you would get a wrong result set.
   
   ### Anything Else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] Kikyou1997 opened a new pull request, #10726: [bugfix] Fix the unexpected concurrent modification to the block returned by VOlapScanNode

[GitBox]

Kikyou1997 opened a new pull request, #10726:
URL: https://github.com/apache/doris/pull/10726

   # Proposed changes
   
   Issue Number: #10725
   
   ## Problem Summary:
   
   I think the bug is caused by  the modification of block introduced by column 
prune feature without the necessary sync within multi-thread, so i just moved 
the those code to the sychronized block.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/)
   2. Has unit tests been added: (No)
   3. Has document been added or modified: (No Need)
   4. Does it need to update dependencies: (No)
   5. Are there any changes that cannot be rolled back: (No)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] Kikyou1997 closed pull request #10726: [bugfix] Fix the unexpected concurrent modification to the block returned by VOlapScanNode

[GitBox]

Kikyou1997 closed pull request #10726: [bugfix] Fix the unexpected concurrent 
modification to the block returned by VOlapScanNode
URL: https://github.com/apache/doris/pull/10726


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris-flink-connector] branch master updated: [TLP] remove incubator (#46)

[jiafengzheng]

This is an automated email from the ASF dual-hosted git repository.

jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-flink-connector.git


The following commit(s) were added to refs/heads/master by this push:
 new b010851  [TLP] remove incubator (#46)
b010851 is described below

commit b01085151b1757202d37e53fd7d41d948633753d
Author: Mingyu Chen 
AuthorDate: Sun Jul 10 08:43:48 2022 +0800

[TLP] remove incubator (#46)

remove incubator
---
 .asf.yaml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/.asf.yaml b/.asf.yaml
index e4438c3..15d373d 100644
--- a/.asf.yaml
+++ b/.asf.yaml
@@ -16,7 +16,7 @@
 #
 
 github:
-  description: Flink Connector for Apache Doris(incubating)
+  description: Flink Connector for Apache Doris
   homepage: https://doris.apache.org/
   labels:
 - data-warehousing
@@ -26,6 +26,7 @@ github:
 - apache
 - doris
 - flink
+- connector
   enabled_merge_buttons:
 squash:  true
 merge:   false


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-spark-connector] hf200012 merged pull request #43: [TLP] remove incubator

[GitBox]

hf200012 merged PR #43:
URL: https://github.com/apache/doris-spark-connector/pull/43


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-flink-connector] hf200012 merged pull request #46: [TLP] remove incubator

[GitBox]

hf200012 merged PR #46:
URL: https://github.com/apache/doris-flink-connector/pull/46


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris-spark-connector] branch master updated: [TLP] remove incubator (#43)

[jiafengzheng]

This is an automated email from the ASF dual-hosted git repository.

jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-spark-connector.git


The following commit(s) were added to refs/heads/master by this push:
 new 798df56  [TLP] remove incubator (#43)
798df56 is described below

commit 798df56d922c6b1e92c32b957bd1f5f76ee5ca54
Author: Mingyu Chen 
AuthorDate: Sun Jul 10 08:44:10 2022 +0800

[TLP] remove incubator (#43)

remove incubator
---
 .asf.yaml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/.asf.yaml b/.asf.yaml
index 647b036..9c1a1b4 100644
--- a/.asf.yaml
+++ b/.asf.yaml
@@ -16,7 +16,7 @@
 #
 
 github:
-  description: Spark Connector for Apache Doris(incubating)
+  description: Spark Connector for Apache Doris
   homepage: https://doris.apache.org/
   labels:
 - data-warehousing
@@ -26,6 +26,7 @@ github:
 - apache
 - doris
 - spark
+- connector
   enabled_merge_buttons:
 squash:  true
 merge:   false


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-manager] hf200012 merged pull request #62: [TLP] remove incubator

[GitBox]

hf200012 merged PR #62:
URL: https://github.com/apache/doris-manager/pull/62


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris-manager] branch master updated: [TLP] remove incubator (#62)

[jiafengzheng]

This is an automated email from the ASF dual-hosted git repository.

jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-manager.git


The following commit(s) were added to refs/heads/master by this push:
 new a4cee39  [TLP] remove incubator (#62)
a4cee39 is described below

commit a4cee39339f93bce804c06be10027850d89ecd7c
Author: Mingyu Chen 
AuthorDate: Sun Jul 10 08:44:31 2022 +0800

[TLP] remove incubator (#62)

remove incubator
---
 .asf.yaml | 2 +-
 README.md | 8 
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/.asf.yaml b/.asf.yaml
index 681abdc..d400cce 100644
--- a/.asf.yaml
+++ b/.asf.yaml
@@ -16,7 +16,7 @@
 #
 
 github:
-  description: Cluster manager for Apache Doris(Incubating)
+  description: Cluster manager for Apache Doris
   homepage: https://doris.apache.org/
   labels:
 - data-warehousing
diff --git a/README.md b/README.md
index edf5ab8..80a59c0 100644
--- a/README.md
+++ b/README.md
@@ -17,12 +17,12 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-# Apache Doris (incubating) Manager
+# Apache Doris Manager
 
 
[![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html)
 [![Join the Doris Community at 
Slack](https://img.shields.io/badge/chat-slack-brightgreen)](https://join.slack.com/t/apachedoriscommunity/shared_invite/zt-11jb8gesh-7IukzSrdea6mqoG0HB4gZg)
 
-The repository contains Manager for Apache Doris (incubating)
+The repository contains Manager for Apache Doris
 
 ## License
 
@@ -30,7 +30,7 @@ The repository contains Manager for Apache Doris (incubating)
 
 ## Report issues or submit pull request
 
-If you find any bugs, feel free to file a [GitHub 
issue](https://github.com/apache/incubator-doris/issues) or fix it by 
submitting a [pull request](https://github.com/apache/incubator-doris/pulls).
+If you find any bugs, feel free to file a [GitHub 
issue](https://github.com/apache/doris/issues) or fix it by submitting a [pull 
request](https://github.com/apache/doris/pulls).
 
 ## Contact Us
 
@@ -42,6 +42,6 @@ Contact us through the following mailing list.
 
 ## Links
 
-* Doris official site - 
+* Doris official site - 
 * Developer Mailing list - . Mail to 
, follow the reply to subscribe the mail list.
 * Slack channel - [Join the 
Slack](https://join.slack.com/t/apachedoriscommunity/shared_invite/zt-11jb8gesh-7IukzSrdea6mqoG0HB4gZg)


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] starocean999 opened a new pull request, #10727: [FIX] need check if aggregate_function_changed_flags is set by fe

[GitBox]

starocean999 opened a new pull request, #10727:
URL: https://github.com/apache/doris/pull/10727

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem Summary:
   
   aggregate_function_changed_flags is an optional field, be shoud check if it 
is set before using it
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yiguolei commented on a diff in pull request #10719: [enhancement](alter) Make alter job more robust by ignoring some task failure

[GitBox]

yiguolei commented on code in PR #10719:
URL: https://github.com/apache/doris/pull/10719#discussion_r917328443


##
fe/fe-core/src/main/java/org/apache/doris/alter/RollupJobV2.java:
##
@@ -444,6 +456,12 @@ protected void runRunningJob() throws AlterCancelException 
{
 tbl.writeLockOrAlterCancelException();
 try {
 Preconditions.checkState(tbl.getState() == OlapTableState.ROLLUP);
+TabletInvertedIndex invertedIndex = 
Catalog.getCurrentInvertedIndex();
+for (List tasks : failedAgentTasks.values()) {
+for (AgentTask task : tasks) {
+invertedIndex.getReplica(task.getTabletId(), 
task.getBackendId()).setBad(true);

Review Comment:
   This tablet id is new rollup tabeltid  or base tablet id?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] caiconghui commented on a diff in pull request #10719: [enhancement](alter) Make alter job more robust by ignoring some task failure

[GitBox]

caiconghui commented on code in PR #10719:
URL: https://github.com/apache/doris/pull/10719#discussion_r917328465


##
fe/fe-core/src/main/java/org/apache/doris/alter/RollupJobV2.java:
##
@@ -444,6 +456,12 @@ protected void runRunningJob() throws AlterCancelException 
{
 tbl.writeLockOrAlterCancelException();
 try {
 Preconditions.checkState(tbl.getState() == OlapTableState.ROLLUP);
+TabletInvertedIndex invertedIndex = 
Catalog.getCurrentInvertedIndex();
+for (List tasks : failedAgentTasks.values()) {
+for (AgentTask task : tasks) {
+invertedIndex.getReplica(task.getTabletId(), 
task.getBackendId()).setBad(true);

Review Comment:
   new rollup tablet id



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] 02/02: [hotfix](dev-1.0.1) support convert between nullable and non-nullable in join node

[morningman]

This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 2cb493947f7c1baae0290f068687633818b35639
Author: morningman 
AuthorDate: Sat Jul 9 22:17:35 2022 +0800

[hotfix](dev-1.0.1) support convert between nullable and non-nullable in 
join node

Authored by HappenLee #10721
---
 be/src/vec/exec/join/vhash_join_node.cpp | 6 +-
 be/src/vec/exec/vaggregation_node.cpp| 5 -
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/be/src/vec/exec/join/vhash_join_node.cpp 
b/be/src/vec/exec/join/vhash_join_node.cpp
index 25be6c80a9..72af7c29b3 100644
--- a/be/src/vec/exec/join/vhash_join_node.cpp
+++ b/be/src/vec/exec/join/vhash_join_node.cpp
@@ -1368,10 +1368,14 @@ Status HashJoinNode::_build_output_block(Block* 
origin_block, Block* output_bloc
 // we should repalce `insert_column_datas` by `insert_range_from`
 
 auto insert_column_datas = [](auto& to, const auto& from, size_t rows) {
-if (to->is_nullable() && !from.is_nullable()) {
+auto [to_null, from_null] = std::pair {to->is_nullable(), 
from.is_nullable()};
+if (to_null && !from_null) {
 auto& null_column = reinterpret_cast(*to);
 null_column.get_nested_column().insert_range_from(from, 0, rows);
 null_column.get_null_map_column().get_data().resize_fill(rows, 0);
+} else if (!to_null && from_null) {
+const auto& null_column = reinterpret_cast(from);
+to->insert_range_from(null_column.get_nested_column(), 0, rows);
 } else {
 to->insert_range_from(from, 0, rows);
 }
diff --git a/be/src/vec/exec/vaggregation_node.cpp 
b/be/src/vec/exec/vaggregation_node.cpp
index 4dd89499a1..2b09884575 100644
--- a/be/src/vec/exec/vaggregation_node.cpp
+++ b/be/src/vec/exec/vaggregation_node.cpp
@@ -77,7 +77,10 @@ static constexpr int STREAMING_HT_MIN_REDUCTION_SIZE =
 AggregationNode::AggregationNode(ObjectPool* pool, const TPlanNode& tnode,
  const DescriptorTbl& descs)
 : ExecNode(pool, tnode, descs),
-  
_aggregate_evaluators_changed_flags(tnode.agg_node.aggregate_function_changed_flags),
+  _aggregate_evaluators_changed_flags(
+  tnode.agg_node.__isset.aggregate_function_changed_flags
+  ? tnode.agg_node.aggregate_function_changed_flags
+  : std::vector {}),
   _intermediate_tuple_id(tnode.agg_node.intermediate_tuple_id),
   _intermediate_tuple_desc(NULL),
   _output_tuple_id(tnode.agg_node.output_tuple_id),


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] 01/02: [hotfix](dev-1.0.1) fix expr compose and replace bug

[morningman]

This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 853cc63bc9be84df41e99c4d8d625cb08a819fb1
Author: morningman 
AuthorDate: Sat Jul 9 12:39:01 2022 +0800

[hotfix](dev-1.0.1) fix expr compose and replace bug

Support a->fn(b) : b->c to a->fn(c)
---
 be/src/vec/exprs/vslot_ref.cpp |  5 +++--
 .../apache/doris/analysis/ExprSubstitutionMap.java | 23 +-
 .../org/apache/doris/planner/HashJoinNode.java |  2 +-
 3 files changed, 22 insertions(+), 8 deletions(-)

diff --git a/be/src/vec/exprs/vslot_ref.cpp b/be/src/vec/exprs/vslot_ref.cpp
index 57038eb63f..8cbc56cb37 100644
--- a/be/src/vec/exprs/vslot_ref.cpp
+++ b/be/src/vec/exprs/vslot_ref.cpp
@@ -20,6 +20,7 @@
 #include 
 
 #include "runtime/descriptors.h"
+#include "util/stack_util.h"
 
 namespace doris::vectorized {
 using doris::Status;
@@ -59,7 +60,7 @@ Status VSlotRef::prepare(doris::RuntimeState* state, const 
doris::RowDescriptor&
 }
 
 Status VSlotRef::execute(VExprContext* context, Block* block, int* 
result_column_id) {
-DCHECK_GE(_column_id, 0);
+CHECK_GE(_column_id, 0) << ", " << debug_string() << ", " << 
get_stack_trace();
 *result_column_id = _column_id;
 return Status::OK();
 }
@@ -69,7 +70,7 @@ const std::string& VSlotRef::expr_name() const {
 }
 std::string VSlotRef::debug_string() const {
 std::stringstream out;
-out << "SlotRef(slot_id=" << _slot_id << VExpr::debug_string() << ")";
+out << "SlotRef(slot_id=" << _slot_id << VExpr::debug_string() << ") 
column id: " << _column_id << ", name: " << *_column_name << ", is nulable: " 
<< _is_nullable;
 return out.str();
 }
 } // namespace doris::vectorized
diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/analysis/ExprSubstitutionMap.java 
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ExprSubstitutionMap.java
index e0df525ce5..6fe94f8578 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/analysis/ExprSubstitutionMap.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ExprSubstitutionMap.java
@@ -17,6 +17,8 @@
 
 package org.apache.doris.analysis;
 
+import org.apache.doris.common.AnalysisException;
+
 import java.util.List;
 import java.util.Objects;
 
@@ -189,7 +191,7 @@ public final class ExprSubstitutionMap {
  * f [A.id, B.id] [A.name, B.name] g [A.id, C.id] [A.age, C.age]
  * return: [A.id, C,id] [A.name, B.name] [A.age, C.age]
  */
-public static ExprSubstitutionMap composeAndReplace(ExprSubstitutionMap f, 
ExprSubstitutionMap g) {
+public static ExprSubstitutionMap composeAndReplace(ExprSubstitutionMap f, 
ExprSubstitutionMap g, Analyzer analyzer) throws AnalysisException {
 if (f == null && g == null) {
 return new ExprSubstitutionMap();
 }
@@ -200,11 +202,22 @@ public final class ExprSubstitutionMap {
 return f;
 }
 ExprSubstitutionMap result = new ExprSubstitutionMap();
-result = ExprSubstitutionMap.combine(result, g);
+// compose f and g
 for (int i = 0; i < g.size(); i++) {
-// case a->b, b->c => a->c
-if (f.mappingForRhsExpr(g.getLhs().get(i)) != null) {
-result.getLhs().set(i, f.mappingForRhsExpr(g.getLhs().get(i)));
+boolean findGMatch = false;
+Expr gLhs = g.getLhs().get(i);
+for (int j = 0; j < f.size(); j++) {
+// case a->fn(b), b->c => a->fn(c)
+Expr fRhs = f.getRhs().get(j);
+if (fRhs.contains(gLhs)) {
+Expr newRhs = fRhs.trySubstitute(g, analyzer, false);
+result.put(f.getLhs().get(j), newRhs);
+findGMatch = true;
+break;
+}
+}
+if (!findGMatch) {
+result.put(g.getLhs().get(i), g.getRhs().get(i));
 }
 }
 // add remaining f
diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java 
b/fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java
index 16e82b644f..f0c20ebec1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java
@@ -441,7 +441,7 @@ public class HashJoinNode extends PlanNode {
 }
 }
 // 4. change the outputSmap
-outputSmap = ExprSubstitutionMap.composeAndReplace(outputSmap, 
srcTblRefToOutputTupleSmap);
+outputSmap = ExprSubstitutionMap.composeAndReplace(outputSmap, 
srcTblRefToOutputTupleSmap, analyzer);
 }
 
 private void replaceOutputSmapForOuterJoin() {


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch dev-1.0.1 updated (c60ed8f18a -> 2cb493947f)

[morningman]

This is an automated email from the ASF dual-hosted git repository.

morningman pushed a change to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/doris.git


from c60ed8f18a [fix](optimizer) join reorder may cause column 
non-existence problem (#10670)
 new 853cc63bc9 [hotfix](dev-1.0.1) fix expr compose and replace bug
 new 2cb493947f [hotfix](dev-1.0.1) support convert between nullable and 
non-nullable in join node

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 be/src/vec/exec/join/vhash_join_node.cpp   |  6 +-
 be/src/vec/exec/vaggregation_node.cpp  |  5 -
 be/src/vec/exprs/vslot_ref.cpp |  5 +++--
 .../apache/doris/analysis/ExprSubstitutionMap.java | 23 +-
 .../org/apache/doris/planner/HashJoinNode.java |  2 +-
 5 files changed, 31 insertions(+), 10 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morningman closed pull request #10722: [bug-fix] fix the bug join node insert null column to not null column

[GitBox]

morningman closed pull request #10722: [bug-fix] fix the bug join node insert 
null column to not null column
URL: https://github.com/apache/doris/pull/10722


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morningman commented on pull request #10722: [bug-fix] fix the bug join node insert null column to not null column

[GitBox]

morningman commented on PR #10722:
URL: https://github.com/apache/doris/pull/10722#issuecomment-1179642083

   Manually merged to dev-1.0.1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morningman commented on pull request #10727: [FIX] need check if aggregate_function_changed_flags is set by fe

[GitBox]

morningman commented on PR #10727:
URL: https://github.com/apache/doris/pull/10727#issuecomment-1179642119

   Manually merged to dev-1.0.1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morningman closed pull request #10727: [FIX] need check if aggregate_function_changed_flags is set by fe

[GitBox]

morningman closed pull request #10727: [FIX] need check if 
aggregate_function_changed_flags is set by fe
URL: https://github.com/apache/doris/pull/10727


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yiguolei commented on a diff in pull request #10719: [enhancement](alter) Make alter job more robust by ignoring some task failure

[GitBox]

yiguolei commented on code in PR #10719:
URL: https://github.com/apache/doris/pull/10719#discussion_r917330602


##
fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java:
##
@@ -494,22 +492,38 @@ protected void runRunningJob() throws 
AlterCancelException {
 List tasks = 
schemaChangeBatchTask.getUnfinishedTasks(2000);
 for (AgentTask task : tasks) {
 if (task.getFailedTimes() >= 3) {
-throw new AlterCancelException("schema change task failed 
after try three times: "
-+ task.getErrorMsg());
+task.setFinished(true);
+AgentTaskQueue.removeTask(task.getBackendId(), 
TTaskType.ALTER, task.getSignature());
+LOG.warn("schema change task failed after try three times: 
" + task.getErrorMsg());
+if (!failedAgentTasks.containsKey(task.getTabletId())) {
+failedAgentTasks.put(task.getTabletId(), 
Lists.newArrayList(task));
+} else {
+failedAgentTasks.get(task.getTabletId()).add(task);
+}
+int expectSucceedTaskNum = tbl.getPartitionInfo()
+
.getReplicaAllocation(task.getPartitionId()).getTotalReplicaNum();
+int failedTaskCount = 
failedAgentTasks.get(task.getTabletId()).size();

Review Comment:
   why not use the replica num in config?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] caiconghui commented on a diff in pull request #10719: [enhancement](alter) Make alter job more robust by ignoring some task failure

[GitBox]

caiconghui commented on code in PR #10719:
URL: https://github.com/apache/doris/pull/10719#discussion_r917330814


##
fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java:
##
@@ -494,22 +492,38 @@ protected void runRunningJob() throws 
AlterCancelException {
 List tasks = 
schemaChangeBatchTask.getUnfinishedTasks(2000);
 for (AgentTask task : tasks) {
 if (task.getFailedTimes() >= 3) {
-throw new AlterCancelException("schema change task failed 
after try three times: "
-+ task.getErrorMsg());
+task.setFinished(true);
+AgentTaskQueue.removeTask(task.getBackendId(), 
TTaskType.ALTER, task.getSignature());
+LOG.warn("schema change task failed after try three times: 
" + task.getErrorMsg());
+if (!failedAgentTasks.containsKey(task.getTabletId())) {
+failedAgentTasks.put(task.getTabletId(), 
Lists.newArrayList(task));
+} else {
+failedAgentTasks.get(task.getTabletId()).add(task);
+}
+int expectSucceedTaskNum = tbl.getPartitionInfo()
+
.getReplicaAllocation(task.getPartitionId()).getTotalReplicaNum();
+int failedTaskCount = 
failedAgentTasks.get(task.getTabletId()).size();

Review Comment:
   we are not sure the status of all agent tasks, but we can confirm that 
replica with failed task is bad, the job is still in running state



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morningman commented on a diff in pull request #10721: [Bug] Fix bug that segments are removed as trash but tablet meta is normal

[GitBox]

morningman commented on code in PR #10721:
URL: https://github.com/apache/doris/pull/10721#discussion_r917331120


##
be/src/olap/tablet_manager.cpp:
##
@@ -154,13 +154,24 @@ Status TabletManager::_add_tablet_unlocked(TTabletId 
tablet_id, const TabletShar
 // During restore process, snapshot loader
 // replaced the old tablet's rowset with new rowsets, but the tablet path 
is reused, if drop files
 // here, the new rowset's file will also be dropped, so use keep files here
-bool keep_files = force ? true : false;
+bool keep_files = force;
 if (force ||
 (new_version > old_version || (new_version == old_version && new_time 
> old_time))) {
 // check if new tablet's meta is in store and add new tablet's meta to 
meta store
 res = _add_tablet_to_map_unlocked(tablet_id, tablet, update_meta, 
keep_files,
   true /*drop_old*/);
 } else {
+if (!keep_files) {
+tablet->set_tablet_state(TABLET_SHUTDOWN);

Review Comment:
   I am confused that, this is a "add tablet" method, but why we finally set 
tablet to `TABLET_SHUTDOWN`?
   Should it be `existed_tablet`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] dataalive closed issue #9711: [Bug] time_round function calculated wrong timestamp

[GitBox]

dataalive closed issue #9711: [Bug] time_round function calculated wrong 
timestamp
URL: https://github.com/apache/doris/issues/9711


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] dataalive commented on issue #9711: [Bug] time_round function calculated wrong timestamp

[GitBox]

dataalive commented on issue #9711:
URL: https://github.com/apache/doris/issues/9711#issuecomment-1179645002

   这个关联的PR： https://github.com/apache/doris/pull/9712 已经合入，这个Issue可以关闭了。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #10719: [enhancement](alter) Make alter job more robust by ignoring some task failure

[GitBox]

github-actions[bot] commented on PR #10719:
URL: https://github.com/apache/doris/pull/10719#issuecomment-1179648559

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #10719: [enhancement](alter) Make alter job more robust by ignoring some task failure

[GitBox]

github-actions[bot] commented on PR #10719:
URL: https://github.com/apache/doris/pull/10719#issuecomment-1179648563

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yiguolei merged pull request #10710: [improve](planner): split output expr to multiple line.

[GitBox]

yiguolei merged PR #10710:
URL: https://github.com/apache/doris/pull/10710


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [improve](planner): split output expr to multiple line. (#10710)

[yiguolei]

This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new a6e4c88663 [improve](planner): split output expr to multiple line. 
(#10710)
a6e4c88663 is described below

commit a6e4c8866356f0de1028815a4a47ef92e7e677e9
Author: jakevin 
AuthorDate: Sun Jul 10 11:35:48 2022 +0800

[improve](planner): split output expr to multiple line. (#10710)

* [improve](planner): split output expr to multiple line.

+---+
| Explain String|
+---+
| PLAN FRAGMENT 0   |
|   OUTPUT EXPRS:   |
|  `user_id`|
|  `default_cluster:test`.`tbl`.`date` |
|  `city`  |
|  `default_cluster:test`.`tbl`.`age`  |
+---+

* *: fix UT and regression-test.
---
 .../org/apache/doris/planner/PlanFragment.java |  4 +--
 .../apache/doris/analysis/StmtRewriterTest.java|  6 ++--
 .../org/apache/doris/planner/QueryPlanTest.java| 40 +++---
 .../doris/planner/TableFunctionPlanTest.java   |  2 +-
 .../data/performance/redundant_conjuncts.out   |  6 ++--
 regression-test/suites/demo/explain_action.groovy  |  4 +--
 6 files changed, 32 insertions(+), 30 deletions(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/planner/PlanFragment.java 
b/fe/fe-core/src/main/java/org/apache/doris/planner/PlanFragment.java
index 487240d226..e3fb413d1a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/PlanFragment.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/PlanFragment.java
@@ -296,8 +296,8 @@ public class PlanFragment extends TreeNode {
 StringBuilder str = new StringBuilder();
 Preconditions.checkState(dataPartition != null);
 if (CollectionUtils.isNotEmpty(outputExprs)) {
-str.append("  OUTPUT EXPRS:");
-
str.append(outputExprs.stream().map(Expr::toSql).collect(Collectors.joining(" | 
")));
+str.append("  OUTPUT EXPRS:\n");
+
str.append(outputExprs.stream().map(Expr::toSql).collect(Collectors.joining("\n 
   ")));
 }
 str.append("\n");
 str.append("  PARTITION: " + 
dataPartition.getExplainString(explainLevel) + "\n");
diff --git 
a/fe/fe-core/src/test/java/org/apache/doris/analysis/StmtRewriterTest.java 
b/fe/fe-core/src/test/java/org/apache/doris/analysis/StmtRewriterTest.java
index 7625921f23..73c18db8d0 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/analysis/StmtRewriterTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/analysis/StmtRewriterTest.java
@@ -380,7 +380,7 @@ public class StmtRewriterTest {
 "CROSS JOIN",
 "predicates:  sum(`salary`) >  avg(`salary`)",
 "order by:  `$a$1`.`$c$2` ASC",
-"OUTPUT EXPRS: `$a$1`.`$c$1`");
+"OUTPUT EXPRS:\n `$a$1`.`$c$1`");
 }
 
 /**
@@ -495,7 +495,7 @@ public class StmtRewriterTest {
 "CROSS JOIN",
 "predicates:  sum(`salary`) >  avg(`salary`)",
 "order by:  `$a$1`.`$c$2` ASC",
-"OUTPUT EXPRS: `$a$1`.`$c$1` |  
`$a$1`.`$c$2`");
+"OUTPUT EXPRS:\n `$a$1`.`$c$1`\n 
`$a$1`.`$c$2`");
 }
 
 /**
@@ -609,7 +609,7 @@ public class StmtRewriterTest {
 "CROSS JOIN",
 "predicates:  sum(`salary`) >  avg(`salary`)",
 "order by:  `$a$1`.`$c$2` ASC",
-"OUTPUT EXPRS: `$a$1`.`$c$1` |  
`$a$1`.`$c$2`");
+"OUTPUT EXPRS:\n `$a$1`.`$c$1`\n 
`$a$1`.`$c$2`");
 }
 
 /**
diff --git 
a/fe/fe-core/src/test/java/org/apache/doris/planner/QueryPlanTest.java 
b/fe/fe-core/src/test/java/org/apache/doris/planner/QueryPlanTest.java
index b8499836df..01e2527542 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/planner/QueryPlanTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/planner/QueryPlanTest.java
@@ -439,7 +439,7 @@ public class QueryPlanTest extends TestWithFeService {
 sql = "insert into test.bitmap_table select id, id2 from 
test.bitmap_table_2;";
 explainString = getSQLPlanOrErrorMsg("explain " + sql);
 Assert.assertTrue(explainString.contains("OLAP TABLE SINK"));
-Assert.assertTrue(explainString.contains("OUTPUT EXPRS:`id` | `id2`"));
+Assert.assertTrue(explainString.contains("OUTPUT EXPRS:\n`id`\n
`id2`"));
 
Assert.assertTrue(UtFrameUtils.checkPlanResultContainsNode(explainString, 0, 
"OlapScanNode"));
 
 assertSQLPlanOrErrorMsgContains("insert int

[GitHub] [doris] Lchangliang opened a new issue, #10728: [Bug] select count(*) return 0 use vectorized_engine

[GitBox]

Lchangliang opened a new issue, #10728:
URL: https://github.com/apache/doris/issues/10728

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Version
   
   master
   
   ### What's Wrong?
   
   ```
   CREATE TABLE uniq_keys_test (
   `user_id` LARGEINT NOT NULL COMMENT "用户id",
   `date` DATE NOT NULL COMMENT "数据灌入日期时间",
   `city` VARCHAR(20) COMMENT "用户所在城市",
   `age` SMALLINT COMMENT "用户年龄",
   `sex` TINYINT COMMENT "用户性别",
   `last_visit_date` DATETIME DEFAULT "1970-01-01 00:00:00" 
COMMENT "用户最后一次访问时间",
   `last_update_date` DATETIME DEFAULT "1970-01-01 00:00:00" 
COMMENT "用户最后一次更新时间",
   `last_visit_date_not_null` DATETIME NOT NULL DEFAULT 
"1970-01-01 00:00:00" COMMENT "用户最后一次访问时间",
   `cost` BIGINT DEFAULT "0" COMMENT "用户总消费",
   `max_dwell_time` INT DEFAULT "0" COMMENT "用户最大停留时间",
   `min_dwell_time` INT DEFAULT "9" COMMENT "用户最小停留时间")
   UNIQUE KEY(`user_id`, `date`, `city`, `age`, `sex`) DISTRIBUTED 
BY HASH(`user_id`)
   PROPERTIES ( "replication_num" = "1" );
   INSERT INTO uniq_keys_test VALUES
(1, '2017-10-01', 'Beijing', 10, 1, '2020-01-01', '2020-01-01', 
'2020-01-01', 1, 30, 20);
   select count(*) from uniq_keys_test;
   ```
   the result is zero.
   
   ### What You Expected?
   
   the result is one.
   
   ### How to Reproduce?
   
   _No response_
   
   ### Anything Else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] Kikyou1997 commented on issue #9846: [Enhancement] there is redundant column in scannode

[GitBox]

Kikyou1997 commented on issue #9846:
URL: https://github.com/apache/doris/issues/9846#issuecomment-1179650287

   > done #10615
   
   This issue should be reopened now,  This PR #10615 has some memory problems 
which was triggerred on TPC-DS tests as described in issue #10725.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yiguolei commented on a diff in pull request #10717: [refactor] remove PredicateColumn

[GitBox]

yiguolei commented on code in PR #10717:
URL: https://github.com/apache/doris/pull/10717#discussion_r917335252


##
be/src/olap/rowset/segment_v2/bitshuffle_page.h:
##
@@ -392,7 +392,14 @@ class BitShufflePageDecoder : public PageDecoder {
 
 size_t max_fetch = std::min(*n, static_cast(_num_elements - 
_cur_index));
 
-dst->insert_many_fix_len_data(get_data(_cur_index), max_fetch);
+// todo(wb) remove this branch after the data format is completely 
unified
+if constexpr (Type == OLAP_FIELD_TYPE_DATE) {
+dst->insert_many_date(get_data(_cur_index), max_fetch);
+} else if constexpr (Type == OLAP_FIELD_TYPE_DATETIME) {

Review Comment:
   我倾向在 column 中做这种判断，比如我们可以调用insert fixed length data， 
但是在这个函数里我们判断一下column的类型，做if else 判断



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yiguolei commented on a diff in pull request #10717: [refactor] remove PredicateColumn

[GitBox]

yiguolei commented on code in PR #10717:
URL: https://github.com/apache/doris/pull/10717#discussion_r917335637


##
be/src/olap/schema.cpp:
##
@@ -127,48 +126,45 @@ vectorized::IColumn::MutablePtr 
Schema::get_predicate_column_nullable_ptr(FieldT
 vectorized::IColumn::MutablePtr Schema::get_predicate_column_ptr(FieldType 
type) {
 switch (type) {
 case OLAP_FIELD_TYPE_BOOL:
-return doris::vectorized::PredicateColumnType::create();
+return 
doris::vectorized::ColumnVector::create();

Review Comment:
   move all these method to TabletColumn



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] platoneko commented on issue #10728: [Bug] select count(*) return 0 use vectorized_engine

[GitBox]

platoneko commented on issue #10728:
URL: https://github.com/apache/doris/issues/10728#issuecomment-1179651264

   assign me


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yiguolei commented on a diff in pull request #10717: [refactor] remove PredicateColumn

[GitBox]

yiguolei commented on code in PR #10717:
URL: https://github.com/apache/doris/pull/10717#discussion_r917336066


##
be/src/olap/bloom_filter_predicate.h:
##
@@ -81,18 +80,42 @@ class BloomFilterColumnPredicate : public ColumnPredicate {
 new_size += 
_specific_filter->find_uint32_t(dict_col->get_hash_value(idx));
 }
 }
+} else if (column.is_column_string()) {
+if constexpr (std::is_same_v) {

Review Comment:
   file_type  or fieldType?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] weizuo93 commented on a diff in pull request #10721: [Bug] Fix bug that segments are removed as trash but tablet meta is normal

[GitBox]

weizuo93 commented on code in PR #10721:
URL: https://github.com/apache/doris/pull/10721#discussion_r917337271


##
be/src/olap/tablet_manager.cpp:
##
@@ -154,13 +154,24 @@ Status TabletManager::_add_tablet_unlocked(TTabletId 
tablet_id, const TabletShar
 // During restore process, snapshot loader
 // replaced the old tablet's rowset with new rowsets, but the tablet path 
is reused, if drop files
 // here, the new rowset's file will also be dropped, so use keep files here
-bool keep_files = force ? true : false;
+bool keep_files = force;
 if (force ||
 (new_version > old_version || (new_version == old_version && new_time 
> old_time))) {
 // check if new tablet's meta is in store and add new tablet's meta to 
meta store
 res = _add_tablet_to_map_unlocked(tablet_id, tablet, update_meta, 
keep_files,
   true /*drop_old*/);
 } else {
+if (!keep_files) {
+tablet->set_tablet_state(TABLET_SHUTDOWN);

Review Comment:
   > I am confused that, this is a "add tablet" method, but why we finally set 
tablet to `TABLET_SHUTDOWN`? Should it be `existed_tablet`?
   
   @morningman 
   Yes, it may be `existed_tablet` due to misoperation of the disk. You can 
refer to this issue #10720 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] platoneko opened a new issue, #10729: [Bug] clang compile error: private field '_profile' is not used

[GitBox]

platoneko opened a new issue, #10729:
URL: https://github.com/apache/doris/issues/10729

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Version
   
   master
   
   ### What's Wrong?
   
   private field '_profile' is not used in ParquetRowGroupReader
   
   ### What You Expected?
   
   fix it
   
   ### How to Reproduce?
   
   _No response_
   
   ### Anything Else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] wangbo commented on a diff in pull request #10717: [refactor] remove PredicateColumn

[GitBox]

wangbo commented on code in PR #10717:
URL: https://github.com/apache/doris/pull/10717#discussion_r917337848


##
be/src/olap/rowset/segment_v2/bitshuffle_page.h:
##
@@ -392,7 +392,14 @@ class BitShufflePageDecoder : public PageDecoder {
 
 size_t max_fetch = std::min(*n, static_cast(_num_elements - 
_cur_index));
 
-dst->insert_many_fix_len_data(get_data(_cur_index), max_fetch);
+// todo(wb) remove this branch after the data format is completely 
unified
+if constexpr (Type == OLAP_FIELD_TYPE_DATE) {
+dst->insert_many_date(get_data(_cur_index), max_fetch);
+} else if constexpr (Type == OLAP_FIELD_TYPE_DATETIME) {

Review Comment:
   1 分支放这的好处是可以用constexpr
   2 如果要放column里，如果不加模板，单纯用if判断的话，就没法用constexpr
   如果要用模板的话，那只能给column加个模板，这个改动太大了，所有用到column的地方都得加一个模板标识<>
   另外模板应该没法给insert方法加，因为那个是虚函数
   3 
这个问题我觉得从最终方案考虑，如果最终我们决定对于date类型还是用columnvector保存，那么一定会出现一个columnvector保存多种类型的情况，那么就需要columnvector能够保存一个类型的模板，这个问题就解决了。
   如果我们计划加一个新的类型比如dateColumn用于保存date类型，那么我建议保持现状，暂时先用这个分支



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] hf200012 opened a new pull request, #10730: [TLP] how-to-contribute remove incubator

[GitBox]

hf200012 opened a new pull request, #10730:
URL: https://github.com/apache/doris/pull/10730

   # Proposed changes
   
   [TLP] how-to-contribute remove incubator
   
   Issue Number: close #xxx
   
   ## Problem Summary:
   
   Describe the overview of changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #10448: [Load][Vectorized] opt the mem use of aggregate function in load to speed up

[GitBox]

github-actions[bot] commented on PR #10448:
URL: https://github.com/apache/doris/pull/10448#issuecomment-1179660273

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #10448: [Load][Vectorized] opt the mem use of aggregate function in load to speed up

[GitBox]

github-actions[bot] commented on PR #10448:
URL: https://github.com/apache/doris/pull/10448#issuecomment-1179660279

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] BiteTheDDDDt merged pull request #10448: [Load][Vectorized] opt the mem use of aggregate function in load to speed up

[GitBox]

BiteThet merged PR #10448:
URL: https://github.com/apache/doris/pull/10448


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated (a6e4c88663 -> 502ac4e76b)

[panxiaolei]

This is an automated email from the ASF dual-hosted git repository.

panxiaolei pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


from a6e4c88663 [improve](planner): split output expr to multiple line. 
(#10710)
 add 502ac4e76b [Load][Vectorized] opt the mem use of aggregate function in 
load to speed up (#10448)

No new revisions were added by this update.

Summary of changes:
 be/src/olap/memtable.cpp | 37 +
 be/src/olap/memtable.h   | 32 +++-
 2 files changed, 40 insertions(+), 29 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] platoneko opened a new pull request, #10731: [Enhancement] Garbage collection of unused data on remote storage backend

[GitBox]

platoneko opened a new pull request, #10731:
URL: https://github.com/apache/doris/pull/10731

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem Summary:
   
   This PR will record remote unused rowset info in rocksdb, and remove remote 
data in  background gc thread.
   There are still some scenes that may generate garbage:
   1. Kill BE while BE is uploading data to remote.
   2. Although we have used `self_ owned_ remote_ rowsets` to record  which 
remote rowsets can be safely deleted, BE replica scheduling still generated 
some garbage data in our fuzzy test. Fortunately, garbage data accounts for a 
small proportion(~90GB data, only ~80MB garbage in our fuzzy test).
   A simple solution to scenario 2: delete all remote data with tablet prefix 
when drop table/partition. Although remote garbage data will still exist during 
tables' lifetime, we can ensure that there is no garbage data after dropping 
table/partition.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   4. Has unit tests been added: (Yes/No/No Need)
   5. Has document been added or modified: (Yes/No/No Need)
   6. Does it need to update dependencies: (Yes/No)
   7. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] platoneko opened a new pull request, #10732: [fix] remove unused private field _profile

[GitBox]

platoneko opened a new pull request, #10732:
URL: https://github.com/apache/doris/pull/10732

   # Proposed changes
   
   Issue Number: close #10729 
   
   ## Problem Summary:
   
   Describe the overview of changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] 924060929 commented on a diff in pull request #10659: [enhancement](nereids) make SSB works

[GitBox]

924060929 commented on code in PR #10659:
URL: https://github.com/apache/doris/pull/10659#discussion_r917338424


##
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##
@@ -81,33 +74,14 @@ public void plan(StatementBase queryStmt,
 
 PhysicalPlanTranslator physicalPlanTranslator = new 
PhysicalPlanTranslator();
 PlanTranslatorContext planTranslatorContext = new 
PlanTranslatorContext();
-physicalPlanTranslator.translatePlan(physicalPlan, 
planTranslatorContext);
+PlanFragment root = physicalPlanTranslator.translatePlan(physicalPlan, 
planTranslatorContext);
 
 scanNodeList = planTranslatorContext.getScanNodeList();
 descTable = planTranslatorContext.getDescTable();
 fragments = new 
ArrayList<>(planTranslatorContext.getPlanFragmentList());
-for (PlanFragment fragment : fragments) {
-fragment.finalize(queryStmt);
-}
-Collections.reverse(fragments);
-PlanFragment root = fragments.get(0);
-
-// compute output exprs
-Map outputCandidates = Maps.newHashMap();
-List outputExprs = Lists.newArrayList();
-for (TupleId tupleId : root.getPlanRoot().getTupleIds()) {
-TupleDescriptor tupleDescriptor = descTable.getTupleDesc(tupleId);
-for (SlotDescriptor slotDescriptor : tupleDescriptor.getSlots()) {
-SlotRef slotRef = new SlotRef(slotDescriptor);
-outputCandidates.put(slotDescriptor.getId().asInt(), slotRef);
-}
-}
-physicalPlan.getOutput().stream()
-.forEach(i -> 
outputExprs.add(planTranslatorContext.findExpr(i)));
-root.setOutputExprs(outputExprs);
-root.getPlanRoot().convertToVectoriezd();
 
-logicalPlanAdapter.setResultExprs(outputExprs);
+// set output exprs
+logicalPlanAdapter.setResultExprs(root.getOutputExprs());

Review Comment:
   great refactor



##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AggregateDisassemble.java:
##
@@ -17,144 +17,156 @@
 
 package org.apache.doris.nereids.rules.rewrite;
 
-import org.apache.doris.analysis.FunctionName;
-import org.apache.doris.catalog.Catalog;
-import org.apache.doris.catalog.Function;
-import org.apache.doris.catalog.Function.CompareMode;
-import org.apache.doris.catalog.Type;
-import org.apache.doris.nereids.operators.Operator;
 import org.apache.doris.nereids.operators.plans.AggPhase;
 import org.apache.doris.nereids.operators.plans.logical.LogicalAggregate;
 import org.apache.doris.nereids.rules.Rule;
 import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.Alias;
 import org.apache.doris.nereids.trees.expressions.Expression;
 import org.apache.doris.nereids.trees.expressions.NamedExpression;
-import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
 import org.apache.doris.nereids.trees.expressions.functions.AggregateFunction;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
 import org.apache.doris.nereids.trees.plans.Plan;
-import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalUnaryPlan;
 
-import com.clearspring.analytics.util.Lists;
-import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
 
-import java.util.HashMap;
+import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
 import java.util.stream.Collectors;
 
 /**
- * TODO: if instance count is 1, shouldn't disassemble the agg operator
  * Used to generate the merge agg node for distributed execution.
- * Do this in following steps:
- *  1. clone output expr list, find all agg function
- *  2. set found agg function intermediaType
- *  3. create new child plan rooted at new local agg
- *  4. update the slot referenced by expr of merge agg
- *  5. create plan rooted at merge agg, return it.
+ * NOTICE: GLOBAL output expressions' ExprId should SAME with ORIGIN output 
expressions' ExprId.
+ * If we have a query: SELECT SUM(v1 * v2) + 1 FROM t GROUP BY k + 1
+ * the initial plan is:
+ *   Aggregate(phase: [GLOBAL], outputExpr: [Alias(k + 1) #1, Alias(SUM(v1 * 
v2) + 1) #2], groupByExpr: [k + 1])
+ *   +-- childPlan
+ * we should rewrite to:
+ *   Aggregate(phase: [GLOBAL], outputExpr: [Alias(b) #1, Alias(SUM(a) + 1) 
#2], groupByExpr: [b])
+ *   +-- Aggregate(phase: [LOCAL], outputExpr: [SUM(v1 * v2) as a, (k + 1) as 
b], groupByExpr: [k + 1])
+ *   +-- childPlan

Review Comment:
   great comment



##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AggregateDisassemble.java:
##
@@ -17,144 +17,156 @@
 
 package org.apache.doris.nereids.rules.rewrite;
 
-import org.apache.doris.analysis.FunctionName;
-

[GitHub] [doris] compasses opened a new issue, #10733: [Feature] Add NGRAM bloom filter index to speed up like queries.

[GitBox]

compasses opened a new issue, #10733:
URL: https://github.com/apache/doris/issues/10733

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Description
   
   To speed up like queries we have pushed the like function to storage layer 
in PR #10355 , which can get 2x~3x performance gain, no matter vectorized or 
not. But we want to go the extra mile, and make it more faster and less 
resource overhead. Base on that, we are  going to implement a new index for 
like queries.
   
   We have researched several solutions such as pg_trgm from postgresql、ngrambf 
from clickhouse and FST from elasticsearch.  Since Doris have bloom filter 
index  already, in consideration of complexity、function scope and 
compatibility. Finally, we will choose the way as clickhouse did 
```ngrambf_v1(n, size_of_bloom_filter_in_bytes, number_of_hash_functions, 
random_seed)```: the input column string is split into n-grams (first parameter 
– n-gram size), and then stored in a bloom filter. During query, the like 
pattern will also be split to n-grams and generate a bloom filter to do the 
filter, use the bloom filter to skip granule.
   
   For doris here is the details:
   1. Reuse the exist bloom filter index read/write process, and the storage 
layer will be unaffected.
   2. Add a new kind of bloom filter index, example : 
"ngram_bloom_filter_columns" = "(col1,n,512), (col2,n,512)"，n-gram size，
   512-bloom filter size in bytes，n and 512 all can be configured，and both have 
default value like (3,512).
   3. Add new type of algorithm: NGRAM_BLOOM_FILTER, which will extract gram 
and calculate the bloom filter.
   4. For the new algorithm the HashStrategy will follow the clickhouse
   5. Query will support index filter pages for like queries , if exist the 
ngram bloom filter, which base the #10355 
   6. Support add index for history data：ALTER TABLE  SET 
("ngram_bloom_filter_columns" = "(col1,n,512), (col2,n,512)").
   
   
![image](https://user-images.githubusercontent.com/10161171/178133582-e9266441-88b1-49ba-9ac2-241b460db404.png)
   
   That's all, thanks.
   
   
   
   ### Use case
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] Gabriel39 opened a new pull request, #10734: Revert "[Enhancement] Add column prune support for VOlapScanNode

[GitBox]

Gabriel39 opened a new pull request, #10734:
URL: https://github.com/apache/doris/pull/10734

   This reverts commit e37d29485f2357441b24ed2b7a175bca004e536c.
   
   Some bugs are introduced by this commit
   
   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem Summary:
   
   Describe the overview of changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #10734: Revert "[Enhancement] Add column prune support for VOlapScanNode

[GitBox]

github-actions[bot] commented on PR #10734:
URL: https://github.com/apache/doris/pull/10734#issuecomment-1179668836

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #10734: Revert "[Enhancement] Add column prune support for VOlapScanNode

[GitBox]

github-actions[bot] commented on PR #10734:
URL: https://github.com/apache/doris/pull/10734#issuecomment-1179668843

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #10732: [fix] remove unused private field _profile

[GitBox]

github-actions[bot] commented on PR #10732:
URL: https://github.com/apache/doris/pull/10732#issuecomment-1179668939

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #10732: [fix] remove unused private field _profile

[GitBox]

github-actions[bot] commented on PR #10732:
URL: https://github.com/apache/doris/pull/10732#issuecomment-1179668941

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org