[GitHub] [doris] github-actions[bot] commented on pull request #11573: [fix](date) fix the value may be changed during the parsing of date and datetime types

[GitBox]

github-actions[bot] commented on PR #11573:
URL: https://github.com/apache/doris/pull/11573#issuecomment-1207352096

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] dataroaring opened a new pull request, #11576: (test) add dryRun option and group all cases into either p0 or p1

[GitBox]

dataroaring opened a new pull request, #11576:
URL: https://github.com/apache/doris/pull/11576

   1. add dryRun option to list tests
   2. group all cases into p0 p1 p2
   
   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] dataroaring merged pull request #11542: [enhancement](compaction) add idle schedule and max_size limit for base compaction

[GitBox]

dataroaring merged PR #11542:
URL: https://github.com/apache/doris/pull/11542


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [enhancement](compaction) add idle schedule and max_size limit for base compaction (#11542)

[dataroaring]

This is an automated email from the ASF dual-hosted git repository.

dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new bd4048f8fb [enhancement](compaction) add idle schedule and max_size 
limit for base compaction (#11542)
bd4048f8fb is described below

commit bd4048f8fbbfe4cc153101eb7fe57dc3405581a2
Author: yixiutt <102007456+yixi...@users.noreply.github.com>
AuthorDate: Sun Aug 7 16:21:57 2022 +0800

[enhancement](compaction) add idle schedule and max_size limit for base 
compaction (#11542)

Co-authored-by: yixiutt 
---
 be/src/common/config.h  |  5 +
 be/src/olap/base_compaction.cpp | 30 ++
 be/src/olap/base_compaction.h   |  4 
 be/src/util/thread.cpp  | 17 +
 be/src/util/thread.h|  2 ++
 5 files changed, 54 insertions(+), 4 deletions(-)

diff --git a/be/src/common/config.h b/be/src/common/config.h
index 5bafe0899c..dbeebf41bc 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -262,6 +262,11 @@ CONF_mInt64(base_compaction_num_cumulative_deltas, "5");
 CONF_mDouble(base_cumulative_delta_ratio, "0.3");
 CONF_mInt64(base_compaction_interval_seconds_since_last_operation, "86400");
 CONF_mInt32(base_compaction_write_mbytes_per_sec, "5");
+CONF_Bool(enable_base_compaction_idle_sched, "true");
+
+// dup key not compaction big files
+CONF_Bool(enable_dup_key_base_compaction_skip_big_file, "true");
+CONF_mInt64(base_compaction_dup_key_max_file_size_mbytes, "1024");
 
 // config the cumulative compaction policy
 // Valid configs: num_based, size_based
diff --git a/be/src/olap/base_compaction.cpp b/be/src/olap/base_compaction.cpp
index 2e8d1f82aa..22485c2b7a 100644
--- a/be/src/olap/base_compaction.cpp
+++ b/be/src/olap/base_compaction.cpp
@@ -49,6 +49,9 @@ Status BaseCompaction::prepare_compact() {
 }
 
 Status BaseCompaction::execute_compact_impl() {
+if (config::enable_base_compaction_idle_sched) {
+Thread::set_idle_sched();
+}
 std::unique_lock lock(_tablet->get_base_compaction_lock(), 
std::try_to_lock);
 if (!lock.owns_lock()) {
 LOG(WARNING) << "another base compaction is running. tablet=" << 
_tablet->full_name();
@@ -81,16 +84,35 @@ Status BaseCompaction::execute_compact_impl() {
 return Status::OK();
 }
 
+void BaseCompaction::_filter_input_rowset() {
+// if enable dup key skip big file and no delete predicate
+// we skip big files too save resources
+if (!config::enable_dup_key_base_compaction_skip_big_file ||
+_tablet->keys_type() != KeysType::DUP_KEYS || 
_tablet->delete_predicates().size() != 0) {
+return;
+}
+int64_t max_size = config::base_compaction_dup_key_max_file_size_mbytes * 
1024 * 1024;
+// first find a proper rowset for start
+auto rs_iter = _input_rowsets.begin();
+while (rs_iter != _input_rowsets.end()) {
+if ((*rs_iter)->rowset_meta()->total_disk_size() >= max_size) {
+rs_iter = _input_rowsets.erase(rs_iter);
+} else {
+break;
+}
+}
+}
+
 Status BaseCompaction::pick_rowsets_to_compact() {
 _input_rowsets.clear();
 _tablet->pick_candidate_rowsets_to_base_compaction(&_input_rowsets);
-if (_input_rowsets.size() <= 1) {
-return Status::OLAPInternalError(OLAP_ERR_BE_NO_SUITABLE_VERSION);
-}
-
 std::sort(_input_rowsets.begin(), _input_rowsets.end(), 
Rowset::comparator);
 RETURN_NOT_OK(check_version_continuity(_input_rowsets));
 RETURN_NOT_OK(_check_rowset_overlapping(_input_rowsets));
+_filter_input_rowset();
+if (_input_rowsets.size() <= 1) {
+return Status::OLAPInternalError(OLAP_ERR_BE_NO_SUITABLE_VERSION);
+}
 
 // If there are delete predicate rowsets in tablet, start_version > 0 
implies some rowsets before
 // delete version cannot apply these delete predicates, which can cause 
incorrect query result.
diff --git a/be/src/olap/base_compaction.h b/be/src/olap/base_compaction.h
index 21a6e24bc3..96a4a362f4 100644
--- a/be/src/olap/base_compaction.h
+++ b/be/src/olap/base_compaction.h
@@ -47,6 +47,10 @@ private:
 // a rowset with overlapping segments should be compacted by cumulative 
compaction first.
 Status _check_rowset_overlapping(const vector& rowsets);
 
+// filter input rowset in some case:
+// 1. dup key without delete predicate
+void _filter_input_rowset();
+
 DISALLOW_COPY_AND_ASSIGN(BaseCompaction);
 };
 
diff --git a/be/src/util/thread.cpp b/be/src/util/thread.cpp
index 39060d645b..db8af4bb42 100644
--- a/be/src/util/thread.cpp
+++ b/be/src/util/thread.cpp
@@ -73,6 +73,8 @@ public:
 
 static void set_thread_name(const std::string& name, int64_t tid);
 
+static void set_idle_sched(int64_t tid);
+
 // not the system TID, since pthread_t is less prone to being recycled.
 void add_thread(const pthread

[GitHub] [doris] BiteTheDDDDt opened a new pull request, #11577: [Chore](thirdparty) upgrade phmap and s2geo

[GitBox]

BiteThet opened a new pull request, #11577:
URL: https://github.com/apache/doris/pull/11577

   part of #9312
   
   ## Problem summary
   
   This pr upgrade phmap from 1.33 to 1.35, add abseil and upgrade s2geo to 
0.10.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [X] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [X] No Need
   3. Has document been added or modified:
   - [X] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [X] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [X] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] mrhhsg opened a new pull request, #11578: [improvement](third-party) Build re2 with release mode

[GitBox]

mrhhsg opened a new pull request, #11578:
URL: https://github.com/apache/doris/pull/11578

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [x] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [x] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [x] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [x] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [x] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11578: [improvement](third-party) Build re2 with release mode

[GitBox]

github-actions[bot] commented on PR #11578:
URL: https://github.com/apache/doris/pull/11578#issuecomment-1207387949

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch dev-1.1.2 updated: [bugfix](memtracker)fix exceed memory limit log (#11485)

[zouxinyi]

This is an automated email from the ASF dual-hosted git repository.

zouxinyi pushed a commit to branch dev-1.1.2
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/dev-1.1.2 by this push:
 new 487fd3c1d7 [bugfix](memtracker)fix exceed memory limit log (#11485)
487fd3c1d7 is described below

commit 487fd3c1d715477c3d3f321b6e8ce0addc30bc12
Author: Xinyi Zou 
AuthorDate: Thu Aug 4 10:22:20 2022 +0800

[bugfix](memtracker)fix exceed memory limit log (#11485)
---
 be/src/exec/cross_join_node.cpp  |  1 -
 be/src/exec/except_node.cpp  |  1 -
 be/src/exec/hash_join_node.cpp   |  2 -
 be/src/exec/intersect_node.cpp   |  1 -
 be/src/exec/set_operation_node.cpp   |  1 -
 be/src/runtime/memory/mem_tracker_limiter.cpp| 56 ++--
 be/src/runtime/memory/mem_tracker_limiter.h  |  9 ++--
 be/src/runtime/memory/mem_tracker_task_pool.cpp  |  8 ++--
 be/src/runtime/memory/thread_mem_tracker_mgr.cpp | 29 ++--
 be/src/runtime/memory/thread_mem_tracker_mgr.h   | 41 +++--
 be/src/runtime/plan_fragment_executor.cpp|  2 +-
 be/src/runtime/thread_context.cpp| 15 ---
 be/src/runtime/thread_context.h  | 40 +
 be/src/vec/exec/join/vhash_join_node.cpp |  1 -
 be/src/vec/exec/vaggregation_node.cpp|  2 -
 be/src/vec/exec/vcross_join_node.cpp |  1 -
 be/src/vec/exec/vset_operation_node.cpp  |  1 -
 17 files changed, 64 insertions(+), 147 deletions(-)

diff --git a/be/src/exec/cross_join_node.cpp b/be/src/exec/cross_join_node.cpp
index b26f4f2cd4..0743fe04c4 100644
--- a/be/src/exec/cross_join_node.cpp
+++ b/be/src/exec/cross_join_node.cpp
@@ -52,7 +52,6 @@ Status CrossJoinNode::close(RuntimeState* state) {
 Status CrossJoinNode::construct_build_side(RuntimeState* state) {
 // Do a full scan of child(1) and store all build row batches.
 RETURN_IF_ERROR(child(1)->open(state));
-SCOPED_UPDATE_MEM_EXCEED_CALL_BACK("Cross join, while getting next from 
child 1");
 
 while (true) {
 RowBatch* batch = _build_batch_pool->add(
diff --git a/be/src/exec/except_node.cpp b/be/src/exec/except_node.cpp
index 58a9b67f2f..dd92859671 100644
--- a/be/src/exec/except_node.cpp
+++ b/be/src/exec/except_node.cpp
@@ -40,7 +40,6 @@ Status ExceptNode::init(const TPlanNode& tnode, RuntimeState* 
state) {
 
 Status ExceptNode::open(RuntimeState* state) {
 RETURN_IF_ERROR(SetOperationNode::open(state));
-SCOPED_UPDATE_MEM_EXCEED_CALL_BACK("Except Node, while probing the hash 
table.");
 // if a table is empty, the result must be empty
 if (_hash_tbl->size() == 0) {
 _hash_tbl_iterator = _hash_tbl->begin();
diff --git a/be/src/exec/hash_join_node.cpp b/be/src/exec/hash_join_node.cpp
index 7c572ff95d..02f52d2124 100644
--- a/be/src/exec/hash_join_node.cpp
+++ b/be/src/exec/hash_join_node.cpp
@@ -186,7 +186,6 @@ Status HashJoinNode::construct_hash_table(RuntimeState* 
state) {
 // The hash join node needs to keep in memory all build tuples, including 
the tuple
 // row ptrs.  The row ptrs are copied into the hash table's internal 
structure so they
 // don't need to be stored in the _build_pool.
-SCOPED_UPDATE_MEM_EXCEED_CALL_BACK("Hash join, while constructing the hash 
table.");
 RowBatch build_batch(child(1)->row_desc(), state->batch_size(), 
mem_tracker().get());
 RETURN_IF_ERROR(child(1)->open(state));
 
@@ -304,7 +303,6 @@ Status HashJoinNode::get_next(RuntimeState* state, 
RowBatch* out_batch, bool* eo
 // In most cases, no additional memory overhead will be applied for at 
this stage,
 // but if the expression calculation in this node needs to apply for 
additional memory,
 // it may cause the memory to exceed the limit.
-SCOPED_UPDATE_MEM_EXCEED_CALL_BACK("Hash join, while execute get_next.");
 SCOPED_TIMER(_runtime_profile->total_time_counter());
 
 if (reached_limit()) {
diff --git a/be/src/exec/intersect_node.cpp b/be/src/exec/intersect_node.cpp
index 9f5eb3ece1..a79810734d 100644
--- a/be/src/exec/intersect_node.cpp
+++ b/be/src/exec/intersect_node.cpp
@@ -43,7 +43,6 @@ Status IntersectNode::init(const TPlanNode& tnode, 
RuntimeState* state) {
 // 2 probe with child(1), then filter the hash table and find the matched 
item, use them to rebuild a hash table
 // repeat [2] this for all the rest child
 Status IntersectNode::open(RuntimeState* state) {
-SCOPED_UPDATE_MEM_EXCEED_CALL_BACK("Intersect Node, while probing the hash 
table.");
 RETURN_IF_ERROR(SetOperationNode::open(state));
 // if a table is empty, the result must be empty
 if (_hash_tbl->size() == 0) {
diff --git a/be/src/exec/set_operation_node.cpp 
b/be/src/exec/set_operation_node.cpp
index e0bfbb199c..7a9d3a334f 100644
--- a/be/src/exec/set_operation_node.cpp
+++ b/be/src/exec/set_operation_node.c

[GitHub] [doris] morningman merged pull request #11574: [doc](asf) update .asf.ymal to stop sending notification to dev@doris

[GitBox]

morningman merged PR #11574:
URL: https://github.com/apache/doris/pull/11574


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [doc](asf) update .asf.ymal to stop sending notification to dev@doris (#11574)

[morningman]

This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 7deebf7086 [doc](asf) update .asf.ymal to stop sending notification to 
dev@doris (#11574)
7deebf7086 is described below

commit 7deebf70861165fabc859e17aa441cf90bea6f5f
Author: Mingyu Chen 
AuthorDate: Sun Aug 7 20:31:24 2022 +0800

[doc](asf) update .asf.ymal to stop sending notification to dev@doris 
(#11574)
---
 .asf.yaml | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/.asf.yaml b/.asf.yaml
index adc150ff89..d3f49b8a62 100644
--- a/.asf.yaml
+++ b/.asf.yaml
@@ -61,8 +61,6 @@ github:
   required_pull_request_reviews:
 dismiss_stale_reviews: true
 required_approving_review_count: 1
-  notifications:
-pullrequests_status:  commits@doris.apache.org
   collaborators:
 - jackwener
 - tinke
@@ -73,3 +71,6 @@ github:
 - dataroaring
 - morrySnow
 - dataalive
+
+notifications:
+pullrequests_status:  commits@doris.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] Gabriel39 commented on pull request #11573: [fix](date) fix the value may be changed during the parsing of date and datetime types

[GitBox]

Gabriel39 commented on PR #11573:
URL: https://github.com/apache/doris/pull/11573#issuecomment-1207399199

   Could you add a case in FE UT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yiguolei merged pull request #11578: [improvement](third-party) Build re2 with release mode

[GitBox]

yiguolei merged PR #11578:
URL: https://github.com/apache/doris/pull/11578


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [improvement](thirdparty) Build re2 with release mode (#11578)

[yiguolei]

This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 8b9d299472 [improvement](thirdparty) Build re2 with release mode 
(#11578)
8b9d299472 is described below

commit 8b9d299472043070a1369ee41d8e0484a6f36a6c
Author: Jerry Hu 
AuthorDate: Sun Aug 7 20:50:07 2022 +0800

[improvement](thirdparty) Build re2 with release mode (#11578)
---
 thirdparty/build-thirdparty.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/thirdparty/build-thirdparty.sh b/thirdparty/build-thirdparty.sh
index e2f8c16b20..ea3479788e 100755
--- a/thirdparty/build-thirdparty.sh
+++ b/thirdparty/build-thirdparty.sh
@@ -596,7 +596,7 @@ build_re2() {
 check_if_source_exist "${RE2_SOURCE}"
 cd "${TP_SOURCE_DIR}/${RE2_SOURCE}"
 
-"${CMAKE_CMD}" -G "${GENERATOR}" -DBUILD_SHARED_LIBS=0 
-DCMAKE_INSTALL_PREFIX="${TP_INSTALL_DIR}"
+"${CMAKE_CMD}" -DCMAKE_BUILD_TYPE=Release -G "${GENERATOR}" 
-DBUILD_SHARED_LIBS=0 -DCMAKE_INSTALL_PREFIX="${TP_INSTALL_DIR}"
 "${BUILD_SYSTEM}" -j "${PARALLEL}" install
 }
 


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yiguolei merged pull request #11386: [improvement](profile)fix profile may cause query slow

[GitBox]

yiguolei merged PR #11386:
URL: https://github.com/apache/doris/pull/11386


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yiguolei closed issue #11385: [Enhancement] Fix open query profile may cause query slow

[GitBox]

yiguolei closed issue #11385: [Enhancement] Fix open query profile may cause 
query slow
URL: https://github.com/apache/doris/issues/11385


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: fix profile may cause query slow (#11386)

[yiguolei]

This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 8802a41918 fix profile may cause query slow (#11386)
8802a41918 is described below

commit 8802a419186bdfdf22eba2160e9bfaa72fca6888
Author: wangbo 
AuthorDate: Sun Aug 7 20:52:52 2022 +0800

fix profile may cause query slow (#11386)

Co-authored-by: Wang Bo 
---
 .../src/main/java/org/apache/doris/qe/ConnectProcessor.java   | 11 +++
 .../src/main/java/org/apache/doris/qe/StmtExecutor.java   |  5 -
 2 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java 
b/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
index 4ca8492ed0..a9b18776e1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
@@ -20,6 +20,7 @@ package org.apache.doris.qe;
 import org.apache.doris.analysis.InsertStmt;
 import org.apache.doris.analysis.KillStmt;
 import org.apache.doris.analysis.Queriable;
+import org.apache.doris.analysis.QueryStmt;
 import org.apache.doris.analysis.SqlParser;
 import org.apache.doris.analysis.SqlScanner;
 import org.apache.doris.analysis.StatementBase;
@@ -45,6 +46,7 @@ import org.apache.doris.mysql.MysqlPacket;
 import org.apache.doris.mysql.MysqlProto;
 import org.apache.doris.mysql.MysqlSerializer;
 import org.apache.doris.mysql.MysqlServerStatusFlag;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
 import org.apache.doris.nereids.parser.NereidsParser;
 import org.apache.doris.plugin.AuditEvent.EventType;
 import org.apache.doris.proto.Data;
@@ -421,6 +423,15 @@ public class ConnectProcessor {
 
 MysqlChannel channel = ctx.getMysqlChannel();
 channel.sendAndFlush(packet);
+// note(wb) we should write profile after return result to mysql client
+// because write profile maybe take too much time
+// explain query stmt do not have profile
+if (executor != null && !executor.getParsedStmt().isExplain()
+&& (executor.getParsedStmt() instanceof QueryStmt // currently 
only QueryStmt and insert need profile
+|| executor.getParsedStmt() instanceof LogicalPlanAdapter
+|| executor.getParsedStmt() instanceof InsertStmt)) {
+executor.writeProfile(true);
+}
 }
 
 public TMasterOpResult proxyExecute(TMasterOpRequest request) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java 
b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
index b6f1345bbc..c1edb216da 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
@@ -434,10 +434,6 @@ public class StmtExecutor implements ProfileWriter {
 span.setAttribute("queryId", 
DebugUtil.printId(newQueryId));
 }
 handleQueryStmt();
-// explain query stmt do not have profile
-if (!parsedStmt.isExplain()) {
-writeProfile(true);
-}
 break;
 } catch (RpcException e) {
 if (i == retryTime - 1) {
@@ -469,7 +465,6 @@ public class StmtExecutor implements ProfileWriter {
 handleInsertStmt();
 if (!((InsertStmt) parsedStmt).getQueryStmt().isExplain()) 
{
 queryType = "Insert";
-writeProfile(true);
 }
 } catch (Throwable t) {
 LOG.warn("handle insert stmt fail", t);


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] luozenglin commented on pull request #11573: [fix](date) fix the value may be changed during the parsing of date and datetime types

[GitBox]

luozenglin commented on PR #11573:
URL: https://github.com/apache/doris/pull/11573#issuecomment-1207404949

   > Could you add a case in FE UT?
   
   done


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] compasses opened a new pull request, #11579: [Feature](NGram BloomFilter Index) add new ngram bloom filter index to speed up like query

[GitBox]

compasses opened a new pull request, #11579:
URL: https://github.com/apache/doris/pull/11579

   # Proposed changes
   
   Issue Number: close #10733
   
   ## Problem summary

   Describe your changes.
   
   This PR implement  the new bloom filter index: NGram bloom filter index, 
which was proposed in  #10733.
   The new index can improve the like query performance greatly, from our some 
test case , can  get order of magnitude  improve.
   For how to use it you can check the docs in this PR, and the index based on 
the ```enable_function_pushdown```, you need set it to ```true```, to make the 
index work for like query.
   
   
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] No
   2. Has unit tests been added:
   - [ ] Yes
   3. Has document been added or modified:
   - [ ] Yes
   4. Does it need to update dependencies:
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] qidaye commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error

[GitBox]

qidaye commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r939676905


##
fe/fe-core/src/test/resources/data/es/es7_aliases_mapping.json:
##
@@ -0,0 +1,50 @@
+{
+  "test2_202207": {
+"mappings": {

Review Comment:
   Where is `aliases` field?



##
fe/fe-core/src/main/java/org/apache/doris/datasource/EsExternalDataSource.java:
##
@@ -166,7 +167,10 @@ public List listDatabaseNames(SessionContext ctx) {
 
 @Override
 public List listTableNames(SessionContext ctx, String dbName) {
-return esRestClient.getIndexes();
+List indexes = 
esRestClient.getIndexes().stream().distinct().collect(Collectors.toList());
+esRestClient.getAliases().entrySet().stream().filter(e -> 
indexes.contains(e.getKey()))

Review Comment:
   Do we treat `alias` as an independent index?
   Should we keep the correspondence between the alias and the real index？



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] stalary commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error

[GitBox]

stalary commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r939679536


##
fe/fe-core/src/test/resources/data/es/es7_aliases_mapping.json:
##
@@ -0,0 +1,50 @@
+{
+  "test2_202207": {
+"mappings": {

Review Comment:
   My problem, I'm going to replace it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] stalary commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error

[GitBox]

stalary commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r939679754


##
fe/fe-core/src/main/java/org/apache/doris/datasource/EsExternalDataSource.java:
##
@@ -166,7 +167,10 @@ public List listDatabaseNames(SessionContext ctx) {
 
 @Override
 public List listTableNames(SessionContext ctx, String dbName) {
-return esRestClient.getIndexes();
+List indexes = 
esRestClient.getIndexes().stream().distinct().collect(Collectors.toList());
+esRestClient.getAliases().entrySet().stream().filter(e -> 
indexes.contains(e.getKey()))

Review Comment:
   Currently, the method of geting the mapping is compatible. Alias and index 
can be queried. Therefore, the mapping relationship is not maintained.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morningman opened a new pull request, #11580: [fix](storage-policy) fix bug that missing field when refreshing storage policy

[GitBox]

morningman opened a new pull request, #11580:
URL: https://github.com/apache/doris/pull/11580

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   1. Change all required fields to optional
   Although they all "required", but it not recommanded to use `required`, 
because it is hard to modify in future.
   2. Fix a missing field bug
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   3. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   5. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   6. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morningman opened a new issue, #11581: [Bug] fail to refresh storge policy. host=172.19.0.11, port=9222, code=OK, reason=No more data to read.

[GitBox]

morningman opened a new issue, #11581:
URL: https://github.com/apache/doris/issues/11581

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Version
   
   master
   
   ### What's Wrong?
   
   be.INFO with error:
   ```
fail to refresh storge policy. host=172.19.0.11, port=9222, code=OK, 
reason=No more data to read.
   ```
   
   ### What You Expected?
   
   No error
   
   ### How to Reproduce?
   
   Start a new cluster, add a backend.
   
   ### Anything Else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morningman opened a new pull request, #11582: [feature-wip](new-scan) add scanner scheduling framework

[GitBox]

morningman opened a new pull request, #11582:
URL: https://github.com/apache/doris/pull/11582

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] stalary commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error

[GitBox]

stalary commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r939679536


##
fe/fe-core/src/test/resources/data/es/es7_aliases_mapping.json:
##
@@ -0,0 +1,50 @@
+{
+  "test2_202207": {
+"mappings": {

Review Comment:
   My problem, I'm going to replace it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] stalary commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error

[GitBox]

stalary commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r939734077


##
fe/fe-core/src/test/resources/data/es/es7_aliases_mapping.json:
##
@@ -0,0 +1,50 @@
+{
+  "test2_202207": {
+"mappings": {

Review Comment:
   `JSONObject test2Aliases = EsUtil.getMappingProps("test2", 
loadJsonFromFile("data/es/es7_aliases_mapping.json"),
   null);` _mapping not contains aliases, I use test2 to get 
real index.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11580: [fix](storage-policy) fix bug that missing field when refreshing storage policy

[GitBox]

github-actions[bot] commented on PR #11580:
URL: https://github.com/apache/doris/pull/11580#issuecomment-1207529698

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11575: [fix](doc) remove docs for direct compiling on Centos

[GitBox]

github-actions[bot] commented on PR #11575:
URL: https://github.com/apache/doris/pull/11575#issuecomment-1207530703

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris-website] branch master updated: update backup

[jiafengzheng]

This is an automated email from the ASF dual-hosted git repository.

jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git


The following commit(s) were added to refs/heads/master by this push:
 new 62f974fae4f update backup
62f974fae4f is described below

commit 62f974fae4f05080e23a92167a28fe4b75b9e113
Author: jiafeng.zhang 
AuthorDate: Mon Aug 8 08:55:13 2022 +0800

update backup
---
 .../Backup-and-Restore/BACKUP.md   | 44 +
 .../Backup-and-Restore/BACKUP.md   | 57 ++
 2 files changed, 101 insertions(+)

diff --git 
a/docs/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/BACKUP.md
 
b/docs/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/BACKUP.md
index d779cb64e8c..bc025c00691 100644
--- 
a/docs/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/BACKUP.md
+++ 
b/docs/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/BACKUP.md
@@ -86,6 +86,50 @@ TO example_repo
 EXCLUDE (example_tbl);
 ```
 
+4. Create a repository named s3_repo to link cloud storage directly without 
going through the broker.
+
+```
+CREATE REPOSITORY `s3_repo`
+WITH S3
+ON LOCATION "s3://s3-repo"
+PROPERTIES
+(
+"AWS_ENDPOINT" = "http://s3-REGION.amazonaws.com";,
+"AWS_ACCESS_KEY" = "AWS_ACCESS_KEY",
+"AWS_SECRET_KEY"="AWS_SECRET_KEY",
+"AWS_REGION" = "REGION"
+);
+```
+
+5. Create a repository named hdfs_repo to link HDFS directly without going 
through the broker.
+
+```
+CREATE REPOSITORY `hdfs_repo`
+WITH hdfs
+ON LOCATION "hdfs://hadoop-name-node:54310/path/to/repo/"
+PROPERTIES
+(
+"fs.defaultFS"="hdfs://hadoop-name-node:54310",
+"hadoop.username"="user"
+);
+```
+
+6. Create a repository named minio_repo to link minio storage directly through 
the s3 protocol.
+
+```
+CREATE REPOSITORY `minio_repo`
+WITH S3
+ON LOCATION "s3://minio_repo"
+PROPERTIES
+(
+"AWS_ENDPOINT" = "http://minio.com";,
+"AWS_ACCESS_KEY" = "MINIO_USER",
+"AWS_SECRET_KEY"="MINIO_PASSWORD",
+"AWS_REGION" = "REGION",
+"use_path_style" = "true"
+);
+```
+
 ### Keywords
 
 ```text
diff --git 
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/BACKUP.md
 
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/BACKUP.md
index e5b1778713a..cc384e18a33 100644
--- 
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/BACKUP.md
+++ 
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/BACKUP.md
@@ -86,6 +86,63 @@ TO example_repo
 EXCLUDE (example_tbl);
 ```
 
+4. 创建名为 hdfs_repo 的仓库，依赖 Baidu hdfs broker 
"hdfs_broker"，数据根目录为：hdfs://hadoop-name-node:54310/path/to/repo/
+
+```
+CREATE REPOSITORY `hdfs_repo`
+WITH BROKER `hdfs_broker`
+ON LOCATION "hdfs://hadoop-name-node:54310/path/to/repo/"
+PROPERTIES
+(
+"username" = "user",
+"password" = "password"
+);
+```
+
+5. 创建名为 s3_repo 的仓库，直接链接云存储，而不通过broker.
+
+```
+CREATE REPOSITORY `s3_repo`
+WITH S3
+ON LOCATION "s3://s3-repo"
+PROPERTIES
+(
+"AWS_ENDPOINT" = "http://s3-REGION.amazonaws.com";,
+"AWS_ACCESS_KEY" = "AWS_ACCESS_KEY",
+"AWS_SECRET_KEY"="AWS_SECRET_KEY",
+"AWS_REGION" = "REGION"
+);
+```
+
+6. 创建名为 hdfs_repo 的仓库，直接链接HDFS，而不通过broker.
+
+```
+CREATE REPOSITORY `hdfs_repo`
+WITH hdfs
+ON LOCATION "hdfs://hadoop-name-node:54310/path/to/repo/"
+PROPERTIES
+(
+"fs.defaultFS"="hdfs://hadoop-name-node:54310",
+"hadoop.username"="user"
+);
+```
+
+7. 创建名为 minio_repo 的仓库，直接通过 s3 协议链接 minio.
+
+```
+CREATE REPOSITORY `minio_repo`
+WITH S3
+ON LOCATION "s3://minio_repo"
+PROPERTIES
+(
+"AWS_ENDPOINT" = "http://minio.com";,
+"AWS_ACCESS_KEY" = "MINIO_USER",
+"AWS_SECRET_KEY"="MINIO_PASSWORD",
+"AWS_REGION" = "REGION",
+"use_path_style" = "true"
+);
+```
+
 ### Keywords
 
 ```text


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yiguolei closed issue #11572: [Bug] the value may be changed of date and datetime types

[GitBox]

yiguolei closed issue #11572: [Bug] the value may be changed of date and 
datetime types
URL: https://github.com/apache/doris/issues/11572


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yiguolei merged pull request #11573: [fix](date) fix the value may be changed during the parsing of date and datetime types

[GitBox]

yiguolei merged PR #11573:
URL: https://github.com/apache/doris/pull/11573


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [fix](date) fix the value may be changed during the parsing of date and datetime types (#11573)

[yiguolei]

This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 4f5db35990 [fix](date) fix the value may be changed during the parsing 
of date and datetime types (#11573)
4f5db35990 is described below

commit 4f5db359908c3452850f9bf91a4eb80fe55df1d7
Author: luozenglin <37725793+luozeng...@users.noreply.github.com>
AuthorDate: Mon Aug 8 08:58:30 2022 +0800

[fix](date) fix the value may be changed during the parsing of date and 
datetime types (#11573)

* [fix](date) fix the value may be changed during the parsing of date and 
datetime types
---
 .../main/java/org/apache/doris/analysis/DateLiteral.java|  5 -
 .../java/org/apache/doris/analysis/DateLiteralTest.java | 13 +
 2 files changed, 17 insertions(+), 1 deletion(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java 
b/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
index b1160a9118..789eca23b9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
@@ -48,6 +48,7 @@ import java.time.ZonedDateTime;
 import java.time.format.DateTimeFormatter;
 import java.time.format.DateTimeFormatterBuilder;
 import java.time.format.DateTimeParseException;
+import java.time.format.ResolverStyle;
 import java.time.format.TextStyle;
 import java.time.temporal.ChronoField;
 import java.time.temporal.TemporalAccessor;
@@ -415,7 +416,9 @@ public class DateLiteral extends LiteralExpr {
 builder.appendLiteral(":");
 }
 }
-DateTimeFormatter formatter = builder.toFormatter();
+// The default resolver style is 'SMART', which parses 
"2022-06-31" as "2022-06-30"
+// and does not throw an exception. 'STRICT' is used here.
+DateTimeFormatter formatter = 
builder.toFormatter().withResolverStyle(ResolverStyle.STRICT);
 dateTime = formatter.parse(s);
 parsed = true;
 }
diff --git 
a/fe/fe-core/src/test/java/org/apache/doris/analysis/DateLiteralTest.java 
b/fe/fe-core/src/test/java/org/apache/doris/analysis/DateLiteralTest.java
index 43271660e7..b0318b0e21 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/analysis/DateLiteralTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/analysis/DateLiteralTest.java
@@ -241,11 +241,24 @@ public class DateLiteralTest {
 Assert.assertEquals(2023, literal.getYear());
 Assert.assertEquals(6, literal.getMonth());
 Assert.assertEquals(1, literal.getDay());
+
+literal = new DateLiteral("2020-02-29", Type.DATEV2);
+Assert.assertEquals(2020, literal.getYear());
+Assert.assertEquals(2, literal.getMonth());
+Assert.assertEquals(29, literal.getDay());
 } catch (AnalysisException e) {
 e.printStackTrace();
 hasException = true;
 }
 Assert.assertFalse(hasException);
+
+try {
+new DateLiteral("2022-02-29", Type.DATEV2);
+} catch (AnalysisException e) {
+e.printStackTrace();
+hasException = true;
+}
+Assert.assertTrue(hasException);
 }
 
 @Test


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] stalary opened a new issue, #11583: [Bug](multi-catalog)

[GitBox]

stalary opened a new issue, #11583:
URL: https://github.com/apache/doris/issues/11583

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Version
   
   master
   
   ### What's Wrong?
   
   Remote metadata is always pulled twice.
   
   ### What You Expected?
   
   Only once.
   
   ### How to Reproduce?
   
   use doe. see log.
   
   ### Anything Else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris-website] branch master updated: export doc fix

[jiafengzheng]

This is an automated email from the ASF dual-hosted git repository.

jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git


The following commit(s) were added to refs/heads/master by this push:
 new bcf11a07aed export doc fix
bcf11a07aed is described below

commit bcf11a07aed8781335df4e39f3c01099ceba0f40
Author: jiafeng.zhang 
AuthorDate: Mon Aug 8 09:25:17 2022 +0800

export doc fix

export doc fix
---
 docs/data-operate/export/export-manual.md  | 27 +-
 .../current/data-operate/export/export-manual.md   | 25 +++-
 2 files changed, 50 insertions(+), 2 deletions(-)

diff --git a/docs/data-operate/export/export-manual.md 
b/docs/data-operate/export/export-manual.md
index 221f2e1eebf..541bc6f3c0d 100644
--- a/docs/data-operate/export/export-manual.md
+++ b/docs/data-operate/export/export-manual.md
@@ -26,7 +26,7 @@ under the License.
 
 # Data export
 
-Export is a function provided by Doris to export data. This function can 
export user-specified table or partition data in text format to remote storage 
through Broker process, such as HDFS/BOS.
+Export is a function provided by Doris to export data. This function can 
export user-specified table or partition data in text format to remote storage 
through Broker process, such as HDFS / Object storage (supports S3 protocol) 
etc.
 
 This document mainly introduces the basic principles, usage, best practices 
and precautions of Export.
 
@@ -106,6 +106,8 @@ For detailed usage of Export, please refer to [SHOW 
EXPORT](../../sql-manual/sql
 
 Export's detailed commands can be passed through `HELP EXPORT;` Examples are 
as follows:
 
+### Export to hdfs
+
 ```sql
 EXPORT TABLE db1.tbl1 
 PARTITION (p1,p2)
@@ -134,6 +136,29 @@ WITH BROKER "hdfs"
 * `timeout`: homework timeout. Default 2 hours. Unit seconds.
 * `tablet_num_per_task`: The maximum number of fragments allocated per query 
plan. The default is 5.
 
+### Export to object storage (supports S3 protocol)
+
+Create a repository named s3_repo to link cloud storage directly without going 
through the broker.
+
+```sql
+CREATE REPOSITORY `s3_repo`
+WITH S3
+ON LOCATION "s3://s3-repo"
+PROPERTIES
+(
+"AWS_ENDPOINT" = "http://s3-REGION.amazonaws.com";,
+"AWS_ACCESS_KEY" = "AWS_ACCESS_KEY",
+"AWS_SECRET_KEY"="AWS_SECRET_KEY",
+"AWS_REGION" = "REGION"
+);
+```
+
+- `AWS_ACCESS_KEY`/`AWS_SECRET_KEY`：Is your key to access the OSS API.
+- `AWS_ENDPOINT`：Endpoint indicates the access domain name of OSS external 
services.
+- `AWS_REGION`：Region indicates the region where the OSS data center is 
located.
+
+### View export status
+
 After submitting a job, the job status can be imported by querying the   [SHOW 
EXPORT](../../sql-manual/sql-reference/Show-Statements/SHOW-EXPORT.md)  
command. The results are as follows:
 
 ```sql
diff --git 
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/export/export-manual.md
 
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/export/export-manual.md
index 267061abd8e..fa687589dd3 100644
--- 
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/export/export-manual.md
+++ 
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/export/export-manual.md
@@ -26,7 +26,7 @@ under the License.
 
 # 数据导出
 
-数据导出（Export）是 Doris 提供的一种将数据导出的功能。该功能可以将用户指定的表或分区的数据，以文本的格式，通过 Broker 
进程导出到远端存储上，如 HDFS/BOS 等。
+数据导出（Export）是 Doris 提供的一种将数据导出的功能。该功能可以将用户指定的表或分区的数据，以文本的格式，通过 Broker 
进程导出到远端存储上，如 HDFS / 对象存储（支持S3协议） 等。
 
 本文档主要介绍 Export 的基本原理、使用方式、最佳实践以及注意事项。
 
@@ -126,6 +126,27 @@ WITH BROKER "hdfs"
 * `timeout`：作业超时时间。默认 2小时。单位秒。
 * `tablet_num_per_task`：每个查询计划分配的最大分片数。默认为 5。
 
+### 导出到对象存储
+
+创建名为 s3_repo 的仓库，直接链接云存储，而不通过broker.
+
+```sql
+CREATE REPOSITORY `s3_repo`
+WITH S3
+ON LOCATION "s3://s3-repo"
+PROPERTIES
+(
+"AWS_ENDPOINT" = "http://s3-REGION.amazonaws.com";,
+"AWS_ACCESS_KEY" = "AWS_ACCESS_KEY",
+"AWS_SECRET_KEY"="AWS_SECRET_KEY",
+"AWS_REGION" = "REGION"
+);
+```
+
+- `AWS_ACCESS_KEY`/`AWS_SECRET_KEY`：是您访问OSS API 的密钥.
+- `AWS_ENDPOINT`：表示OSS的数据中心所在的地域.
+- `AWS_REGION`：Endpoint表示OSS对外服务的访问域名.
+
 ### 查看导出状态
 
 提交作业后，可以通过  [SHOW 
EXPORT](../../sql-manual/sql-reference/Show-Statements/SHOW-EXPORT.md) 
命令查询导入作业状态。结果举例如下：
@@ -168,6 +189,8 @@ FinishTime: 2019-06-25 17:08:34
 * Timeout：作业超时时间。单位是秒。该时间从 CreateTime 开始计算。
 * ErrorMsg：如果作业出现错误，这里会显示错误原因。
 
+
+
 ## 最佳实践
 
 ### 查询计划的拆分


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] hf200012 opened a new pull request, #11584: [doc](fix)Export doc fix

[GitBox]

hf200012 opened a new pull request, #11584:
URL: https://github.com/apache/doris/pull/11584

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [x] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [x] No
   - [ ] No Need
   3. Has document been added or modified:
   - [x] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [x] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [x] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] deardeng commented on pull request #11580: [fix](storage-policy) fix bug that missing field when refreshing storage policy

[GitBox]

deardeng commented on PR #11580:
URL: https://github.com/apache/doris/pull/11580#issuecomment-1207554552

   LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11584: [doc](fix)Export doc fix

[GitBox]

github-actions[bot] commented on PR #11584:
URL: https://github.com/apache/doris/pull/11584#issuecomment-1207561184

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11584: [doc](fix)Export doc fix

[GitBox]

github-actions[bot] commented on PR #11584:
URL: https://github.com/apache/doris/pull/11584#issuecomment-1207561203

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11506: [improvement](doc)Description of bitmap type query result is null

[GitBox]

github-actions[bot] commented on PR #11506:
URL: https://github.com/apache/doris/pull/11506#issuecomment-1207561502

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11506: [improvement](doc)Description of bitmap type query result is null

[GitBox]

github-actions[bot] commented on PR #11506:
URL: https://github.com/apache/doris/pull/11506#issuecomment-1207561518

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] hf200012 merged pull request #11506: [improvement](doc)Description of bitmap type query result is null

[GitBox]

hf200012 merged PR #11506:
URL: https://github.com/apache/doris/pull/11506


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated (4f5db35990 -> 6ea3465264)

[jiafengzheng]

This is an automated email from the ASF dual-hosted git repository.

jiafengzheng pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


from 4f5db35990 [fix](date) fix the value may be changed during the parsing 
of date and datetime types (#11573)
 add 6ea3465264 [improvement](doc)Description of bitmap type query result 
is null (#11506)

No new revisions were added by this update.

Summary of changes:
 docs/en/docs/faq/sql-faq.md| 10 ++
 docs/zh-CN/docs/faq/sql-faq.md | 10 ++
 2 files changed, 20 insertions(+)


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11575: [fix](doc) remove docs for direct compiling on Centos

[GitBox]

github-actions[bot] commented on PR #11575:
URL: https://github.com/apache/doris/pull/11575#issuecomment-1207563136

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] hf200012 merged pull request #11575: [fix](doc) remove docs for direct compiling on Centos

[GitBox]

hf200012 merged PR #11575:
URL: https://github.com/apache/doris/pull/11575


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [fix](doc) remove docs for direct compiling on Centos (#11575)

[jiafengzheng]

This is an automated email from the ASF dual-hosted git repository.

jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 40b50400b2 [fix](doc) remove docs for direct compiling on Centos 
(#11575)
40b50400b2 is described below

commit 40b50400b2d1de9d68f5d2107c141e1bc3472e95
Author: Yongqiang YANG <98214048+dataroar...@users.noreply.github.com>
AuthorDate: Mon Aug 8 09:56:47 2022 +0800

[fix](doc) remove docs for direct compiling on Centos (#11575)

I tried to compile doris on Centos directly according to docs, however
it does not work. It is very difficult to find tools needed by doris
compilation on Centos.
---
 docs/en/docs/install/source-install/compilation.md | 24 +-
 .../docs/install/source-install/compilation.md | 22 +---
 2 files changed, 2 insertions(+), 44 deletions(-)

diff --git a/docs/en/docs/install/source-install/compilation.md 
b/docs/en/docs/install/source-install/compilation.md
index c836e84d5f..dec258d706 100644
--- a/docs/en/docs/install/source-install/compilation.md
+++ b/docs/en/docs/install/source-install/compilation.md
@@ -144,7 +144,7 @@ This document focuses on how to code Doris through source 
code.
 You can also create a Doris development environment mirror yourself, referring 
specifically to the `docker/README.md` file.
 
 
-## Direct Compilation (CentOS/Ubuntu)
+## Direct Compilation (Ubuntu)
 
 You can try to compile Doris directly in your own Linux environment.
 
@@ -179,28 +179,6 @@ You can try to compile Doris directly in your own Linux 
environment.
ln -s /usr/bin/g++-11 /usr/bin/g++
ln -s /usr/bin/gcc-11 /usr/bin/gcc
sudo apt-get install autoconf automake libtool autopoint
-   ```
-If you are using CentOS you can use the following command to install 
the dependencies
-
-   ```
-   sudo yum groupinstall 'Development Tools' && sudo yum install maven 
cmake byacc flex automake libtool bison binutils-devel zip unzip ncurses-devel 
curl git wget python2 glibc-static libstdc++-static java-1.8.0-openjdk
-   sudo yum install centos-release-scl
-   sudo yum install devtoolset-10
-   scl enable devtoolset-10 bash
-   ```
-   If devtoolset-10 is not found in current repo. Oracle has already 
rebuilt the devtoolset-10 packages. You can use this repo file:
-   ```
-   [ol7_software_collections]
-   name=Software Collection packages for Oracle Linux 7 ($basearch)
-   
baseurl=http://yum.oracle.com/repo/OracleLinux/OL7/SoftwareCollections/$basearch/
-   gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
-   gpgcheck=1
-   enabled=1
-   ```
-   After installation, set environment variables `PATH`, `JAVA_HOME`, etc.
-   > nit: you can find the jdk install directory by using command 
`alternatives --list`
-
-   Doris 0.14.0 will use gcc7 env to compile.
 
 2. Compile Doris
 
diff --git a/docs/zh-CN/docs/install/source-install/compilation.md 
b/docs/zh-CN/docs/install/source-install/compilation.md
index 85a313e6d8..02e685b4b3 100644
--- a/docs/zh-CN/docs/install/source-install/compilation.md
+++ b/docs/zh-CN/docs/install/source-install/compilation.md
@@ -143,7 +143,7 @@ under the License.
 你也可以自己创建一个 Doris 开发环境镜像，具体可参阅 `docker/README.md` 文件。
 
 
-## 直接编译（CentOS/Ubuntu）
+## 直接编译（Ubuntu）
 
 你可以在自己的 linux 环境中直接尝试编译 Doris。
 
@@ -174,26 +174,6 @@ under the License.
sudo apt-get install autoconf automake libtool autopoint
```
 
-  如果是CentOS 可以执行以下命令
-   ```
-   sudo yum groupinstall 'Development Tools' && sudo yum install maven 
cmake byacc flex automake libtool bison binutils-devel zip unzip ncurses-devel 
curl git wget python2 glibc-static libstdc++-static java-1.8.0-openjdk
-   sudo yum install centos-release-scl
-   sudo yum install devtoolset-10
-   scl enable devtoolset-10 bash
-   ```
-  如果当前仓库没有提供devtoolset-10 可以添加如下repo 使用oracle 提供 package
-   ```
-   [ol7_software_collections]
-   name=Software Collection packages for Oracle Linux 7 ($basearch)
-   
baseurl=http://yum.oracle.com/repo/OracleLinux/OL7/SoftwareCollections/$basearch/
-   gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
-   gpgcheck=1
-   enabled=1
-   ```
-
-   安装完成后，自行设置环境变量 `PATH`, `JAVA_HOME` 等。(可以通过`alternatives --list`命令找到jdk的安装目录)
-   注意： Doris 0.14.0 的版本仍然使用gcc7 的依赖编译，之后的代码将使用gcc10 的依赖
-
 2. 编译 Doris
 
 与使用 Docker 开发镜像编译一样，编译之前先检查是否支持avx2指令


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris-website] branch master updated: remove docs for direct compiling on Centos

[jiafengzheng]

This is an automated email from the ASF dual-hosted git repository.

jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git


The following commit(s) were added to refs/heads/master by this push:
 new 46e49bd1b56  remove docs for direct compiling on Centos
46e49bd1b56 is described below

commit 46e49bd1b5617bbae2e608761a2dc5aad4ebd741
Author: jiafeng.zhang 
AuthorDate: Mon Aug 8 09:57:24 2022 +0800

 remove docs for direct compiling on Centos

 remove docs for direct compiling on Centos
---
 docs/data-operate/export/export-manual.md  |  2 +-
 docs/install/source-install/compilation.md | 23 +-
 .../current/install/source-install/compilation.md  | 22 +
 3 files changed, 3 insertions(+), 44 deletions(-)

diff --git a/docs/data-operate/export/export-manual.md 
b/docs/data-operate/export/export-manual.md
index 541bc6f3c0d..915647b3039 100644
--- a/docs/data-operate/export/export-manual.md
+++ b/docs/data-operate/export/export-manual.md
@@ -112,7 +112,7 @@ Export's detailed commands can be passed through `HELP 
EXPORT;` Examples are as
 EXPORT TABLE db1.tbl1 
 PARTITION (p1,p2)
 [WHERE [expr]]
-TO "bos://bj-test-cmy/export/" 
+TO "hdfs://bj-test-cmy/export/" 
 PROPERTIES
 (
 "label"="mylabel",
diff --git a/docs/install/source-install/compilation.md 
b/docs/install/source-install/compilation.md
index 1bae27a9969..8c6c9d57b3d 100644
--- a/docs/install/source-install/compilation.md
+++ b/docs/install/source-install/compilation.md
@@ -184,28 +184,7 @@ You can try to compile Doris directly in your own Linux 
environment.
ln -s /usr/bin/gcc-11 /usr/bin/gcc
sudo apt-get install autoconf automake libtool autopoint
```
-If you are using CentOS you can use the following command to install 
the dependencies
-
-   ```
-   sudo yum groupinstall 'Development Tools' && sudo yum install maven 
cmake byacc flex automake libtool bison binutils-devel zip unzip ncurses-devel 
curl git wget python2 glibc-static libstdc++-static java-1.8.0-openjdk
-   sudo yum install centos-release-scl
-   sudo yum install devtoolset-10
-   scl enable devtoolset-10 bash
-   ```
-   If devtoolset-10 is not found in current repo. Oracle has already 
rebuilt the devtoolset-10 packages. You can use this repo file:
-   ```
-   [ol7_software_collections]
-   name=Software Collection packages for Oracle Linux 7 ($basearch)
-   
baseurl=http://yum.oracle.com/repo/OracleLinux/OL7/SoftwareCollections/$basearch/
-   gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
-   gpgcheck=1
-   enabled=1
-   ```
-   After installation, set environment variables `PATH`, `JAVA_HOME`, etc.
-   > nit: you can find the jdk install directory by using command 
`alternatives --list`
-
-   Doris 0.14.0 will use gcc7 env to compile.
-
+
 2. Compile Doris
 
Compiling with the Docker development image, check whether the avx2 
instruction is supported before compiling
diff --git 
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/source-install/compilation.md
 
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/source-install/compilation.md
index 5e37b676112..85a95749312 100644
--- 
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/source-install/compilation.md
+++ 
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/source-install/compilation.md
@@ -151,7 +151,7 @@ under the License.
 你也可以自己创建一个 Doris 开发环境镜像，具体可参阅 `docker/README.md` 文件。
 
 
-## 直接编译（CentOS/Ubuntu）
+## 直接编译（Ubuntu）
 
 你可以在自己的 linux 环境中直接尝试编译 Doris。
 
@@ -182,26 +182,6 @@ under the License.
sudo apt-get install autoconf automake libtool autopoint
```
 
-  如果是CentOS 可以执行以下命令
-   ```
-   sudo yum groupinstall 'Development Tools' && sudo yum install maven 
cmake byacc flex automake libtool bison binutils-devel zip unzip ncurses-devel 
curl git wget python2 glibc-static libstdc++-static java-1.8.0-openjdk
-   sudo yum install centos-release-scl
-   sudo yum install devtoolset-10
-   scl enable devtoolset-10 bash
-   ```
-  如果当前仓库没有提供devtoolset-10 可以添加如下repo 使用oracle 提供 package
-   ```
-   [ol7_software_collections]
-   name=Software Collection packages for Oracle Linux 7 ($basearch)
-   
baseurl=http://yum.oracle.com/repo/OracleLinux/OL7/SoftwareCollections/$basearch/
-   gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
-   gpgcheck=1
-   enabled=1
-   ```
-
-   安装完成后，自行设置环境变量 `PATH`, `JAVA_HOME` 等。(可以通过`alternatives --list`命令找到jdk的安装目录)
-   注意： Doris 0.14.0 的版本仍然使用gcc7 的依赖编译，之后的代码将使用gcc10 的依赖
-
 2. 编译 Doris
 
 与使用 Docker 开发镜像编译一样，编译之前先检查是否支持avx2指令


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] englefly closed pull request #11566: [fix](optimization) InferFiltersRule bug: a self inner join on a view, which contains where clause, will cause mis-inferrence.

[GitBox]

englefly closed pull request #11566: [fix](optimization)  InferFiltersRule bug: 
a self inner join on a view, which contains where clause, will cause 
mis-inferrence.
URL: https://github.com/apache/doris/pull/11566


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11547: [Bug](es) Fix es not support aliases error

[GitBox]

github-actions[bot] commented on PR #11547:
URL: https://github.com/apache/doris/pull/11547#issuecomment-1207576118

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11547: [Bug](es) Fix es not support aliases error

[GitBox]

github-actions[bot] commented on PR #11547:
URL: https://github.com/apache/doris/pull/11547#issuecomment-1207576162

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] xy720 opened a new pull request, #11585: [fix](array-type) Fix incorrect in function-set for array type

[GitBox]

xy720 opened a new pull request, #11585:
URL: https://github.com/apache/doris/pull/11585

   # Proposed changes
   
   There is some wrong logic in FunctionSet.java and it may causes potential 
risks for array functions invoke.
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [x] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [x] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [x] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [x] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [x] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morningman merged pull request #11488: [feature-wip](parquet-reader) add predicate filter and column reader

[GitBox]

morningman merged PR #11488:
URL: https://github.com/apache/doris/pull/11488


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated (40b50400b2 -> e8a344b683)

[morningman]

This is an automated email from the ASF dual-hosted git repository.

morningman pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


from 40b50400b2 [fix](doc) remove docs for direct compiling on Centos 
(#11575)
 add e8a344b683 [feature-wip](parquet-reader) add predicate filter and 
column reader (#11488)

No new revisions were added by this update.

Summary of changes:
 be/src/vec/exec/file_hdfs_scanner.cpp  |  28 ++-
 be/src/vec/exec/file_hdfs_scanner.h|  14 +-
 .../exec/format/parquet/vparquet_column_reader.cpp |  51 -
 .../exec/format/parquet/vparquet_column_reader.h   |  36 +++-
 .../exec/format/parquet/vparquet_group_reader.cpp  | 215 +++--
 .../exec/format/parquet/vparquet_group_reader.h|  53 +++--
 be/src/vec/exec/format/parquet/vparquet_reader.cpp |  53 +++--
 be/src/vec/exec/format/parquet/vparquet_reader.h   |  29 ++-
 be/test/CMakeLists.txt |   1 +
 .../exec/parquet/parquet_reader_test.cpp}  |  33 ++--
 10 files changed, 436 insertions(+), 77 deletions(-)
 copy be/test/{exprs/mock_vexpr.h => vec/exec/parquet/parquet_reader_test.cpp} 
(52%)


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] xy720 commented on pull request #11213: [feature-wip](array-type) support the array type in reverse function

[GitBox]

xy720 commented on PR #11213:
URL: https://github.com/apache/doris/pull/11213#issuecomment-1207580336

   Please rebase master


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] Gabriel39 opened a new pull request, #11586: [Improvement](debug message) add necessary info to DCHECK message

[GitBox]

Gabriel39 opened a new pull request, #11586:
URL: https://github.com/apache/doris/pull/11586

   # Proposed changes
   
   Sometimes we meet the error reporting we have wrong function return type. 
But now message is too poor to help us to find out the problem. So I think it's 
better to add the function name to debug message
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] dataroaring merged pull request #11541: [feature-wip](unique-key-merge-on-write) optimize rowid conversion and add ut

[GitBox]

dataroaring merged PR #11541:
URL: https://github.com/apache/doris/pull/11541


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated (e8a344b683 -> 1e6a3610a7)

[dataroaring]

This is an automated email from the ASF dual-hosted git repository.

dataroaring pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


from e8a344b683 [feature-wip](parquet-reader) add predicate filter and 
column reader (#11488)
 add 1e6a3610a7 [feature-wip](unique-key-merge-on-write) optimize rowid 
conversion and add ut (#11541)

No new revisions were added by this update.

Summary of changes:
 be/src/olap/iterators.h|   1 -
 be/src/olap/merger.cpp |  16 +-
 be/src/olap/rowid_conversion.h | 105 
 be/src/olap/rowset/rowset_reader.h |   1 -
 be/src/vec/olap/block_reader.cpp   |   5 +-
 be/src/vec/olap/vcollect_iterator.cpp  |  32 ++-
 be/src/vec/olap/vcollect_iterator.h|   5 +-
 be/test/olap/rowid_conversion_test.cpp | 447 -
 8 files changed, 532 insertions(+), 80 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yiguolei merged pull request #11086: [doc](tablet-health) modify content about tablet state

[GitBox]

yiguolei merged PR #11086:
URL: https://github.com/apache/doris/pull/11086


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [doc](tablet-health) modify content about tablet state (#11086)

[yiguolei]

This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new b93860902f [doc](tablet-health) modify content about tablet state 
(#11086)
b93860902f is described below

commit b93860902f203c365c7d0e82442cc4b43d139e15
Author: caiconghui <55968745+caicong...@users.noreply.github.com>
AuthorDate: Mon Aug 8 10:43:13 2022 +0800

[doc](tablet-health) modify content about tablet state (#11086)

Co-authored-by: caiconghui1 
---
 .../maint-monitor/tablet-repair-and-balance.md | 48 +++---
 .../maint-monitor/tablet-repair-and-balance.md | 40 +-
 2 files changed, 44 insertions(+), 44 deletions(-)

diff --git 
a/docs/en/docs/admin-manual/maint-monitor/tablet-repair-and-balance.md 
b/docs/en/docs/admin-manual/maint-monitor/tablet-repair-and-balance.md
index 44898313db..709800376c 100644
--- a/docs/en/docs/admin-manual/maint-monitor/tablet-repair-and-balance.md
+++ b/docs/en/docs/admin-manual/maint-monitor/tablet-repair-and-balance.md
@@ -259,42 +259,42 @@ Both replica repair and balancing are accomplished by 
replica copies between BEs
 
 In addition, by default, we provide two separate slots per disk for balancing 
tasks. The purpose is to prevent high-load nodes from losing space by balancing 
because slots are occupied by repair tasks.
 
-## Duplicate Status View
+## Tablet State View
 
-Duplicate status view mainly looks at the status of the duplicate, as well as 
the status of the duplicate repair and balancing tasks. Most of these states 
**exist only in** Master FE nodes. Therefore, the following commands need to be 
executed directly to Master FE.
+Tablet state view mainly looks at the state of the tablet, as well as the 
state of the tablet repair and balancing tasks. Most of these states **exist 
only in** Master FE nodes. Therefore, the following commands need to be 
executed directly to Master FE.
 
-### Duplicate status
+### Tablet state
 
 1. Global state checking
 
-   Through `SHOW PROC'/ statistic'; `commands can view the replica status 
of the entire cluster.
+   Through `SHOW PROC'/cluster_health/tablet_health'; `commands can view 
the replica status of the entire cluster.
 
-```
-   
+--+-+--+--+--+---+++---+
-   | DbId | DbName  | TableNum | PartitionNum | 
IndexNum | TabletNum | ReplicaNum | UnhealthyTabletNum | InconsistentTabletNum |
-   
+--+-+--+--+--+---+++---+
-   | 35153636 | default_cluster:DF_Newrisk  | 3| 3| 3  
  | 96| 288| 0  | 0 |
-   | 48297972 | default_cluster:PaperData   | 0| 0| 0  
  | 0 | 0  | 0  | 0 |
-   | 5909381  | default_cluster:UM_TEST | 7| 7| 10 
  | 320   | 960| 1  | 0 |
-   | Total| 240 | 10   | 10   | 13 
  | 416   | 1248   | 1  | 0 |
-   
+--+-+--+--+--+---+++---+
+ ```
+
+---++---++---+--+--+--++-+---+-+--+--+--+-+-+-++
+| DbId  | DbName | TabletNum | HealthyNum | 
ReplicaMissingNum | VersionIncompleteNum | ReplicaRelocatingNum | RedundantNum 
| ReplicaMissingInClusterNum | ReplicaMissingForTagNum | ForceRedundantNum | 
ColocateMismatchNum | ColocateRedundantNum | NeedFurtherRepairNum | 
UnrecoverableNum | ReplicaCompactionTooSlowNum | InconsistentNum | OversizeNum 
| CloningNum |
+
+---++---++---+--+--+--++-+---+-+--+--+--+-+-+-++
+| 10005 | default_cluster:doris_audit_db | 84| 84 | 0  
   | 0| 0| 0| 0 

[GitHub] [doris] yiguolei merged pull request #11538: [Bug](schema change) fix core dump on vectorized_alter_table

[GitBox]

yiguolei merged PR #11538:
URL: https://github.com/apache/doris/pull/11538


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [bugfix](schema change)fix core dump on vectorized_alter_table (#11538)

[yiguolei]

This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 2cd3bf80dc [bugfix](schema change)fix core dump on 
vectorized_alter_table (#11538)
2cd3bf80dc is described below

commit 2cd3bf80dcb3a4418228e97cb79b9673b7828cb2
Author: Pxl 
AuthorDate: Mon Aug 8 10:45:28 2022 +0800

[bugfix](schema change)fix core dump on vectorized_alter_table (#11538)
---
 be/src/common/config.h |   2 +-
 be/src/olap/schema_change.cpp  |  19 +--
 be/src/vec/exec/volap_scanner.cpp  |  16 +++---
 .../data/rollup/test_materialized_view_hll.out | Bin 0 -> 451 bytes
 .../rollup/test_materialized_view_hll.groovy   |  63 +
 5 files changed, 75 insertions(+), 25 deletions(-)

diff --git a/be/src/common/config.h b/be/src/common/config.h
index dbeebf41bc..258000e540 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -252,7 +252,7 @@ CONF_Bool(enable_low_cardinality_optimize, "true");
 CONF_mBool(disable_auto_compaction, "false");
 // whether enable vectorized compaction
 CONF_Bool(enable_vectorized_compaction, "true");
-// whether enable vectorized schema change, material-view or rollup task will 
fail if this config open.
+// whether enable vectorized schema change/material-view/rollup task.
 CONF_Bool(enable_vectorized_alter_table, "false");
 
 // check the configuration of auto compaction in seconds when auto compaction 
disabled
diff --git a/be/src/olap/schema_change.cpp b/be/src/olap/schema_change.cpp
index 6dc20fc6c5..7dfdc36117 100644
--- a/be/src/olap/schema_change.cpp
+++ b/be/src/olap/schema_change.cpp
@@ -817,7 +817,7 @@ Status RowBlockChanger::change_block(vectorized::Block* 
ref_block,
 vectorized::VExprContext* ctx = nullptr;
 RETURN_IF_ERROR(
 vectorized::VExpr::create_expr_tree(&pool, 
*_schema_mapping[idx].expr, &ctx));
-
+Defer defer {[&]() { ctx->close(state); }};
 RETURN_IF_ERROR(ctx->prepare(state, row_desc));
 RETURN_IF_ERROR(ctx->open(state));
 
@@ -834,8 +834,6 @@ Status RowBlockChanger::change_block(vectorized::Block* 
ref_block,
   
ref_block->get_by_position(result_column_id).column));
 }
 swap_idx_map[result_column_id] = idx;
-
-ctx->close(state);
 } else {
 // same type, just swap column
 swap_idx_map[ref_idx] = idx;
@@ -1632,15 +1630,10 @@ bool 
SchemaChangeWithSorting::_external_sorting(vector& src_row
 }
 rs_readers.push_back(rs_reader);
 }
-// get cur schema if rowset schema exist, rowset schema must be newer than 
tablet schema
-TabletSchemaSPtr cur_tablet_schema = 
src_rowsets.back()->rowset_meta()->tablet_schema();
-if (cur_tablet_schema == nullptr) {
-cur_tablet_schema = new_tablet->tablet_schema();
-}
 
 Merger::Statistics stats;
-auto res = Merger::merge_rowsets(new_tablet, READER_ALTER_TABLE, 
cur_tablet_schema, rs_readers,
- rowset_writer, &stats);
+auto res = Merger::merge_rowsets(new_tablet, READER_ALTER_TABLE, 
new_tablet->tablet_schema(),
+ rs_readers, rowset_writer, &stats);
 if (!res) {
 LOG(WARNING) << "failed to merge rowsets. tablet=" << 
new_tablet->full_name()
  << ", version=" << rowset_writer->version().first << "-"
@@ -1662,12 +1655,6 @@ Status 
VSchemaChangeWithSorting::_external_sorting(vector& src_
 rs_readers.push_back(rs_reader);
 }
 
-// get cur schema if rowset schema exist, rowset schema must be newer than 
tablet schema
-auto cur_tablet_schema = 
src_rowsets.back()->rowset_meta()->tablet_schema();
-if (cur_tablet_schema == nullptr) {
-cur_tablet_schema = new_tablet->tablet_schema();
-}
-
 Merger::Statistics stats;
 RETURN_IF_ERROR(Merger::vmerge_rowsets(new_tablet, READER_ALTER_TABLE,
new_tablet->tablet_schema(), 
rs_readers, rowset_writer,
diff --git a/be/src/vec/exec/volap_scanner.cpp 
b/be/src/vec/exec/volap_scanner.cpp
index f7ca037e31..c032aee301 100644
--- a/be/src/vec/exec/volap_scanner.cpp
+++ b/be/src/vec/exec/volap_scanner.cpp
@@ -104,7 +104,7 @@ Status VOlapScanner::prepare(
 ss << "failed to initialize storage reader. tablet=" << 
_tablet->full_name()
<< ", res=" << acquire_reader_st
<< ", backend=" << BackendOptions::get_localhost();
-return Status::InternalError(ss.str().c_str());
+return Status::InternalError(ss.str());
 }
 }
 }
@@ -134,7 +134,7 @@ Status VOlapScanner::open() {
 ss << "failed to initialize sto

[GitHub] [doris] yiguolei closed issue #11537: [Bug] core dump on create materialized view

[GitBox]

yiguolei closed issue #11537: [Bug] core dump on create materialized view
URL: https://github.com/apache/doris/issues/11537


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11586: [Improvement](debug message) add necessary info to DCHECK message

[GitBox]

github-actions[bot] commented on PR #11586:
URL: https://github.com/apache/doris/pull/11586#issuecomment-1207594017

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11586: [Improvement](debug message) add necessary info to DCHECK message

[GitBox]

github-actions[bot] commented on PR #11586:
URL: https://github.com/apache/doris/pull/11586#issuecomment-1207594035

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yiguolei commented on a diff in pull request #11266: [feature](information_schema) add 'segments' table into information_s…

[GitBox]

yiguolei commented on code in PR #11266:
URL: https://github.com/apache/doris/pull/11266#discussion_r939784668


##
fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java:
##
@@ -1699,7 +1699,11 @@ private PlanNode createScanNode(Analyzer analyzer, 
TableRef tblRef, SelectStmt s
 scanNode = new MysqlScanNode(ctx.getNextNodeId(), 
tblRef.getDesc(), (MysqlTable) tblRef.getTable());
 break;
 case SCHEMA:
-scanNode = new SchemaScanNode(ctx.getNextNodeId(), 
tblRef.getDesc());
+if 
(BackendSchemaScanNode.isBackendSchemaTable(tblRef.getDesc().getTable().getName()))
 {

Review Comment:
   BackendPartitionedSchemaScanNode



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris-flink-connector] branch release-1.2.0 created (now f19c2b3)

[diwu]

This is an automated email from the ASF dual-hosted git repository.

diwu pushed a change to branch release-1.2.0
in repository https://gitbox.apache.org/repos/asf/doris-flink-connector.git


  at f19c2b3  [docs] Fix broken link for doris connector docs (#51)

No new revisions were added by this update.


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] carlvinhust2012 commented on pull request #11213: [feature-wip](array-type) support the array type in reverse function

[GitBox]

carlvinhust2012 commented on PR #11213:
URL: https://github.com/apache/doris/pull/11213#issuecomment-1207602588

   > Please rebase master
   
   done


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] LemonLiTree opened a new pull request, #11587: [doc](bloomFilter)fix-doc

[GitBox]

LemonLiTree opened a new pull request, #11587:
URL: https://github.com/apache/doris/pull/11587

   fix-doc


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11571: [Enhancement](hdfs) Support loading hdfs config from hdfs-site.xml

[GitBox]

github-actions[bot] commented on PR #11571:
URL: https://github.com/apache/doris/pull/11571#issuecomment-1207607766

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11571: [Enhancement](hdfs) Support loading hdfs config from hdfs-site.xml

[GitBox]

github-actions[bot] commented on PR #11571:
URL: https://github.com/apache/doris/pull/11571#issuecomment-1207607782

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-website] Jeffrey-SelectDB opened a new pull request, #32: [feature] Add OSS upload & PWA support

[GitBox]

Jeffrey-SelectDB opened a new pull request, #32:
URL: https://github.com/apache/doris-website/pull/32

   - [[feature] add oss 
upload](https://github.com/apache/doris-website/commit/b2cecae1017b21426f897e480fadad10c9400c0e)
   
   - [[PWA] add pwa 
support](https://github.com/apache/doris-website/commit/eb6f5dcee72109a963494bcac3bfc00f4ede524d)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] wuyunfeng commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error

[GitBox]

wuyunfeng commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r939795388


##
fe/fe-core/src/main/java/org/apache/doris/datasource/EsExternalDataSource.java:
##
@@ -166,7 +167,10 @@ public List listDatabaseNames(SessionContext ctx) {
 
 @Override
 public List listTableNames(SessionContext ctx, String dbName) {
-return esRestClient.getIndexes();
+List indexes = 
esRestClient.getIndexes().stream().distinct().collect(Collectors.toList());

Review Comment:
   I think maybe package this line into esRestClient is better, provide 
`listTable` method



##
fe/fe-core/src/main/java/org/apache/doris/datasource/EsExternalDataSource.java:
##
@@ -37,14 +37,15 @@
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
+import java.util.stream.Collectors;
 
 /**
  * External data source for elasticsearch
  */
 @Getter
 public class EsExternalDataSource extends ExternalDataSource {
 
-public static final String DEFAULT_DB = "default";
+public static final String DEFAULT_DB = "default_es_db";

Review Comment:
   can we use the  `cluster_name`_`uuid` for DB?



##
fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/EsUtil.java:
##
@@ -168,7 +171,7 @@ public static JSONObject getMappingProps(String 
sourceIndex, String indexMapping
 if (rootSchema == null) {
 properties = (JSONObject) mappings.get("properties");
 // Compatible es6 with no type passed in.
-if (mappingType == null) {
+if (mappingType == null && properties == null) {

Review Comment:
   && properties == null maybe not necessary， you check the context again
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] wuyunfeng commented on a diff in pull request #11340: [Feature](array) doe support array

[GitBox]

wuyunfeng commented on code in PR #11340:
URL: https://github.com/apache/doris/pull/11340#discussion_r939796674


##
fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/EsUtil.java:
##
@@ -132,16 +133,33 @@ public static boolean getBoolean(Map 
properties, String name) th
 }
 }
 
-/**
- * Get mapping properties JSONObject.
- **/
-public static JSONObject getMappingProps(String sourceIndex, String 
indexMapping, String mappingType) {
+public static List getArrayFields(String indexMapping) {
+JSONObject mappings = getMapping(indexMapping);
+if (!mappings.containsKey("_meta")) {
+return new ArrayList<>();
+}
+JSONObject meta = (JSONObject) mappings.get("_meta");
+if (!meta.containsKey("doris")) {
+return new ArrayList<>();
+}
+JSONObject dorisMeta = (JSONObject) meta.get("doris");

Review Comment:
   i do not think add the name `doris`to Ealsticsearch index meta  is suitable



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-website] Jeffrey-SelectDB closed pull request #32: [feature] Add OSS upload & PWA support

[GitBox]

Jeffrey-SelectDB closed pull request #32: [feature] Add OSS upload & PWA support
URL: https://github.com/apache/doris-website/pull/32


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morrySnow commented on a diff in pull request #11531: [feature](Nereids):refactor and add outer join LAsscom.

[GitBox]

morrySnow commented on code in PR #11531:
URL: https://github.com/apache/doris/pull/11531#discussion_r939797579


##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinCommute.java:
##
@@ -29,6 +29,9 @@
  */
 @Developing
 public class JoinCommute extends OneExplorationRuleFactory {
+
+public static final JoinCommute SWAP_OUTER_COMMUTE_BOTTOM_JOIN = new 
JoinCommute(true, SwapType.BOTTOM_JOIN);

Review Comment:
   not use anymore?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morrySnow commented on pull request #11566: [fix](optimization) InferFiltersRule bug: a self inner join on a view, which contains where clause, will cause mis-inferrence.

[GitBox]

morrySnow commented on PR #11566:
URL: https://github.com/apache/doris/pull/11566#issuecomment-1207637164

   typo in title: inferrence -> inference


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] stalary commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error

[GitBox]

stalary commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r93987


##
fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/EsUtil.java:
##
@@ -168,7 +171,7 @@ public static JSONObject getMappingProps(String 
sourceIndex, String indexMapping
 if (rootSchema == null) {
 properties = (JSONObject) mappings.get("properties");
 // Compatible es6 with no type passed in.
-if (mappingType == null) {
+if (mappingType == null && properties == null) {

Review Comment:
   The logic that es6 does not pass type is special, and this should not be 
entered in es7 or later.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] stalary commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error

[GitBox]

stalary commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r939811286


##
fe/fe-core/src/main/java/org/apache/doris/datasource/EsExternalDataSource.java:
##
@@ -37,14 +37,15 @@
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
+import java.util.stream.Collectors;
 
 /**
  * External data source for elasticsearch
  */
 @Getter
 public class EsExternalDataSource extends ExternalDataSource {
 
-public static final String DEFAULT_DB = "default";
+public static final String DEFAULT_DB = "default_es_db";

Review Comment:
   A catalog can only support one cluster, so it would be easier to use a 
static name.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] stalary commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error

[GitBox]

stalary commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r939811382


##
fe/fe-core/src/main/java/org/apache/doris/datasource/EsExternalDataSource.java:
##
@@ -166,7 +167,10 @@ public List listDatabaseNames(SessionContext ctx) {
 
 @Override
 public List listTableNames(SessionContext ctx, String dbName) {
-return esRestClient.getIndexes();
+List indexes = 
esRestClient.getIndexes().stream().distinct().collect(Collectors.toList());

Review Comment:
   Good idea. Let me modify it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-website] Jeffrey-SelectDB opened a new pull request, #33: [CDN] Upload files to OSS

[GitBox]

Jeffrey-SelectDB opened a new pull request, #33:
URL: https://github.com/apache/doris-website/pull/33

   The Doris website is too slow to access in China. We have stored static 
resources on the CDN to speed up access.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] eldenmoon commented on pull request #11564: [Fix](stream-load-json) fix VJsonReader::_write_data_to_column invali…

[GitBox]

eldenmoon commented on PR #11564:
URL: https://github.com/apache/doris/pull/11564#issuecomment-1207656546

   @xy720 I will add a regression-test for this fix


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] stalary commented on a diff in pull request #11340: [Feature](array) doe support array

[GitBox]

stalary commented on code in PR #11340:
URL: https://github.com/apache/doris/pull/11340#discussion_r939816612


##
fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/EsUtil.java:
##
@@ -132,16 +133,33 @@ public static boolean getBoolean(Map 
properties, String name) th
 }
 }
 
-/**
- * Get mapping properties JSONObject.
- **/
-public static JSONObject getMappingProps(String sourceIndex, String 
indexMapping, String mappingType) {
+public static List getArrayFields(String indexMapping) {
+JSONObject mappings = getMapping(indexMapping);
+if (!mappings.containsKey("_meta")) {
+return new ArrayList<>();
+}
+JSONObject meta = (JSONObject) mappings.get("_meta");
+if (!meta.containsKey("doris")) {
+return new ArrayList<>();
+}
+JSONObject dorisMeta = (JSONObject) meta.get("doris");

Review Comment:
   In the automatic table creation scenario, users are required to create 
special mappings for the array fields. trino use it `JsonNode metaNode = 
nullSafeNode(mappings, "_meta");
   JsonNode metaProperties = nullSafeNode(metaNode, "trino");`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] freemandealer commented on issue #10095: [Bug] Error occurs when showing query profile

[GitBox]

freemandealer commented on issue #10095:
URL: https://github.com/apache/doris/issues/10095#issuecomment-1207658406

   same issus here.
   
   I was tring the doris query profile facility.
   
   after enable profile using `SET is_report_success=true;`
   
   then execute select statements
   
   `show query profile "/"\G` gives me:
   ```
   *** 1. row ***
  QueryId: e4a384b03c9c4aaa-ab5fcd6a7299f6db
 User: root
DefaultDb: default_cluster:testdb
  SQL: select id,actor_login from github_1 limit 10
QueryType: Query
StartTime: 2022-08-08 12:28:22
  EndTime: 2022-08-08 12:28:23
TotalTime: 46ms
   QueryState: EOF
   ```
   
   so I send `show query profile "/e4a384b03c9c4aaa-ab5fcd6a7299f6db"\G`
   
   return error:
   ```
   ERROR 1105 (HY000): errCode = 2, detailMessage = failed to get fragment 
profile tree. err: errCode = 2, detailMessage = Invalid instance profile, 
without sender or exec node: Instance e4a384b03c9c4aaa-ab5fcd6a7299f6dc 
(host=TNetworkAddress(hostname:172.16.70.241, port:9061)):
   ```
   
   All 4 BE processes are running correctly, without WARNING log.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [feature-wip](parquet-reader)decode parquet data (#11536)

[morningman]

This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 37d1180cca [feature-wip](parquet-reader)decode parquet data (#11536)
37d1180cca is described below

commit 37d1180cca26494f65ec6074ab68a4350cf083e6
Author: Ashin Gau 
AuthorDate: Mon Aug 8 12:44:06 2022 +0800

[feature-wip](parquet-reader)decode parquet data (#11536)
---
 be/src/common/config.h |   1 +
 be/src/io/buffered_reader.cpp  |  19 ++-
 be/src/io/buffered_reader.h|  22 ++--
 be/src/util/block_compression.cpp  |  30 +
 be/src/util/block_compression.h|   4 +
 be/src/util/rle_encoding.h |  56 +
 be/src/vec/CMakeLists.txt  |   4 +-
 be/src/vec/exec/format/parquet/level_decoder.cpp   |  76 
 be/src/vec/exec/format/parquet/level_decoder.h |  61 ++
 be/src/vec/exec/format/parquet/parquet_common.cpp  |  60 ++
 be/src/vec/exec/format/parquet/parquet_common.h| 108 +
 be/src/vec/exec/format/parquet/schema_desc.cpp |  11 +-
 .../parquet/vparquet_column_chunk_reader.cpp   | 130 +++--
 .../format/parquet/vparquet_column_chunk_reader.h  | 111 +-
 .../exec/format/parquet/vparquet_page_reader.cpp   |  30 +++--
 .../vec/exec/format/parquet/vparquet_page_reader.h |  16 ++-
 .../test_data/parquet_scanner/type-decoder.parquet | Bin 0 -> 338 bytes
 be/test/vec/exec/parquet/parquet_thrift_test.cpp   |  55 +
 18 files changed, 734 insertions(+), 60 deletions(-)

diff --git a/be/src/common/config.h b/be/src/common/config.h
index 258000e540..59e56bcde3 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -792,6 +792,7 @@ CONF_Int32(object_pool_buffer_size, "100");
 // ParquetReaderWrap prefetch buffer size
 CONF_Int32(parquet_reader_max_buffer_size, "50");
 CONF_Bool(parquet_predicate_push_down, "true");
+CONF_Int32(parquet_header_max_size, "8388608");
 
 // When the rows number reached this limit, will check the filter rate the of 
bloomfilter
 // if it is lower than a specific threshold, the predicate will be disabled.
diff --git a/be/src/io/buffered_reader.cpp b/be/src/io/buffered_reader.cpp
index 8e2446b9e1..ca40979321 100644
--- a/be/src/io/buffered_reader.cpp
+++ b/be/src/io/buffered_reader.cpp
@@ -185,10 +185,11 @@ bool BufferedReader::closed() {
 return _reader->closed();
 }
 
-BufferedFileStreamReader::BufferedFileStreamReader(FileReader* file, int64_t 
offset, int64_t length)
+BufferedFileStreamReader::BufferedFileStreamReader(FileReader* file, uint64_t 
offset,
+   uint64_t length)
 : _file(file), _file_start_offset(offset), _file_end_offset(offset + 
length) {}
 
-Status BufferedFileStreamReader::seek(int64_t position) {
+Status BufferedFileStreamReader::seek(uint64_t position) {
 if (_file_position != position) {
 RETURN_IF_ERROR(_file->seek(position));
 _file_position = position;
@@ -196,8 +197,8 @@ Status BufferedFileStreamReader::seek(int64_t position) {
 return Status::OK();
 }
 
-Status BufferedFileStreamReader::read_bytes(const uint8_t** buf, int64_t 
offset,
-int64_t* bytes_to_read) {
+Status BufferedFileStreamReader::read_bytes(const uint8_t** buf, uint64_t 
offset,
+size_t* bytes_to_read) {
 if (offset < _file_start_offset) {
 return Status::IOError("Out-of-bounds Access");
 }
@@ -230,19 +231,15 @@ Status BufferedFileStreamReader::read_bytes(const 
uint8_t** buf, int64_t offset,
 RETURN_IF_ERROR(seek(_buf_end_offset));
 bool eof = false;
 int64_t buf_remaining = _buf_end_offset - _buf_start_offset;
-RETURN_IF_ERROR(
-_file->read(_buf.get() + buf_remaining, _buf_size - buf_remaining, 
&to_read, &eof));
+RETURN_IF_ERROR(_file->read(_buf.get() + buf_remaining, to_read, &to_read, 
&eof));
 *bytes_to_read = buf_remaining + to_read;
 _buf_end_offset += to_read;
 *buf = _buf.get();
 return Status::OK();
 }
 
-Status BufferedFileStreamReader::read_bytes(Slice& slice, int64_t offset) {
-int64_t bytes_to_read = slice.size;
-Status st = read_bytes((const uint8_t**)&slice.data, offset, 
&bytes_to_read);
-slice.size = bytes_to_read;
-return st;
+Status BufferedFileStreamReader::read_bytes(Slice& slice, uint64_t offset) {
+return read_bytes((const uint8_t**)&slice.data, offset, &slice.size);
 }
 
 } // namespace doris
diff --git a/be/src/io/buffered_reader.h b/be/src/io/buffered_reader.h
index d4a5f37927..2cfcaaa413 100644
--- a/be/src/io/buffered_reader.h
+++ b/be/src/io/buffered_reader.h
@@ -93,34 +93,34 @@ public:
  * @param of

[GitHub] [doris] morningman commented on pull request #11536: [feature-wip](parquet-reader)decode parquet data

[GitBox]

morningman commented on PR #11536:
URL: https://github.com/apache/doris/pull/11536#issuecomment-1207659539

   quick merge for test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morningman merged pull request #11536: [feature-wip](parquet-reader)decode parquet data

[GitBox]

morningman merged PR #11536:
URL: https://github.com/apache/doris/pull/11536


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] HappenLee commented on a diff in pull request #11468: [Refactor](push-down predicate) Derive push-down predicate from vconjuncts

[GitBox]

HappenLee commented on code in PR #11468:
URL: https://github.com/apache/doris/pull/11468#discussion_r939826917


##
be/src/vec/exec/volap_scan_node.cpp:
##
@@ -966,152 +794,83 @@ static bool ignore_cast(SlotDescriptor* slot, Expr* 
expr) {
 return false;
 }
 
-bool VOlapScanNode::should_push_down_in_predicate(doris::SlotDescriptor* slot,
-  doris::InPredicate* pred) {
-if (Expr::type_without_cast(pred->get_child(0)) != 
TExprNodeType::SLOT_REF) {
-// not a slot ref(column)
-return false;
-}
-
-std::vector slot_ids;
-if (pred->get_child(0)->get_slot_ids(&slot_ids) != 1) {
-// not a single column predicate
-return false;
-}
-
-if (slot_ids[0] != slot->id()) {
-// predicate not related to current column
-return false;
-}
-
-if (pred->get_child(0)->type().type != slot->type().type) {
-if (!ignore_cast(slot, pred->get_child(0))) {
-// the type of predicate not match the slot's type
-return false;
-}
-}
-
-VLOG_CRITICAL << slot->col_name() << " fixed_values add num: " << 
pred->hybrid_set()->size();
-
-// if there are too many elements in InPredicate, exceed the limit,
-// we will not push any condition of this column to storage engine.
-// because too many conditions pushed down to storage engine may even
-// slow down the query process.
-// ATTN: This is just an experience value. You may need to try
-// different thresholds to improve performance.
-if (pred->hybrid_set()->size() > _max_pushdown_conditions_per_column) {
-VLOG_NOTICE << "Predicate value num " << pred->hybrid_set()->size() << 
" exceed limit "
-<< _max_pushdown_conditions_per_column;
-return false;
-}
-
-return true;
-}
-
-std::pair 
VOlapScanNode::should_push_down_eq_predicate(doris::SlotDescriptor* slot,
-
doris::Expr* pred, int conj_idx,
-int 
child_idx) {
-auto result_pair = std::make_pair(false, nullptr);
-
-// Do not get slot_ref of column, should not push_down to Storage Engine
-if (Expr::type_without_cast(pred->get_child(child_idx)) != 
TExprNodeType::SLOT_REF) {
-return result_pair;
-}
-
-std::vector slot_ids;
-if (pred->get_child(child_idx)->get_slot_ids(&slot_ids) != 1) {
-// not a single column predicate
-return result_pair;
-}
-
-if (slot_ids[0] != slot->id()) {
-// predicate not related to current column
-return result_pair;
-}
-
-if (pred->get_child(child_idx)->type().type != slot->type().type) {
-if (!ignore_cast(slot, pred->get_child(child_idx))) {
-// the type of predicate not match the slot's type
-return result_pair;
+template 
+Status VOlapScanNode::change_value_range(ColumnValueRange& 
temp_range, void* value,
+ const ChangeFixedValueRangeFunc& func,
+ std::string& fn_name, int 
slot_ref_child) {

Review Comment:
   not change const std::string



##
be/src/vec/exec/volap_scan_node.cpp:
##
@@ -2019,4 +1342,534 @@ Status VOlapScanNode::get_hints(TabletSharedPtr table, 
const TPaloScanRange& sca
 return Status::OK();
 }
 
+template 
+bool VOlapScanNode::_should_push_down_in_predicate(VInPredicate* pred, 
VExprContext* expr_ctx) {
+if (pred->is_not_in() != IsNotIn) {
+return false;
+}
+InState* state = reinterpret_cast(
+expr_ctx->fn_context(pred->fn_context_index())
+->get_function_state(FunctionContext::FRAGMENT_LOCAL));
+HybridSetBase* set = state->hybrid_set.get();
+
+// if there are too many elements in InPredicate, exceed the limit,
+// we will not push any condition of this column to storage engine.
+// because too many conditions pushed down to storage engine may even
+// slow down the query process.
+// ATTN: This is just an experience value. You may need to try
+// different thresholds to improve performance.
+if (set->size() > _max_pushdown_conditions_per_column) {
+VLOG_NOTICE << "Predicate value num " << set->size() << " exceed limit 
"
+<< _max_pushdown_conditions_per_column;
+return false;
+}
+return true;
+}
+
+bool VOlapScanNode::_should_push_down_function_filter(VectorizedFnCall* 
fn_call,
+  VExprContext* expr_ctx,
+  std::string* 
constant_str,
+  
doris_udf::FunctionContext** fn_ctx) {
+// Now only `like` function filters is supported to push down
+if (fn_call->fn().name.function_name != "like") {
+return 

[GitHub] [doris] wuyunfeng commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error

[GitBox]

wuyunfeng commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r939828810


##
fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/EsUtil.java:
##
@@ -168,7 +171,7 @@ public static JSONObject getMappingProps(String 
sourceIndex, String indexMapping
 if (rootSchema == null) {
 properties = (JSONObject) mappings.get("properties");
 // Compatible es6 with no type passed in.
-if (mappingType == null) {
+if (mappingType == null && properties == null) {

Review Comment:
   OK，we should change the location 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] stalary commented on a diff in pull request #11547: [Bug](es) Fix es not support aliases error

[GitBox]

stalary commented on code in PR #11547:
URL: https://github.com/apache/doris/pull/11547#discussion_r939838763


##
fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/EsUtil.java:
##
@@ -168,7 +171,7 @@ public static JSONObject getMappingProps(String 
sourceIndex, String indexMapping
 if (rootSchema == null) {
 properties = (JSONObject) mappings.get("properties");
 // Compatible es6 with no type passed in.
-if (mappingType == null) {
+if (mappingType == null && properties == null) {

Review Comment:
   Okay, so I'm going to refactor this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-flink-connector] JNSimba opened a new issue, #52: [Feature] Flink Doris Connector Release Note 1.2.0

[GitBox]

JNSimba opened a new issue, #52:
URL: https://github.com/apache/doris-flink-connector/issues/52

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Description
   
   1.Support flink1.15 read and write
   
   ### Use case
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] caiconghui closed issue #11570: [Enhancement] Support loading hdfs config from hdfs_site.xml

[GitBox]

caiconghui closed issue #11570: [Enhancement] Support loading hdfs config from 
hdfs_site.xml
URL: https://github.com/apache/doris/issues/11570


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] caiconghui merged pull request #11571: [Enhancement](hdfs) Support loading hdfs config from hdfs-site.xml

[GitBox]

caiconghui merged PR #11571:
URL: https://github.com/apache/doris/pull/11571


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [Enhancement](hdfs) Support loading hdfs config from hdfs-site.xml (#11571)

[caiconghui]

This is an automated email from the ASF dual-hosted git repository.

caiconghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 411254c128 [Enhancement](hdfs) Support loading hdfs config from 
hdfs-site.xml (#11571)
411254c128 is described below

commit 411254c128d6cbe6cb18b29b2e54109306c32a08
Author: caiconghui <55968745+caicong...@users.noreply.github.com>
AuthorDate: Mon Aug 8 14:18:28 2022 +0800

[Enhancement](hdfs) Support loading hdfs config from hdfs-site.xml (#11571)
---
 bin/start_fe.sh|  2 +-
 build.sh   |  1 +
 conf/hdfs-site.xml | 23 ++
 .../java/org/apache/doris/backup/HdfsStorage.java  |  3 ++-
 .../doris/catalog/HiveMetaStoreClientHelper.java   |  5 +++--
 .../org/apache/doris/common/util/BrokerUtil.java   |  3 ++-
 .../apache/doris/external/iceberg/HiveCatalog.java |  3 ++-
 .../org/apache/doris/planner/HudiScanNode.java |  3 ++-
 .../planner/external/ExternalHiveScanProvider.java |  3 ++-
 9 files changed, 38 insertions(+), 8 deletions(-)

diff --git a/bin/start_fe.sh b/bin/start_fe.sh
index 684e8630b0..5421e495f7 100755
--- a/bin/start_fe.sh
+++ b/bin/start_fe.sh
@@ -147,7 +147,7 @@ echo $final_java_opt >> $LOG_DIR/fe.out
 for f in $DORIS_HOME/lib/*.jar; do
 CLASSPATH=$f:${CLASSPATH}
 done
-export CLASSPATH=${CLASSPATH}:${DORIS_HOME}/lib
+export CLASSPATH=${CLASSPATH}:${DORIS_HOME}/lib:${DORIS_HOME}/conf
 
 pidfile=$PID_DIR/fe.pid
 
diff --git a/build.sh b/build.sh
index e35f8b08fe..ede250 100755
--- a/build.sh
+++ b/build.sh
@@ -384,6 +384,7 @@ if [ ${BUILD_FE} -eq 1 ]; then
 
 cp -r -p ${DORIS_HOME}/bin/*_fe.sh ${DORIS_OUTPUT}/fe/bin/
 cp -r -p ${DORIS_HOME}/conf/fe.conf ${DORIS_OUTPUT}/fe/conf/
+cp -r -p ${DORIS_HOME}/conf/*.xml ${DORIS_OUTPUT}/fe/conf/
 rm -rf ${DORIS_OUTPUT}/fe/lib/*
 cp -r -p ${DORIS_HOME}/fe/fe-core/target/lib/* ${DORIS_OUTPUT}/fe/lib/
 rm -f ${DORIS_OUTPUT}/fe/lib/palo-fe.jar
diff --git a/conf/hdfs-site.xml b/conf/hdfs-site.xml
new file mode 100644
index 00..32235bf8bc
--- /dev/null
+++ b/conf/hdfs-site.xml
@@ -0,0 +1,23 @@
+
+
+
+
+
+
+
+
diff --git a/fe/fe-core/src/main/java/org/apache/doris/backup/HdfsStorage.java 
b/fe/fe-core/src/main/java/org/apache/doris/backup/HdfsStorage.java
index 41128a0378..5090d7c67f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/backup/HdfsStorage.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/backup/HdfsStorage.java
@@ -30,6 +30,7 @@ import org.apache.hadoop.fs.FSDataOutputStream;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.HdfsConfiguration;
 import org.apache.hadoop.security.UserGroupInformation;
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
@@ -86,7 +87,7 @@ public class HdfsStorage extends BlobStorage {
 checkHDFS(caseInsensitiveProperties);
 String hdfsFsName = 
caseInsensitiveProperties.get(BrokerUtil.HADOOP_FS_NAME).toString();
 String username = 
caseInsensitiveProperties.get(BrokerUtil.HADOOP_USER_NAME).toString();
-Configuration conf = new Configuration();
+Configuration conf = new HdfsConfiguration();
 boolean isSecurityEnabled = false;
 for (Map.Entry propEntry : 
caseInsensitiveProperties.entrySet()) {
 conf.set(propEntry.getKey(), propEntry.getValue());
diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java
 
b/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java
index 6553e99b3a..8d95cad038 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java
@@ -45,6 +45,7 @@ import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.LocatedFileStatus;
 import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.hdfs.HdfsConfiguration;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.metastore.HiveMetaStoreClient;
 import org.apache.hadoop.hive.metastore.api.FieldSchema;
@@ -206,7 +207,7 @@ public class HiveMetaStoreClientHelper {
 
 // create Configuration for the given properties
 private static Configuration getConfiguration(Map 
properties, boolean onS3) {
-Configuration configuration = new Configuration(false);
+Configuration configuration = new HdfsConfiguration();
 for (Map.Entry entry : properties.entrySet()) {
 if (!entry.getKey().equals(HiveTable.HIVE_METASTORE_URIS)) {
 configuration.set(entry.getKey(), entry.getValue());
@@ -347,7 +348,7 

[GitHub] [doris] github-actions[bot] commented on pull request #11552: [improvement](datax) improvement json import and support csv writing

[GitBox]

github-actions[bot] commented on PR #11552:
URL: https://github.com/apache/doris/pull/11552#issuecomment-1207719075

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #11552: [improvement](datax) improvement json import and support csv writing

[GitBox]

github-actions[bot] commented on PR #11552:
URL: https://github.com/apache/doris/pull/11552#issuecomment-1207719107

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] pengxiangyu closed pull request #10405: Add MigrationHandler

[GitBox]

pengxiangyu closed pull request #10405: Add MigrationHandler
URL: https://github.com/apache/doris/pull/10405


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] carlvinhust2012 commented on a diff in pull request #11406: [feature-wip](array-type) add the array_join function

[GitBox]

carlvinhust2012 commented on code in PR #11406:
URL: https://github.com/apache/doris/pull/11406#discussion_r939879767


##
be/src/vec/functions/array/function_array_join.h:
##
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#pragma once
+
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_const.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/functions/function.h"
+#include "vec/functions/function_helpers.h"
+
+namespace doris::vectorized {
+
+class FunctionArrayJoin : public IFunction {
+public:
+static constexpr auto name = "array_join";
+static FunctionPtr create() { return 
std::make_shared(); }
+using NullMapType = PaddedPODArray;
+
+/// Get function name.
+String get_name() const override { return name; }
+
+bool is_variadic() const override { return true; }
+
+size_t get_number_of_arguments() const override { return 3; }
+
+DataTypePtr get_return_type_impl(const DataTypes& arguments) const 
override {
+DCHECK(is_array(arguments[0]))
+<< "first argument for function: " << name << " should be 
DataTypeArray"
+<< " and arguments[0] is " << arguments[0]->get_name();
+DCHECK(is_string_or_fixed_string(arguments[1]))
+<< "second argument for function: " << name << " should be 
DataTypeString"
+<< ", and arguments[1] is " << arguments[1]->get_name();
+if (arguments.size() > 2) {
+DCHECK(is_string_or_fixed_string(arguments[2]))
+<< "third argument for function: " << name << " should be 
DataTypeString"
+<< ", and arguments[2] is " << arguments[2]->get_name();
+}
+
+return std::make_shared();
+}
+
+Status execute_impl(FunctionContext* context, Block& block, const 
ColumnNumbers& arguments,
+size_t result, size_t input_rows_count) override {
+ColumnPtr src_column =
+
block.get_by_position(arguments[0]).column->convert_to_full_column_if_const();
+const auto& src_column_array = 
check_and_get_column(*src_column);
+DCHECK(src_column_array != nullptr);
+const auto& src_offsets = src_column_array->get_offsets();
+const auto* src_nested_column = &src_column_array->get_data();
+DCHECK(src_nested_column != nullptr);
+
+const NullMapType* src_null_map = nullptr;
+if (src_nested_column->is_nullable()) {
+const ColumnNullable* src_nested_nullable_col =
+check_and_get_column(*src_nested_column);
+src_nested_column = 
src_nested_nullable_col->get_nested_column_ptr();
+src_null_map = 
&src_nested_nullable_col->get_null_map_column().get_data();
+}
+
+ColumnPtr sep_column =
+
block.get_by_position(arguments[1]).column->convert_to_full_column_if_const();
+ColumnPtr null_replace_column =
+(arguments.size() > 2 ? block.get_by_position(arguments[2])
+
.column->convert_to_full_column_if_const()
+  : nullptr);
+
+std::string sep_str = _get_string_from_column(sep_column);
+std::string null_replace_str = 
_get_string_from_column(null_replace_column);
+
+DataTypePtr src_column_type = block.get_by_position(arguments[0]).type;
+auto nested_type = assert_cast(*src_column_type).get_nested_type();
+
+auto dest_column_ptr = ColumnString::create();
+DCHECK(dest_column_ptr != nullptr);
+dest_column_ptr->reserve(input_rows_count);
+
+auto res_val = _execute_by_type(*src_nested_column, src_offsets, 
src_null_map, sep_str,
+null_replace_str, nested_type, 
dest_column_ptr);
+if (!res_val) {
+return Status::RuntimeError(
+fmt::format("execute failed or unsupported types for 
function {}({},{},{})",
+get_name(), 
block.get_by_position(arguments[0]).type->get_name(),
+