[GitHub] [doris] xinyiZzz commented on a diff in pull request #13585: [enhancement](memtracker) Refactor mem tracker limiter hierarchy and optimize the usage of bthread mem tracker
xinyiZzz commented on code in PR #13585:
URL: https://github.com/apache/doris/pull/13585#discussion_r1014789102
##
be/src/service/doris_main.cpp:
##
@@ -376,15 +375,15 @@ int main(int argc, char** argv) {
apache::thrift::GlobalOutput.setOutputFunction(doris::thrift_output);
Status status = Status::OK();
-#ifdef LIBJVM
-// Init jni
-status = doris::JniUtil::Init();
-if (!status.ok()) {
-LOG(WARNING) << "Failed to initialize JNI: " << status.get_error_msg();
-doris::shutdown_logging();
-exit(1);
-}
-#endif
+// #ifdef LIBJVM
+// // Init jni
Review Comment:
for local test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13991: [fix](memtracker) Fix DCHECK `!std::count(_consumer_tracker_stack.begin(), _consumer_tracker_stack.end(), tracker) ThreadMemTrack
github-actions[bot] commented on PR #13991: URL: https://github.com/apache/doris/pull/13991#issuecomment-1304748027 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13991: [fix](memtracker) Fix DCHECK `!std::count(_consumer_tracker_stack.begin(), _consumer_tracker_stack.end(), tracker) ThreadMemTrack
github-actions[bot] commented on PR #13991: URL: https://github.com/apache/doris/pull/13991#issuecomment-1304748038 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #13991: [fix](memtracker) Fix DCHECK `!std::count(_consumer_tracker_stack.begin(), _consumer_tracker_stack.end(), tracker) ThreadMemTrackerMgr`
dataroaring merged PR #13991: URL: https://github.com/apache/doris/pull/13991 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](memtracker) Fix DCHECK !std::count(_consumer_tracker_stack.begin(), _consumer_tracker_stack.end(), tracker)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new c7b2b90504 [fix](memtracker) Fix DCHECK
!std::count(_consumer_tracker_stack.begin(), _consumer_tracker_stack.end(),
tracker)
c7b2b90504 is described below
commit c7b2b9050463020a3c44e579a79abebbbc20dd89
Author: Xinyi Zou
AuthorDate: Sun Nov 6 16:41:03 2022 +0800
[fix](memtracker) Fix DCHECK !std::count(_consumer_tracker_stack.begin(),
_consumer_tracker_stack.end(), tracker)
---
be/src/runtime/memory/thread_mem_tracker_mgr.h | 10 ++
be/src/runtime/thread_context.cpp | 7 ---
be/src/runtime/thread_context.h| 1 +
3 files changed, 11 insertions(+), 7 deletions(-)
diff --git a/be/src/runtime/memory/thread_mem_tracker_mgr.h
b/be/src/runtime/memory/thread_mem_tracker_mgr.h
index 4661fbea5d..03ccacdeba 100644
--- a/be/src/runtime/memory/thread_mem_tracker_mgr.h
+++ b/be/src/runtime/memory/thread_mem_tracker_mgr.h
@@ -78,7 +78,7 @@ public:
// Must be fast enough! Thread update_tracker may be called very
frequently.
// So for performance, add tracker as early as possible, and then call
update_tracker.
-void push_consumer_tracker(MemTracker* mem_tracker);
+bool push_consumer_tracker(MemTracker* mem_tracker);
void pop_consumer_tracker();
std::string last_consumer_tracker() {
return _consumer_tracker_stack.empty() ? "" :
_consumer_tracker_stack.back()->label();
@@ -191,12 +191,14 @@ inline void ThreadMemTrackerMgr::clear() {
init_impl();
}
-inline void ThreadMemTrackerMgr::push_consumer_tracker(MemTracker* tracker) {
+inline bool ThreadMemTrackerMgr::push_consumer_tracker(MemTracker* tracker) {
DCHECK(tracker) << print_debug_string();
-DCHECK(!std::count(_consumer_tracker_stack.begin(),
_consumer_tracker_stack.end(), tracker))
-<< print_debug_string();
+if (std::count(_consumer_tracker_stack.begin(),
_consumer_tracker_stack.end(), tracker)) {
+return false;
+}
_consumer_tracker_stack.push_back(tracker);
tracker->release(_untracked_mem);
+return true;
}
inline void ThreadMemTrackerMgr::pop_consumer_tracker() {
diff --git a/be/src/runtime/thread_context.cpp
b/be/src/runtime/thread_context.cpp
index babee85c3e..442fe755a6 100644
--- a/be/src/runtime/thread_context.cpp
+++ b/be/src/runtime/thread_context.cpp
@@ -76,20 +76,21 @@
SwitchThreadMemTrackerLimiter::~SwitchThreadMemTrackerLimiter() {
}
AddThreadMemTrackerConsumer::AddThreadMemTrackerConsumer(MemTracker*
mem_tracker) {
-
thread_context()->_thread_mem_tracker_mgr->push_consumer_tracker(mem_tracker);
+_need_pop =
thread_context()->_thread_mem_tracker_mgr->push_consumer_tracker(mem_tracker);
}
AddThreadMemTrackerConsumer::AddThreadMemTrackerConsumer(
const std::shared_ptr& mem_tracker)
: _mem_tracker(mem_tracker) {
-
thread_context()->_thread_mem_tracker_mgr->push_consumer_tracker(_mem_tracker.get());
+_need_pop =
+
thread_context()->_thread_mem_tracker_mgr->push_consumer_tracker(_mem_tracker.get());
}
AddThreadMemTrackerConsumer::~AddThreadMemTrackerConsumer() {
#ifndef NDEBUG
DorisMetrics::instance()->add_thread_mem_tracker_consumer_count->increment(1);
#endif // NDEBUG
-thread_context()->_thread_mem_tracker_mgr->pop_consumer_tracker();
+if (_need_pop)
thread_context()->_thread_mem_tracker_mgr->pop_consumer_tracker();
}
} // namespace doris
diff --git a/be/src/runtime/thread_context.h b/be/src/runtime/thread_context.h
index 7d213bfa4d..48ff9cc389 100644
--- a/be/src/runtime/thread_context.h
+++ b/be/src/runtime/thread_context.h
@@ -311,6 +311,7 @@ public:
private:
std::shared_ptr _mem_tracker = nullptr; // Avoid mem_tracker
being released midway.
+bool _need_pop = false;
};
class StopCheckThreadMemTrackerLimit {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on a diff in pull request #13585: [enhancement](memtracker) Refactor mem tracker limiter hierarchy and optimize the usage of bthread mem tracker
xinyiZzz commented on code in PR #13585:
URL: https://github.com/apache/doris/pull/13585#discussion_r1014792426
##
be/src/service/internal_service.cpp:
##
@@ -64,12 +63,14 @@ static void thread_context_deleter(void* d) {
}
template
-class NewHttpClosure : public ::google::protobuf::Closure {
+class NewClosure : public ::google::protobuf::Closure {
Review Comment:
NewClosure is not used for mem tracker, it is used to delete the recreated
request when brpc attachment transmits data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on a diff in pull request #13585: [enhancement](memtracker) Refactor mem tracker limiter hierarchy and optimize the usage of bthread mem tracker
xinyiZzz commented on code in PR #13585:
URL: https://github.com/apache/doris/pull/13585#discussion_r1014792426
##
be/src/service/internal_service.cpp:
##
@@ -64,12 +63,14 @@ static void thread_context_deleter(void* d) {
}
template
-class NewHttpClosure : public ::google::protobuf::Closure {
+class NewClosure : public ::google::protobuf::Closure {
Review Comment:
NewClosure is not used for mem tracker, it is used to delete the recreated
request when brpc attachment transmits data
see: https://github.com/apache/doris/pull/7164
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on a diff in pull request #13585: [enhancement](memtracker) Refactor mem tracker limiter hierarchy and optimize the usage of bthread mem tracker
xinyiZzz commented on code in PR #13585:
URL: https://github.com/apache/doris/pull/13585#discussion_r1014792845
##
be/src/service/internal_service.cpp:
##
@@ -64,12 +63,14 @@ static void thread_context_deleter(void* d) {
}
template
-class NewHttpClosure : public ::google::protobuf::Closure {
+class NewClosure : public ::google::protobuf::Closure {
Review Comment:
rename to NewHttpClosure, it seems better to understand
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13976: [feature](Nereids) support statement having aggregate function in order by list
github-actions[bot] commented on PR #13976: URL: https://github.com/apache/doris/pull/13976#issuecomment-1304781929 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13976: [feature](Nereids) support statement having aggregate function in order by list
github-actions[bot] commented on PR #13976: URL: https://github.com/apache/doris/pull/13976#issuecomment-1304781935 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] SaintBacchus commented on issue #13146: [Enhancement][MultiTableMaterializedView] The schedule framework for the MTMV
SaintBacchus commented on issue #13146: URL: https://github.com/apache/doris/issues/13146#issuecomment-1304798851 Second impl https://github.com/apache/doris/issues/13751 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #13921: [improvement](join) Share hash table in fragments for broadcast join
HappenLee commented on code in PR #13921:
URL: https://github.com/apache/doris/pull/13921#discussion_r1014839824
##
be/src/runtime/fragment_mgr.cpp:
##
@@ -738,6 +745,34 @@ void FragmentMgr::_set_scan_concurrency(const
TExecPlanFragmentParams& params,
#endif
}
+void FragmentMgr::_setup_shared_hashtable_for_broadcast_join(const
TExecPlanFragmentParams& params,
+ RuntimeState*
state,
+
QueryFragmentsCtx* fragments_ctx) {
+if (!params.__isset.fragment || !params.fragment.__isset.plan ||
+params.fragment.plan.nodes.empty()) {
+return;
+}
+
+for (auto& node : params.fragment.plan.nodes) {
+if (node.node_type != TPlanNodeType::HASH_JOIN_NODE ||
!node.__isset.hash_join_node) {
+continue;
+}
+if (node.runtime_filters.empty()) {
Review Comment:
why the broadcast join node must have rf ？ rethink the logic？
##
be/src/exprs/runtime_filter.cpp:
##
@@ -1108,7 +1141,8 @@ Status IRuntimeFilter::publish() {
RETURN_IF_ERROR(
_state->runtime_filter_mgr()->get_consume_filter(_filter_id,
&consumer_filter));
// push down
-std::swap(this->_wrapper, consumer_filter->_wrapper);
+//std::swap(this->_wrapper, consumer_filter->_wrapper);
Review Comment:
delete the useless code
##
be/src/vec/runtime/shared_hash_table_controller.cpp:
##
@@ -0,0 +1,95 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "shared_hash_table_controller.h"
+
+#include
+
+namespace doris {
+namespace vectorized {
+
+bool SharedHashTableController::should_build_hash_table(RuntimeState* state,
int my_node_id) {
+std::lock_guard lock(_mutex);
+auto it = _builder_fragment_ids.find(my_node_id);
+if (it == _builder_fragment_ids.cend()) {
+_builder_fragment_ids[my_node_id] = state->fragment_instance_id();
+return true;
+}
+return false;
+}
+
+void SharedHashTableController::put_hash_table(SharedHashTableEntry&& entry,
int my_node_id) {
+std::lock_guard lock(_mutex);
+DCHECK(_hash_table_entries.find(my_node_id) == _hash_table_entries.cend());
+_hash_table_entries.insert({my_node_id, std::move(entry)});
+_cv.notify_all();
+}
+
+SharedHashTableEntry& SharedHashTableController::wait_for_hash_table(int
my_node_id) {
+std::unique_lock lock(_mutex);
+auto it = _hash_table_entries.find(my_node_id);
+if (it == _hash_table_entries.cend()) {
+_cv.wait(lock, [this, &it, my_node_id]() {
Review Comment:
maybe `dead lock` when the node should be build hash table is canceled. it's
dangerous
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] platoneko opened a new pull request, #13993: [fix](schema) Release memory of TabletSchemaPB in RowsetMetaPB
platoneko opened a new pull request, #13993: URL: https://github.com/apache/doris/pull/13993 # Proposed changes Issue Number: close #xxx ## Problem summary clear_tablet_schema won't release memory of TabletSchemaPB ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13993: [fix](schema) Release memory of TabletSchemaPB in RowsetMetaPB
hello-stephen commented on PR #13993: URL: https://github.com/apache/doris/pull/13993#issuecomment-1304825794 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.45 seconds load time: 461 seconds storage size: 17181536150 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221106152448_clickbench_pr_40663.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #13994: [feature](multi-catalog) Support data on s3-compatible oss and support aliyun DLF
morningman opened a new pull request, #13994: URL: https://github.com/apache/doris/pull/13994 # Proposed changes Issue Number: close #xxx ## Problem summary ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] carlvinhust2012 opened a new pull request, #13995: [fix](array-type) check whether the param of array function is correct
carlvinhust2012 opened a new pull request, #13995:
URL: https://github.com/apache/doris/pull/13995
# Proposed changes
1. this pr is used to check whether the param of array function is correct.
2. before the change, you will get the common error when you load data and
use array function.
MySQL [example_db]> show load;
++-+---+---++-+-+---+-+--+---+---+-+--+--+---+--+
| JobId | Label | State | Progress
| Type | EtlInfo | TaskInfo|
ErrorMsg
| CreateTime |
EtlStartTime | EtlFinishTime | LoadStartTime | LoadFinishTime | URL |
JobDetails
| TransactionId | ErrorTablets |
++-+---+---++-+-+---+-+--+---+---+-+--+--+---+--+
| 383159 | label_03_14_49_34_898986_1909045295 | CANCELLED | ETL:N/A;
LOAD:N/A | BROKER | NULL| cluster:N/A; timeout(s):14400;
max_filter_ratio:0.8 | type:ETL_RUN_FAIL; msg:errCode = 2, detailMessage = No
matching function with signature: array_union(varchar(-1), varchar(-1)). |
2022-11-04 14:43:28 | NULL | NULL | NULL | 2022-11-04
14:43:31 | NULL | {"Unfinished
backends":{},"ScannedRows":0,"TaskNumber":0,"LoadBytes":0,"All
backends":{},"FileNumber":0,"FileSize":0}
3. after the change, you will get the detail error when you load data and
use array function.
MySQL [example_db]> show load;
++-+---+---++-+-+---+-+--+---+---+-+--+--+---+--+
| JobId | Label | State | Progress
| Type | EtlInfo | TaskInfo|
ErrorMsg
| CreateTime |
EtlStartTime | EtlFinishTime | LoadStartTime | LoadFinishTime | URL |
JobDetails
| TransactionId | ErrorTablets |
++-+---+---++-+-+---+-+--+---+---+-+--+--+---+--+
| 405098 | label_03_14_49_34_898986_1909045295 | CANCELLED | ETL:N/A;
LOAD:N/A | BROKER | NULL| cluster:N/A; timeout(s):14400;
max_filter_ratio:0.8 | type:ETL_RUN_FAIL; msg:errCode = 2, detailMessage = The
first param of function array_union is VARCHAR(*), but it should be array |
2022-11-07 00:04:21 | NULL | NULL | NULL | 2022-11-07
00:04:22 | NULL | {"Unfinished
backends":{},"ScannedRows":0,"TaskNumber":0,"LoadBytes":0,"All
backends":{},"FileNumber":1,"FileSize":11742644} | 48045 | {}
|
Issue Number: #7570
## Problem summary
Describe your changes.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ]
[GitHub] [doris] hello-stephen commented on pull request #13995: [fix](array-type) check whether the param of array function is correct
hello-stephen commented on PR #13995: URL: https://github.com/apache/doris/pull/13995#issuecomment-1304841586 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.3 seconds load time: 450 seconds storage size: 17179601668 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221106164334_clickbench_pr_40688.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] vkingnew opened a new issue, #13996: [Feature] NTH_VALUE windows function support
vkingnew opened a new issue, #13996: URL: https://github.com/apache/doris/issues/13996 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description NTH_VALUE function not support https://dev.mysql.com/doc/refman/8.0/en/window-function-descriptions.html#function_nth-value ### Use case _No response_ ### Related issues _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13994: [feature](multi-catalog) Support data on s3-compatible oss and support aliyun DLF
hello-stephen commented on PR #13994: URL: https://github.com/apache/doris/pull/13994#issuecomment-1304846790 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.01 seconds load time: 449 seconds storage size: 17180294699 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221106170421_clickbench_pr_40682.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] closed pull request #9432: [Feature]Use multi hash tables to aggregate
github-actions[bot] closed pull request #9432: [Feature]Use multi hash tables to aggregate URL: https://github.com/apache/doris/pull/9432 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch asf-site updated: Automated deployment with doris branch @ 3ff8eaabcaa837bf305175c1025408e090bcfe45
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new ec37ca2293c Automated deployment with doris branch @
3ff8eaabcaa837bf305175c1025408e090bcfe45
ec37ca2293c is described below
commit ec37ca2293c06d395f6a3777a5bd612d258826f7
Author: github-actions[bot]
AuthorDate: Mon Nov 7 01:47:11 2022 +
Automated deployment with doris branch @
3ff8eaabcaa837bf305175c1025408e090bcfe45
---
docs/0.15/search-index.json | 2 +-
docs/dev/search-index.json| 2 +-
search-index.json | 2 +-
zh-CN/docs/0.15/search-index.json | 2 +-
zh-CN/docs/dev/search-index.json | 2 +-
zh-CN/search-index.json | 2 +-
6 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/docs/0.15/search-index.json b/docs/0.15/search-index.json
index baeeb4349ad..324064dfb09 100644
--- a/docs/0.15/search-index.json
+++ b/docs/0.15/search-index.json
@@ -1 +1 @@
-[{"documents":[{"i":7143,"t":"Bitmap
Index","u":"/docs/0.15/administrator-guide/alter-table/alter-table-bitmap-index","b":["Docs","Administrator
Guide","Schema Change"]},{"i":7159,"t":"Replace
Table","u":"/docs/0.15/administrator-guide/alter-table/alter-table-replace-table","b":["Docs","Administrator
Guide","Schema
Change"]},{"i":7169,"t":"Rollup","u":"/docs/0.15/administrator-guide/alter-table/alter-table-rollup","b":["Docs","Administrator
Guide","Schema Change"]},{"i":7190,"t":"Schema [...]
\ No newline at end of file
+[{"documents":[{"i":7143,"t":"Bitmap
Index","u":"/docs/0.15/administrator-guide/alter-table/alter-table-bitmap-index","b":["Docs","Administrator
Guide","Schema Change"]},{"i":7159,"t":"Replace
Table","u":"/docs/0.15/administrator-guide/alter-table/alter-table-replace-table","b":["Docs","Administrator
Guide","Schema Change"]},{"i":7169,"t":"Schema
Change","u":"/docs/0.15/administrator-guide/alter-table/alter-table-schema-change","b":["Docs","Administrator
Guide","Schema Change"]},{"i":719 [...]
\ No newline at end of file
diff --git a/docs/dev/search-index.json b/docs/dev/search-index.json
index 9f7fe042655..178efcdb083 100644
--- a/docs/dev/search-index.json
+++ b/docs/dev/search-index.json
@@ -1 +1 @@
-[{"documents":[{"i":12943,"t":"Elastic
scaling","u":"/docs/dev/admin-manual/cluster-management/elastic-expansion","b":["Docs","Admin
Manual","cluster management"]},{"i":12959,"t":"load
balancing","u":"/docs/dev/admin-manual/cluster-management/load-balancing","b":["Docs","Admin
Manual","cluster management"]},{"i":12986,"t":"Cluster
upgrade","u":"/docs/dev/admin-manual/cluster-management/upgrade","b":["Docs","Admin
Manual","cluster management"]},{"i":13000,"t":"BE Configuration","u":"/docs
[...]
\ No newline at end of file
+[{"documents":[{"i":12943,"t":"Elastic
scaling","u":"/docs/dev/admin-manual/cluster-management/elastic-expansion","b":["Docs","Admin
Manual","cluster management"]},{"i":12959,"t":"load
balancing","u":"/docs/dev/admin-manual/cluster-management/load-balancing","b":["Docs","Admin
Manual","cluster management"]},{"i":12986,"t":"Cluster
upgrade","u":"/docs/dev/admin-manual/cluster-management/upgrade","b":["Docs","Admin
Manual","cluster management"]},{"i":13000,"t":"BE Configuration","u":"/docs
[...]
\ No newline at end of file
diff --git a/search-index.json b/search-index.json
index 116780f7938..625f63fe3cf 100644
--- a/search-index.json
+++ b/search-index.json
@@ -1 +1 @@
-[{"documents":[{"i":1,"t":"Elastic
scaling","u":"/docs/admin-manual/cluster-management/elastic-expansion","b":["Docs","Admin
Manual","cluster management"]},{"i":17,"t":"load
balancing","u":"/docs/admin-manual/cluster-management/load-balancing","b":["Docs","Admin
Manual","cluster management"]},{"i":44,"t":"Cluster
upgrade","u":"/docs/admin-manual/cluster-management/upgrade","b":["Docs","Admin
Manual","cluster management"]},{"i":60,"t":"BE
Configuration","u":"/docs/admin-manual/config/be-c [...]
\ No newline at end of file
+[{"documents":[{"i":1,"t":"Elastic
scaling","u":"/docs/admin-manual/cluster-management/elastic-expansion","b":["Docs","Admin
Manual","cluster management"]},{"i":17,"t":"load
balancing","u":"/docs/admin-manual/cluster-management/load-balancing","b":["Docs","Admin
Manual","cluster management"]},{"i":44,"t":"Cluster
upgrade","u":"/docs/admin-manual/cluster-management/upgrade","b":["Docs","Admin
Manual","cluster management"]},{"i":60,"t":"BE
Configuration","u":"/docs/admin-manual/config/be-c [...]
\ No newline at end of file
diff --git a/zh-CN/docs/0.15/search-index.json
b/zh-CN/docs/0.15/search-index.json
index 36aca99498a..9b3e601a14d 100644
--- a/zh-CN/docs/0.15/search-index.json
+++ b/zh-CN/docs/0.15/search-index.json
@@ -1 +1 @@
-[{"documents":[{"i":27940,"t":"Bitmap
索引","u":"/zh-CN/docs/0.15/administrator-guide/alter-table/alter-table-bitmap-index","b":["文档","操作手册","表结构变更
[GitHub] [doris] adonis0147 opened a new pull request, #13997: Test
adonis0147 opened a new pull request, #13997: URL: https://github.com/apache/doris/pull/13997 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #13980: [feature](compaction) support vertical compaction
dataroaring merged PR #13980: URL: https://github.com/apache/doris/pull/13980 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhanghengdashuaibi opened a new issue, #13998: [Feature] doris1.1.3版本的日志审计表没有数据
zhanghengdashuaibi opened a new issue, #13998: URL: https://github.com/apache/doris/issues/13998 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description doris1.1.3版本按照官网的日志审计插件安装后，日志审计表没有数据，也没有报错，请问还有什么漏掉的步骤没做导致的日志审计表没有数据呢 ### Use case _No response_ ### Related issues _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13997: Test
hello-stephen commented on PR #13997: URL: https://github.com/apache/doris/pull/13997#issuecomment-1305004490 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 33.65 seconds load time: 452 seconds storage size: 17179536847 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221107023246_clickbench_pr_40717.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 commented on a diff in pull request #13953: [fix](repeat)remove unmaterialized expr from repeat node
starocean999 commented on code in PR #13953:
URL: https://github.com/apache/doris/pull/13953#discussion_r1014957872
##
fe/fe-core/src/main/java/org/apache/doris/analysis/GroupingInfo.java:
##
@@ -89,7 +89,30 @@ public List getPreRepeatExprs() {
}
public void substitutePreRepeatExprs(ExprSubstitutionMap smap, Analyzer
analyzer) {
+ArrayList originalPreRepeatExprs = new
ArrayList<>(preRepeatExprs);
preRepeatExprs = Expr.substituteList(preRepeatExprs, smap, analyzer,
true);
+
+// remove unmaterialized slotRef from preRepeatExprs
+ArrayList materializedPreRepeatExprs = new ArrayList<>();
+ArrayList unMaterializedSlotRefs = new ArrayList<>();
+for (int i = 0; i < preRepeatExprs.size(); ++i) {
+Expr expr = preRepeatExprs.get(i);
+if (expr instanceof SlotRef && !((SlotRef)
expr).getDesc().isMaterialized()) {
+unMaterializedSlotRefs.add(originalPreRepeatExprs.get(i));
+} else {
+materializedPreRepeatExprs.add(expr);
+}
+}
+preRepeatExprs = materializedPreRepeatExprs;
+
+// remove unmaterialized slotRef from outputTupleSmap and
outputTupleDesc if there is any
+for (Expr expr : unMaterializedSlotRefs) {
+Expr rExpr = outputTupleSmap.get(expr);
+outputTupleSmap.removeByRhsExpr(rExpr);
+if (rExpr instanceof SlotRef) {
+outputTupleDesc.getSlots().remove(((SlotRef) rExpr).getDesc());
Review Comment:
set unmaterialized is better, code is updated, thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Toms1999 commented on pull request #13771: [new] To complete the mysql、hive、pgsql external table to doris by shell
Toms1999 commented on PR #13771: URL: https://github.com/apache/doris/pull/13771#issuecomment-1305007756 up -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13641: [feature-array](array-type) Add array function array_popback
github-actions[bot] commented on PR #13641: URL: https://github.com/apache/doris/pull/13641#issuecomment-1305010650 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xy720 merged pull request #13641: [feature-array](array-type) Add array function array_popback
xy720 merged PR #13641: URL: https://github.com/apache/doris/pull/13641 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13641: [feature-array](array-type) Add array function array_popback
github-actions[bot] commented on PR #13641: URL: https://github.com/apache/doris/pull/13641#issuecomment-1305010667 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (c7b2b90504 -> 7ffe88b579)
This is an automated email from the ASF dual-hosted git repository.
xuyang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from c7b2b90504 [fix](memtracker) Fix DCHECK
!std::count(_consumer_tracker_stack.begin(), _consumer_tracker_stack.end(),
tracker)
add 7ffe88b579 [feature-array](array-type) Add array function
array_popback (#13641)
No new revisions were added by this update.
Summary of changes:
be/src/vec/CMakeLists.txt | 1 +
.../vec/functions/array/function_array_popback.cpp | 84 ++
.../functions/array/function_array_register.cpp| 2 +
be/src/vec/functions/array/function_array_slice.h | 47 +---
.../vec/functions/array/function_array_utils.cpp | 44
be/src/vec/functions/array/function_array_utils.h | 4 ++
.../nullif.md => array-functions/array_popback.md} | 48 ++---
.../array_popback.md} | 36 +-
gensrc/script/doris_builtins_functions.py | 14
.../array_functions/test_array_functions.out | 36 ++
.../array_functions/test_array_functions.groovy| 4 ++
11 files changed, 232 insertions(+), 88 deletions(-)
create mode 100644 be/src/vec/functions/array/function_array_popback.cpp
copy docs/en/docs/sql-manual/sql-functions/{conditional-functions/nullif.md =>
array-functions/array_popback.md} (60%)
copy
docs/zh-CN/docs/sql-manual/sql-functions/{bitmap-functions/bitmap_empty.md =>
array-functions/array_popback.md} (58%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] sky1229329146 opened a new issue, #13999: 以小时自动创建分区，分区创建的时间越来越迟，最后可能第一个按小时的分区不会创建导致数据丢失
sky1229329146 opened a new issue, #13999: URL: https://github.com/apache/doris/issues/13999 1.以小时自动创建分区，第一个分区和第二个分区的创建时间并不是严格的相差1小时，而是比1小时多几秒 2.一天有24小时，则最后一个分区的创建时间就会延迟数十秒才创建 3.到第二天系统再自动创建分区时，第一个分区就明显比第一次创建延迟了很久才创建 放图报错，暂时无法放图 6号创建的分区比3号创建的分区完了30多秒 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] LemonLiTree opened a new pull request, #14000: [typo](docs)add udf doc and optimize udf regression test
LemonLiTree opened a new pull request, #14000: URL: https://github.com/apache/doris/pull/14000 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #13993: [fix](schema) Release memory of TabletSchemaPB in RowsetMetaPB
morningman commented on code in PR #13993:
URL: https://github.com/apache/doris/pull/13993#discussion_r1014967466
##
be/src/olap/rowset/rowset_meta.h:
##
@@ -53,12 +53,14 @@ class RowsetMeta {
}
virtual bool init_from_pb(const RowsetMetaPB& rowset_meta_pb) {
-_rowset_meta_pb = rowset_meta_pb;
-if (_rowset_meta_pb.has_tablet_schema()) {
+if (rowset_meta_pb.has_tablet_schema()) {
_schema = TabletSchemaCache::instance()->insert(
-_rowset_meta_pb.tablet_schema().SerializeAsString());
-_rowset_meta_pb.clear_tablet_schema();
+rowset_meta_pb.tablet_schema().SerializeAsString());
}
+auto& mut_rowset_meta_pb = const_cast(rowset_meta_pb);
Review Comment:
Could you add some comment to explain this?
It is unusual to use `const_cast` to a const parameter, which breaks the
semantics of the method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TaoZex opened a new pull request, #14001: [typo](doc) fix get-starting doc
TaoZex opened a new pull request, #14001: URL: https://github.com/apache/doris/pull/14001 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhanghengdashuaibi commented on issue #13998: [Feature] doris1.1.3版本的日志审计表没有数据
zhanghengdashuaibi commented on issue #13998: URL: https://github.com/apache/doris/issues/13998#issuecomment-1305024412 已解决 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhanghengdashuaibi closed issue #13998: [Feature] doris1.1.3版本的日志审计表没有数据
zhanghengdashuaibi closed issue #13998: [Feature] doris1.1.3版本的日志审计表没有数据 URL: https://github.com/apache/doris/issues/13998 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] TaoZex opened a new pull request, #152: [typo](doc) fix get-starting doc
TaoZex opened a new pull request, #152: URL: https://github.com/apache/doris-website/pull/152 pr in doris repo：https://github.com/apache/doris/pull/14001 fix get-starting doc -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] catpineapple commented on a diff in pull request #13772: [feature](planner) add multi partition
catpineapple commented on code in PR #13772:
URL: https://github.com/apache/doris/pull/13772#discussion_r1014970740
##
fe/fe-core/src/main/java/org/apache/doris/analysis/MultiPartitionDesc.java:
##
@@ -0,0 +1,316 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.analysis.TimestampArithmeticExpr.TimeUnit;
+import org.apache.doris.catalog.DynamicPartitionProperty;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.util.DynamicPartitionUtil;
+import org.apache.doris.planner.DateTools;
+
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Lists;
+
+import java.time.DayOfWeek;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.format.DateTimeFormatter;
+import java.time.temporal.ChronoField;
+import java.time.temporal.WeekFields;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+// to describe the key list partition's information in create table stmt
+public class MultiPartitionDesc implements AllPartitionDesc {
+public static final String HOURS_FORMAT = "MMddHH";
+public static final String HOUR_FORMAT = "-MM-dd HH";
+public static final String DATES_FORMAT = "MMdd";
+public static final String DATE_FORMAT = "-MM-dd";
+public static final String MONTHS_FORMAT = "MM";
+public static final String MONTH_FORMAT = "-MM";
+public static final String YEAR_FORMAT = "";
+public static final String DATETIME_FORMAT = "-MM-dd HH:mm:ss";
+
+
+
+private final String partitionPrefix = "p_";
+private LocalDateTime startTime;
+private LocalDateTime endTime;
+
+private DateTimeFormatter startDateTimeFormat;
+private DateTimeFormatter endDateTimeFormat;
+
+
+private Long timeInterval;
+private final PartitionKeyDesc partitionKeyDesc;
+private TimestampArithmeticExpr.TimeUnit timeUnitType;
+private final Map properties;
+private final List singlePartitionDescList =
Lists.newArrayList();
+
+private final ImmutableSet
timeUnitTypeMultiPartition = ImmutableSet.of(
+TimestampArithmeticExpr.TimeUnit.HOUR,
+TimestampArithmeticExpr.TimeUnit.DAY,
+TimestampArithmeticExpr.TimeUnit.WEEK,
+TimestampArithmeticExpr.TimeUnit.MONTH,
+TimestampArithmeticExpr.TimeUnit.YEAR
+);
+
+private final Integer maxAllowedLimit = Config.max_multi_partition_num;
+
+public MultiPartitionDesc(PartitionKeyDesc partitionKeyDesc,
+ Map properties) throws
AnalysisException {
+this.partitionKeyDesc = partitionKeyDesc;
+this.properties = properties;
+this.timeIntervalTrans();
+this.timeTrans();
+}
+
+public List getSinglePartitionDescList() throws
AnalysisException {
+if (singlePartitionDescList.size() == 0) {
+buildMultiPartitionToSinglePartitionDescs();
+}
+return singlePartitionDescList;
+}
+
+private List
buildMultiPartitionToSinglePartitionDescs() throws AnalysisException {
+String partitionName;
+long countNum = 0;
+int dayOfWeek = 1;
+int dayOfMonth = 1;
+String partitionPrefix = this.partitionPrefix;
+LocalDateTime startTime = this.startTime;
+if (properties != null) {
+if
(properties.containsKey(DynamicPartitionProperty.START_DAY_OF_WEEK)) {
+String dayOfWeekStr =
properties.get(DynamicPartitionProperty.START_DAY_OF_WEEK);
+try {
+DynamicPartitionUtil.checkStartDayOfWeek(dayOfWeekStr);
+} catch (DdlException e) {
+throw new AnalysisException(e.getMessage());
+}
+dayOfWeek = Integer.parseInt(dayOfWeekStr);
+}
+if
(properties.containsKey(DynamicPartitionProperty.START_DAY_OF_MONTH)) {
+String dayOfMonthStr =
properties.get(DynamicPartitionProperty.START_DAY_OF_MONTH);
+try {
+
[GitHub] [doris] catpineapple commented on a diff in pull request #13772: [feature](planner) add multi partition
catpineapple commented on code in PR #13772:
URL: https://github.com/apache/doris/pull/13772#discussion_r1014976484
##
fe/fe-core/src/main/java/org/apache/doris/analysis/MultiPartitionDesc.java:
##
@@ -0,0 +1,316 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.analysis.TimestampArithmeticExpr.TimeUnit;
+import org.apache.doris.catalog.DynamicPartitionProperty;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.util.DynamicPartitionUtil;
+import org.apache.doris.planner.DateTools;
+
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Lists;
+
+import java.time.DayOfWeek;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.format.DateTimeFormatter;
+import java.time.temporal.ChronoField;
+import java.time.temporal.WeekFields;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+// to describe the key list partition's information in create table stmt
+public class MultiPartitionDesc implements AllPartitionDesc {
+public static final String HOURS_FORMAT = "MMddHH";
+public static final String HOUR_FORMAT = "-MM-dd HH";
+public static final String DATES_FORMAT = "MMdd";
+public static final String DATE_FORMAT = "-MM-dd";
+public static final String MONTHS_FORMAT = "MM";
+public static final String MONTH_FORMAT = "-MM";
+public static final String YEAR_FORMAT = "";
+public static final String DATETIME_FORMAT = "-MM-dd HH:mm:ss";
+
+
+
+private final String partitionPrefix = "p_";
+private LocalDateTime startTime;
+private LocalDateTime endTime;
+
+private DateTimeFormatter startDateTimeFormat;
+private DateTimeFormatter endDateTimeFormat;
+
+
+private Long timeInterval;
+private final PartitionKeyDesc partitionKeyDesc;
+private TimestampArithmeticExpr.TimeUnit timeUnitType;
+private final Map properties;
+private final List singlePartitionDescList =
Lists.newArrayList();
+
+private final ImmutableSet
timeUnitTypeMultiPartition = ImmutableSet.of(
+TimestampArithmeticExpr.TimeUnit.HOUR,
+TimestampArithmeticExpr.TimeUnit.DAY,
+TimestampArithmeticExpr.TimeUnit.WEEK,
+TimestampArithmeticExpr.TimeUnit.MONTH,
+TimestampArithmeticExpr.TimeUnit.YEAR
+);
+
+private final Integer maxAllowedLimit = Config.max_multi_partition_num;
+
+public MultiPartitionDesc(PartitionKeyDesc partitionKeyDesc,
+ Map properties) throws
AnalysisException {
+this.partitionKeyDesc = partitionKeyDesc;
+this.properties = properties;
+this.timeIntervalTrans();
+this.timeTrans();
+}
+
+public List getSinglePartitionDescList() throws
AnalysisException {
+if (singlePartitionDescList.size() == 0) {
+buildMultiPartitionToSinglePartitionDescs();
+}
+return singlePartitionDescList;
+}
+
+private List
buildMultiPartitionToSinglePartitionDescs() throws AnalysisException {
+String partitionName;
+long countNum = 0;
+int dayOfWeek = 1;
+int dayOfMonth = 1;
+String partitionPrefix = this.partitionPrefix;
+LocalDateTime startTime = this.startTime;
+if (properties != null) {
+if
(properties.containsKey(DynamicPartitionProperty.START_DAY_OF_WEEK)) {
+String dayOfWeekStr =
properties.get(DynamicPartitionProperty.START_DAY_OF_WEEK);
+try {
+DynamicPartitionUtil.checkStartDayOfWeek(dayOfWeekStr);
+} catch (DdlException e) {
+throw new AnalysisException(e.getMessage());
+}
+dayOfWeek = Integer.parseInt(dayOfWeekStr);
+}
+if
(properties.containsKey(DynamicPartitionProperty.START_DAY_OF_MONTH)) {
+String dayOfMonthStr =
properties.get(DynamicPartitionProperty.START_DAY_OF_MONTH);
+try {
+
[GitHub] [doris] catpineapple commented on a diff in pull request #13772: [feature](planner) add multi partition
catpineapple commented on code in PR #13772:
URL: https://github.com/apache/doris/pull/13772#discussion_r1014976532
##
fe/fe-core/src/main/java/org/apache/doris/planner/DateTools.java:
##
@@ -0,0 +1,48 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.planner;
+
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.format.DateTimeFormatter;
+import java.time.temporal.ChronoField;
+import java.time.temporal.TemporalAccessor;
+
+public class DateTools {
+// format string DateTime And Full Zero for hour,minute,second
+public static LocalDateTime formatDateTimeAndFullZero(String datetime,
DateTimeFormatter formatter) {
Review Comment:
I will
##
fe/fe-core/src/main/cup/sql_parser.cup:
##
@@ -2927,6 +2938,22 @@ single_partition_desc ::=
:}
;
+multi_partition_desc ::=
+fixed_multi_partition_key_desc:desc
+opt_key_value_map:properties
+{:
+RESULT = new MultiPartitionDesc(desc, properties);
+:}
+;
+
+fixed_multi_partition_key_desc ::=
+// FROM (lower) TO (upper) INTERVAL time_interval time_type
+KW_FROM LPAREN partition_key_list:lower RPAREN KW_TO LPAREN
partition_key_list:upper RPAREN KW_INTERVAL INTEGER_LITERAL:time_interval
ident:time_type
Review Comment:
I will
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 opened a new pull request, #14002: [Bug](udf) Make UDF's type always nullable
Gabriel39 opened a new pull request, #14002: URL: https://github.com/apache/doris/pull/14002 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] catpineapple commented on a diff in pull request #13772: [feature](planner) add multi partition
catpineapple commented on code in PR #13772:
URL: https://github.com/apache/doris/pull/13772#discussion_r1014976699
##
fe/fe-core/src/main/java/org/apache/doris/analysis/MultiPartitionDesc.java:
##
@@ -0,0 +1,316 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.analysis.TimestampArithmeticExpr.TimeUnit;
+import org.apache.doris.catalog.DynamicPartitionProperty;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.util.DynamicPartitionUtil;
+import org.apache.doris.planner.DateTools;
+
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Lists;
+
+import java.time.DayOfWeek;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.format.DateTimeFormatter;
+import java.time.temporal.ChronoField;
+import java.time.temporal.WeekFields;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+// to describe the key list partition's information in create table stmt
+public class MultiPartitionDesc implements AllPartitionDesc {
+public static final String HOURS_FORMAT = "MMddHH";
+public static final String HOUR_FORMAT = "-MM-dd HH";
+public static final String DATES_FORMAT = "MMdd";
+public static final String DATE_FORMAT = "-MM-dd";
+public static final String MONTHS_FORMAT = "MM";
+public static final String MONTH_FORMAT = "-MM";
+public static final String YEAR_FORMAT = "";
+public static final String DATETIME_FORMAT = "-MM-dd HH:mm:ss";
+
+
+
+private final String partitionPrefix = "p_";
+private LocalDateTime startTime;
+private LocalDateTime endTime;
+
+private DateTimeFormatter startDateTimeFormat;
+private DateTimeFormatter endDateTimeFormat;
+
+
+private Long timeInterval;
+private final PartitionKeyDesc partitionKeyDesc;
+private TimestampArithmeticExpr.TimeUnit timeUnitType;
+private final Map properties;
+private final List singlePartitionDescList =
Lists.newArrayList();
+
+private final ImmutableSet
timeUnitTypeMultiPartition = ImmutableSet.of(
+TimestampArithmeticExpr.TimeUnit.HOUR,
+TimestampArithmeticExpr.TimeUnit.DAY,
+TimestampArithmeticExpr.TimeUnit.WEEK,
+TimestampArithmeticExpr.TimeUnit.MONTH,
+TimestampArithmeticExpr.TimeUnit.YEAR
+);
+
+private final Integer maxAllowedLimit = Config.max_multi_partition_num;
+
+public MultiPartitionDesc(PartitionKeyDesc partitionKeyDesc,
+ Map properties) throws
AnalysisException {
+this.partitionKeyDesc = partitionKeyDesc;
+this.properties = properties;
+this.timeIntervalTrans();
+this.timeTrans();
+}
+
+public List getSinglePartitionDescList() throws
AnalysisException {
+if (singlePartitionDescList.size() == 0) {
+buildMultiPartitionToSinglePartitionDescs();
+}
+return singlePartitionDescList;
+}
+
+private List
buildMultiPartitionToSinglePartitionDescs() throws AnalysisException {
+String partitionName;
+long countNum = 0;
+int dayOfWeek = 1;
+int dayOfMonth = 1;
+String partitionPrefix = this.partitionPrefix;
+LocalDateTime startTime = this.startTime;
+if (properties != null) {
+if
(properties.containsKey(DynamicPartitionProperty.START_DAY_OF_WEEK)) {
+String dayOfWeekStr =
properties.get(DynamicPartitionProperty.START_DAY_OF_WEEK);
+try {
+DynamicPartitionUtil.checkStartDayOfWeek(dayOfWeekStr);
+} catch (DdlException e) {
+throw new AnalysisException(e.getMessage());
+}
+dayOfWeek = Integer.parseInt(dayOfWeekStr);
+}
+if
(properties.containsKey(DynamicPartitionProperty.START_DAY_OF_MONTH)) {
+String dayOfMonthStr =
properties.get(DynamicPartitionProperty.START_DAY_OF_MONTH);
+try {
+
[GitHub] [doris] englefly opened a new pull request, #14003: [enhance](nereids) remove cast from numeric literal to decimal
englefly opened a new pull request, #14003: URL: https://github.com/apache/doris/pull/14003 # Proposed changes 1. remove cast from numeric literal to decimal literal 2. fix decimal scale bug: we will miss scale if first operand scale is smaller than the second operand scale. For example, 1.0 + l_quantity, decimal(2,1) + decimal(15,2), we expect (15,2), not (15,1) Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 4. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 5. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 6. Does it need to update dependencies: - [ ] Yes - [ ] No 7. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly commented on pull request #14003: [enhance](nereids) remove cast from numeric literal to decimal
englefly commented on PR #14003: URL: https://github.com/apache/doris/pull/14003#issuecomment-1305038673 @morrySnow PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Henry2SS opened a new pull request, #14004: [fix](typo) fix function typo which is potentially misleading
Henry2SS opened a new pull request, #14004: URL: https://github.com/apache/doris/pull/14004 # Proposed changes Issue Number: close #xxx ## Problem summary fix function typo which is potentially misleading ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #10392: [Enhancement][Vectorized] Use SIMD to skip batches of null data in nu…
hello-stephen commented on PR #10392: URL: https://github.com/apache/doris/pull/10392#issuecomment-1305042864 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.18 seconds load time: 444 seconds storage size: 17180200823 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221107034804_clickbench_pr_40780.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13978: [Bug](Bitmap) fix sub_bitmap calculate wrong result to return null
github-actions[bot] commented on PR #13978: URL: https://github.com/apache/doris/pull/13978#issuecomment-1305043281 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13978: [Bug](Bitmap) fix sub_bitmap calculate wrong result to return null
github-actions[bot] commented on PR #13978: URL: https://github.com/apache/doris/pull/13978#issuecomment-1305043292 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #13763: [feature](function)add search functions: multi_search_all_positions & multi_match_any
dataroaring merged PR #13763: URL: https://github.com/apache/doris/pull/13763 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (7ffe88b579 -> e8d2fb6778)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 7ffe88b579 [feature-array](array-type) Add array function
array_popback (#13641)
add e8d2fb6778 [feature](function)add search functions:
multi_search_all_positions & multi_match_any (#13763)
No new revisions were added by this update.
Summary of changes:
be/src/vec/CMakeLists.txt | 2 +
.../functions/functions_multi_string_position.cpp | 236 +
.../functions/functions_multi_string_search.cpp| 283 +
be/src/vec/functions/regexps.h | 256 +++
be/src/vec/functions/simple_function_factory.h | 4 +
.../search/multi_match_any.md} | 28 +-
.../search/multi_search_all_positions.md} | 32 ++-
.../search/multi_match_any.md} | 28 +-
.../search/multi_search_all_positions.md} | 41 ++-
gensrc/script/doris_builtins_functions.py | 4 +
.../test_multi_string_position.out | 25 ++
.../search_functions/test_multi_string_search.out} | 60 ++---
.../test_multi_string_position.groovy | 31 +++
.../test_multi_string_search.groovy| 41 +++
14 files changed, 985 insertions(+), 86 deletions(-)
create mode 100644 be/src/vec/functions/functions_multi_string_position.cpp
create mode 100644 be/src/vec/functions/functions_multi_string_search.cpp
create mode 100644 be/src/vec/functions/regexps.h
copy docs/en/docs/sql-manual/sql-functions/{date-time-functions/months_diff.md
=> string-functions/search/multi_match_any.md} (56%)
copy docs/en/docs/sql-manual/sql-functions/{spatial-functions/st_polygon.md =>
string-functions/search/multi_search_all_positions.md} (51%)
copy
docs/zh-CN/docs/sql-manual/sql-functions/{date-time-functions/months_diff.md =>
string-functions/search/multi_match_any.md} (57%)
copy docs/zh-CN/docs/sql-manual/sql-functions/{bitmap-functions/bitmap_not.md
=> string-functions/search/multi_search_all_positions.md} (51%)
create mode 100644
regression-test/data/query_p0/sql_functions/search_functions/test_multi_string_position.out
copy regression-test/data/{query/sql_functions/test_in_expr.out =>
query_p0/sql_functions/search_functions/test_multi_string_search.out} (59%)
create mode 100644
regression-test/suites/query_p0/sql_functions/search_functions/test_multi_string_position.groovy
create mode 100644
regression-test/suites/query_p0/sql_functions/search_functions/test_multi_string_search.groovy
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #10392: [Enhancement][Vectorized] Use SIMD to skip batches of null data in nu…
HappenLee commented on code in PR #10392:
URL: https://github.com/apache/doris/pull/10392#discussion_r1014982859
##
be/src/vec/aggregate_functions/aggregate_function_null.h:
##
@@ -219,6 +219,94 @@ class AggregateFunctionNullUnary final
}
}
+void add_not_nullable(AggregateDataPtr __restrict place, const IColumn**
columns,
+ size_t row_num, Arena* arena) const {
+const ColumnNullable* column = assert_cast(columns[0]);
+this->set_flag(place);
+const IColumn* nested_column = &column->get_nested_column();
+this->nested_function->add(this->nested_place(place), &nested_column,
row_num, arena);
+}
+
+void add_batch(size_t batch_size, AggregateDataPtr* places, size_t
place_offset,
+ const IColumn** columns, Arena* arena, bool agg_many) const
override {
+int processed_records_num = 0;
+
+// we can use column->has_null() to judge whether whole batch of data
is null and skip batch,
+// but it's maybe too coarse-grained.
+#ifdef __AVX2__
+const ColumnNullable* column = assert_cast(columns[0]);
+// The overhead introduced is negligible here, just an extra memory
read from NullMap
+const NullMap& null_map_data = column->get_null_map_data();
+
+// NullMap use uint8_t type to indicate values is null or not, 1
indicates null, 0 versus.
+// It's important to keep consistent with element type size in NullMap
+constexpr int simd_batch_size = 256 / (8 * sizeof(uint8_t));
+__m256i all0 = _mm256_setzero_si256();
+auto to_read_null_map_position = reinterpret_cast(null_map_data.data());
+
+while (processed_records_num + simd_batch_size < batch_size) {
+to_read_null_map_position = to_read_null_map_position +
processed_records_num;
+// load unaligned data from null_map, 1 means value is null, 0
versus
+__m256i f =
+_mm256_loadu_si256(reinterpret_cast(to_read_null_map_position));
+int mask = _mm256_movemask_epi8(_mm256_cmpgt_epi8(f, all0));
+// all data is null
+if (mask == 0x) {
+} else if (mask == 0) { // all data is not null
+for (size_t i = processed_records_num; i <
processed_records_num + simd_batch_size;
+ i++) {
+AggregateFunctionNullUnary::add_not_nullable(places[i] +
place_offset, columns,
+ i, arena);
+}
+} else {
+// data is partly null
Review Comment:
here still can use SSE to speed up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #14000: [typo](docs)add udf doc and optimize udf regression test
hello-stephen commented on PR #14000: URL: https://github.com/apache/doris/pull/14000#issuecomment-1305045574 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 33.91 seconds load time: 457 seconds storage size: 17177814414 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221107035451_clickbench_pr_40759.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring opened a new pull request, #14005: [improvement](fuzzy) add thread fuzzy
dataroaring opened a new pull request, #14005: URL: https://github.com/apache/doris/pull/14005 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly commented on pull request #13990: [opt](nereids)prune runtime filters which cannot reduce the tuple number of probe table
englefly commented on PR #13990: URL: https://github.com/apache/doris/pull/13990#issuecomment-1305047387 @morrySnow @sohardforaname PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dutyu closed pull request #13894: [enhancement](decommission) speed up decommission process (#13579)
dutyu closed pull request #13894: [enhancement](decommission) speed up decommission process (#13579) URL: https://github.com/apache/doris/pull/13894 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dutyu opened a new pull request, #14006: [enhancement](decommission) speed up decommission process (#13579)
dutyu opened a new pull request, #14006: URL: https://github.com/apache/doris/pull/14006 # Proposed changes Issue Number: close #13579 ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [x] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [x] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #14002: [Bug](udf) Make UDF's type always nullable
hello-stephen commented on PR #14002: URL: https://github.com/apache/doris/pull/14002#issuecomment-1305066223 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.17 seconds load time: 450 seconds storage size: 17179081655 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221107042652_clickbench_pr_40798.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #14003: [enhance](nereids) remove cast from numeric literal to decimal
hello-stephen commented on PR #14003: URL: https://github.com/apache/doris/pull/14003#issuecomment-1305069109 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 33.92 seconds load time: 465 seconds storage size: 17185418741 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221107043339_clickbench_pr_40799.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] YangShaw commented on issue #13616: support CTE in Nereids
YangShaw commented on issue #13616: URL: https://github.com/apache/doris/issues/13616#issuecomment-1305071090 This issue is used to record the progress of the implementation of CTE functions and subtasks in nereids. Currently, the CTE syntax is implemented by fully inlining it(in #12742), without cost-based optimization. Based on approaches and algorithms in paper [Optimization of Common Table Expressions in MPP Database Systems](http://www.vldb.org/pvldb/vol8/p1704-elhelw.pdf), there are still a few more subtasks to be done, which may refactor the current CTE implementation. Subtasks for this issue: - [ ] the completed CTE-related syntax, except keyword RECURSIVE, which is not supported in existing implementation; https://github.com/apache/doris/pull/12742 - [ ] data structures (and corresponding logical and physical operators, if necessary) for CTE, includes CTEProducer, CTEConsumer, CTEAnchor, Sequence; - [ ] transformation rules for CTE, includes anchorToNoOp, anchorToSequence, consumerToInlinedCTEPlan; - [ ] algorithm to check and eliminate invalid CTE derived plan(generated by CTE transformation rules); - [ ] optimization for CTEConsumer, includes predicate push down(specific to CTE), inline single-used CTEs, eliminate unused CTEs recursively; - [ ] choose the best CTE plan alternatives by CBO; - [ ] materialize results of CTEProducer, and deliver those results to corresponding CTEConsumer in correct order(to avoid deadlocks); -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow opened a new issue, #13616: support CTE in Nereids
morrySnow opened a new issue, #13616: URL: https://github.com/apache/doris/issues/13616 This issue is used to record the progress of the implementation of CTE functions and subtasks in nereids. Currently, the CTE syntax is implemented by fully inlining it(in #12742), without cost-based optimization. Based on approaches and algorithms in paper [Optimization of Common Table Expressions in MPP Database Systems](http://www.vldb.org/pvldb/vol8/p1704-elhelw.pdf), there are still a few more subtasks to be done, which may refactor the current CTE implementation. Subtasks for this issue: - [x] the completed CTE-related syntax, except keyword RECURSIVE, which is not supported in existing implementation; https://github.com/apache/doris/pull/12742 - [ ] data structures (and corresponding logical and physical operators, if necessary) for CTE, includes CTEProducer, CTEConsumer, CTEAnchor, Sequence; - [ ] transformation rules for CTE, includes anchorToNoOp, anchorToSequence, consumerToInlinedCTEPlan; - [ ] algorithm to check and eliminate invalid CTE derived plan(generated by CTE transformation rules); - [ ] optimization for CTEConsumer, includes predicate push down(specific to CTE), inline single-used CTEs, eliminate unused CTEs recursively; - [ ] choose the best CTE plan alternatives by CBO; - [ ] materialize results of CTEProducer, and deliver those results to corresponding CTEConsumer in correct order(to avoid deadlocks); -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #14004: [fix](typo) fix function typo which is potentially misleading
hello-stephen commented on PR #14004: URL: https://github.com/apache/doris/pull/14004#issuecomment-1305074322 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.25 seconds load time: 447 seconds storage size: 17178576955 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221107044558_clickbench_pr_40811.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] XieJiann opened a new pull request, #14007: [feat](Nereids) add graph simplifier
XieJiann opened a new pull request, #14007: URL: https://github.com/apache/doris/pull/14007 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #14005: [improvement](fuzzy) add thread fuzzy
hello-stephen commented on PR #14005: URL: https://github.com/apache/doris/pull/14005#issuecomment-1305100252 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.27 seconds load time: 448 seconds storage size: 17180843642 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221107052404_clickbench_pr_40832.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #13984: [fix](thread) catch exception of std::thread
yiguolei merged PR #13984: URL: https://github.com/apache/doris/pull/13984 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.1-lts updated: [fix](thread) catch exception of std::thread (#13984)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-1.1-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.1-lts by this push:
new 2db08f9499 [fix](thread) catch exception of std::thread (#13984)
2db08f9499 is described below
commit 2db08f9499ffdd54ca1d09f6ee3a8ad5584da35f
Author: Yongqiang YANG <98214048+dataroar...@users.noreply.github.com>
AuthorDate: Mon Nov 7 13:36:55 2022 +0800
[fix](thread) catch exception of std::thread (#13984)
* [fix](thread) catch exception of std::thread
Some users encounter core dump due to excpetion of std::thread in
HashJoinNode::open().
* LOG(WARNING) rather than LOG(WARN)
---
be/src/exec/hash_join_node.cpp | 7 ++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/be/src/exec/hash_join_node.cpp b/be/src/exec/hash_join_node.cpp
index 515f7177bb..54fbeadd0d 100644
--- a/be/src/exec/hash_join_node.cpp
+++ b/be/src/exec/hash_join_node.cpp
@@ -236,7 +236,12 @@ Status HashJoinNode::open(RuntimeState* state) {
// main thread
std::promise thread_status;
add_runtime_exec_option("Hash Table Built Asynchronously");
-std::thread(bind(&HashJoinNode::build_side_thread, this, state,
&thread_status)).detach();
+try {
+std::thread(bind(&HashJoinNode::build_side_thread, this, state,
&thread_status)).detach();
+} catch (const std::system_error& e) {
+LOG(WARNING) << "create thread fail, " << e.what();
+return Status::InternalError(e.what());
+}
if (!_runtime_filter_descs.empty()) {
RuntimeFilterSlots runtime_filter_slots(_probe_expr_ctxs,
_build_expr_ctxs,
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #13954: [chore](gutil) remove some gutil macros and solve some macro conflict with brpc
yiguolei merged PR #13954: URL: https://github.com/apache/doris/pull/13954 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (e8d2fb6778 -> 32fea672b0)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from e8d2fb6778 [feature](function)add search functions: multi_search_all_positions & multi_match_any (#13763) add 32fea672b0 [chore](gutil) remove some gutil macros and solve some macro conflict with brpc (#13954) No new revisions were added by this update. Summary of changes: be/src/agent/agent_server.cpp | 6 +- be/src/agent/agent_server.h| 2 - be/src/common/signal_handler.h | 4 +- be/src/gutil/integral_types.h | 19 -- be/src/gutil/macros.h | 84 +- be/src/gutil/once.cc | 2 +- be/src/gutil/port.h| 58 +++--- be/src/gutil/strings/numbers.cc| 4 +- be/src/gutil/strings/numbers.h | 2 +- be/src/gutil/strtoint.cc | 21 --- be/src/runtime/cache/result_node.cpp | 2 +- be/src/runtime/fragment_mgr.cpp| 6 +- be/src/runtime/memory/thread_mem_tracker_mgr.h | 5 +- be/src/runtime/thread_context.h| 3 +- be/src/service/brpc.h | 4 -- be/src/service/brpc_conflict.h | 48 --- be/src/util/threadpool.cpp | 2 +- be/src/vec/runtime/vdata_stream_recvr.cpp | 2 + be/test/util/easy_json-test.cpp| 18 -- 19 files changed, 36 insertions(+), 256 deletions(-) delete mode 100644 be/src/service/brpc_conflict.h - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13994: [feature](multi-catalog) Support data on s3-compatible oss and support aliyun DLF
github-actions[bot] commented on PR #13994: URL: https://github.com/apache/doris/pull/13994#issuecomment-1305110908 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #13985: [enhancement](profile) add instanceNum, tableIds to profile.
yiguolei commented on code in PR #13985:
URL: https://github.com/apache/doris/pull/13985#discussion_r1015023826
##
fe/fe-core/src/main/java/org/apache/doris/common/util/ProfileManager.java:
##
@@ -67,6 +67,11 @@ public class ProfileManager {
public static final String DEFAULT_DB = "Default Db";
public static final String SQL_STATEMENT = "Sql Statement";
public static final String IS_CACHED = "Is Cached";
+
+public static final String TOTAL_INSTANCES_NUM = "Total Instances Num";
+
+public static final String INSTANCES_NUM_PER_FRAGMENT = "Instances Num Per
Fragment";
Review Comment:
Need backends list --> instance num
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [security](fe jar) upgrade commons-codec:commons-codec to 1.13 #13951
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 3c8524b9d8 [security](fe jar) upgrade commons-codec:commons-codec to 1.13 #13951 3c8524b9d8 is described below commit 3c8524b9d8c9147754703472620e0a554b270d41 Author: zhoumengyks <111965739+zhoumeng...@users.noreply.github.com> AuthorDate: Mon Nov 7 13:50:07 2022 +0800 [security](fe jar) upgrade commons-codec:commons-codec to 1.13 #13951 --- fe/pom.xml| 2 +- fs_brokers/apache_hdfs_broker/pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/fe/pom.xml b/fe/pom.xml index 87e1bf33a0..0fd669cfdd 100644 --- a/fe/pom.xml +++ b/fe/pom.xml @@ -169,7 +169,7 @@ under the License. 2.2 1.4 -1.9 +1.13 2.6 3.9 2.2 diff --git a/fs_brokers/apache_hdfs_broker/pom.xml b/fs_brokers/apache_hdfs_broker/pom.xml index a1ca1fe0d5..8aaca1aad8 100644 --- a/fs_brokers/apache_hdfs_broker/pom.xml +++ b/fs_brokers/apache_hdfs_broker/pom.xml @@ -138,7 +138,7 @@ under the License. commons-codec commons-codec -1.9 +1.13 - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #13951: [security](fe jar) upgrade commons-codec:commons-codec to 1.13
yiguolei merged PR #13951: URL: https://github.com/apache/doris/pull/13951 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #13849: [typo](docs) fix docs，delete redundant words
yiguolei merged PR #13849: URL: https://github.com/apache/doris/pull/13849 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [typo](docs) fix docs，delete redundant words #13849
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 7254999f02 [typo](docs) fix docs，delete redundant words #13849 7254999f02 is described below commit 7254999f0266e36088f64b8c3f22ee2080cc8b01 Author: Wanghuan AuthorDate: Mon Nov 7 13:51:10 2022 +0800 [typo](docs) fix docs，delete redundant words #13849 --- docs/zh-CN/docs/data-operate/import/import-scenes/load-atomicity.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh-CN/docs/data-operate/import/import-scenes/load-atomicity.md b/docs/zh-CN/docs/data-operate/import/import-scenes/load-atomicity.md index 5866ec11af..15e84a8066 100644 --- a/docs/zh-CN/docs/data-operate/import/import-scenes/load-atomicity.md +++ b/docs/zh-CN/docs/data-operate/import/import-scenes/load-atomicity.md @@ -28,7 +28,7 @@ under the License. Doris 中的所有导入操作都有原子性保证，即一个导入作业中的数据要么全部成功，要么全部失败。不会出现仅部分数据导入成功的情况。 -在 [BROKER LOAD](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/BROKER-LOAD.md) 中我们也可以实现多多表的原子性导入。 +在 [BROKER LOAD](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/BROKER-LOAD.md) 中我们也可以实现多表的原子性导入。 对于表所附属的 [物化视图](../../../advanced/materialized-view.md)，也同时保证和基表的原子性和一致性。 - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #14006: [enhancement](decommission) speed up decommission process (#13579)
hello-stephen commented on PR #14006: URL: https://github.com/apache/doris/pull/14006#issuecomment-1305126705 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.58 seconds load time: 446 seconds storage size: 17179747628 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221107060320_clickbench_pr_40853.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #14007: [feat](Nereids) add graph simplifier
hello-stephen commented on PR #14007: URL: https://github.com/apache/doris/pull/14007#issuecomment-1305133608 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.34 seconds load time: 443 seconds storage size: 1714254 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221107061011_clickbench_pr_40858.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #13953: [fix](repeat)remove unmaterialized expr from repeat node
yiguolei merged PR #13953: URL: https://github.com/apache/doris/pull/13953 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (7254999f02 -> bb9182d602)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 7254999f02 [typo](docs) fix docs，delete redundant words #13849
add bb9182d602 [fix](repeat)remove unmaterialized expr from repeat node
(#13953)
No new revisions were added by this update.
Summary of changes:
.../org/apache/doris/analysis/GroupingInfo.java| 22 +
.../test_subquery_grouping.out}| 1 +
.../correctness_p0/test_subquery_grouping.groovy | 94 ++
3 files changed, 117 insertions(+)
copy regression-test/data/{correctness/test_crossjoin_inlineview_slot.out =>
correctness_p0/test_subquery_grouping.out} (98%)
create mode 100644
regression-test/suites/correctness_p0/test_subquery_grouping.groovy
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zy-kkk opened a new pull request, #14010: [typo](docs)fix config doc
zy-kkk opened a new pull request, #14010: URL: https://github.com/apache/doris/pull/14010 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei closed pull request #13592: [fix](string) Fix over-allocated memory for string type (#13167)
yiguolei closed pull request #13592: [fix](string) Fix over-allocated memory for string type (#13167) URL: https://github.com/apache/doris/pull/13592 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei opened a new pull request, #14011: 1.1.4-rc01
yiguolei opened a new pull request, #14011: URL: https://github.com/apache/doris/pull/14011 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] XieJiann closed pull request #13902: [feature](Nereids) add graph simplifier
XieJiann closed pull request #13902: [feature](Nereids) add graph simplifier URL: https://github.com/apache/doris/pull/13902 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman closed pull request #13760: [feature](new-scan) enable new scan by default
morningman closed pull request #13760: [feature](new-scan) enable new scan by default URL: https://github.com/apache/doris/pull/13760 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13941: [fix](JSON) Fail to parse JSONPath (libc++)
github-actions[bot] commented on PR #13941: URL: https://github.com/apache/doris/pull/13941#issuecomment-1305151274 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14011: [chore](version) rename version to 1.1-lts
github-actions[bot] commented on PR #14011: URL: https://github.com/apache/doris/pull/14011#issuecomment-1305154216 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14011: [chore](version) rename version to 1.1-lts
github-actions[bot] commented on PR #14011: URL: https://github.com/apache/doris/pull/14011#issuecomment-1305154243 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12620: [refactor](cv)wait on condition variable more gently
github-actions[bot] commented on PR #12620: URL: https://github.com/apache/doris/pull/12620#issuecomment-1305159088 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12620: [refactor](cv)wait on condition variable more gently
github-actions[bot] commented on PR #12620: URL: https://github.com/apache/doris/pull/12620#issuecomment-1305159116 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14010: [typo](docs)fix config doc
github-actions[bot] commented on PR #14010: URL: https://github.com/apache/doris/pull/14010#issuecomment-1305169156 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14010: [typo](docs)fix config doc
github-actions[bot] commented on PR #14010: URL: https://github.com/apache/doris/pull/14010#issuecomment-1305169236 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] freemandealer opened a new issue, #14012: [Enhancement] shrink reserved buffer for page builder to save memory
freemandealer opened a new issue, #14012: URL: https://github.com/apache/doris/issues/14012 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description For table with hundreds of text type columns, flushing its memory table may cost huge memory. These memory are consumed when initializing page builder, as it reserve 1MB for each column. Shrink the reservation may reduce memory consumption substantially in load process, making Doris more capable for larger load concurrency. ### Solution 1. Reduce default reserved buffer for each Page Builder from 1M to 64KB and let it grow if needed. 2. Provide tests to see if this brings any performance degrading. ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #14011: [chore](version) rename version to 1.1-lts
yiguolei merged PR #14011: URL: https://github.com/apache/doris/pull/14011 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.1-lts updated: [chore](version) rename version to 1.1-lts (#14011)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch branch-1.1-lts in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-1.1-lts by this push: new 07344af06a [chore](version) rename version to 1.1-lts (#14011) 07344af06a is described below commit 07344af06a6e572cf734cff69d3afce267111bbe Author: yiguolei <676222...@qq.com> AuthorDate: Mon Nov 7 15:33:06 2022 +0800 [chore](version) rename version to 1.1-lts (#14011) Co-authored-by: yiguolei --- gensrc/script/gen_build_version.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gensrc/script/gen_build_version.sh b/gensrc/script/gen_build_version.sh index 8c22c39f5e..c6f81260b4 100755 --- a/gensrc/script/gen_build_version.sh +++ b/gensrc/script/gen_build_version.sh @@ -25,7 +25,7 @@ # contains the build version based on the git hash or svn revision. ## -build_version="1.1.3-rc02" +build_version="1.1-lts" unset LANG unset LC_CTYPE - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13905: [refractor](array) refractor DataTypeArray from_string
github-actions[bot] commented on PR #13905: URL: https://github.com/apache/doris/pull/13905#issuecomment-1305199181 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13905: [refractor](array) refractor DataTypeArray from_string
github-actions[bot] commented on PR #13905: URL: https://github.com/apache/doris/pull/13905#issuecomment-1305199219 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhouaoe closed issue #13665: [Enhancement] Doris broker support aliyun-oss
zhouaoe closed issue #13665: [Enhancement] Doris broker support aliyun-oss URL: https://github.com/apache/doris/issues/13665 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhouaoe opened a new pull request, #14013: [Enhancement] Doris broker support aliyun-oss #13665
zhouaoe opened a new pull request, #14013:
URL: https://github.com/apache/doris/pull/14013
# Proposed changes
Issue Number: close #13665
## Problem summary
Describe your changes.
1 Upgrade fs_broker module hadoop2.8.3->hadoop2.9.1
2 Broker support oss://
3 Version of jar file hadoop-huaweicloud used by broker is set to 2.8.3
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [x] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [x] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [x] No
- [ ] No Need
4. Does it need to update dependencies:
- [x] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [x] No
## Further comments
**1. Test case :doris broker升级hadoop2.9.1后访问obs**
验证hadoop2.9.1兼容华为obs的访问
LOAD LABEL demo.load_oss_label_hw (
DATA INFILE(
"obs://weinan-test1.obs.cn-east-3.myhuaweicloud.com/data2.csv"
) INTO TABLE example_tbl_hw COLUMNS TERMINATED BY ","
) WITH BROKER "broker_za" (
"fs.obs.access.key" = "x",
"fs.obs.secret.key" = "xxx",
"fs.obs.endpoint" = "https://obs.cn-east-3.myhuaweicloud.com";
)
测试结果符合预期，兼容2.8.3版本的hadoop
执行结果：
后台记录：
结果检查：
**2. Test cast :Doris broker支持OSS协议测试**
1 Load with Broker
用例：先导入6条数据，再通过OSS导入7条数据（在原始数据基础上增加了一条）
原始数据：
1,2017-10-01,北京,20,0,2017-10-01 06:00:00,20,10,10
1,2017-10-01,北京,20,0,2017-10-01 07:00:00,15,2,2
10001,2017-10-01,北京,30,1,2017-10-01 17:05:45,2,22,22
10002,2017-10-02,上海,20,1,2017-10-02 12:59:12,200,5,5
10003,2017-10-02,广州,32,0,2017-10-02 11:20:00,30,11,11
10004,2017-10-01,深圳,35,0,2017-10-01 10:00:15,100,3,3
10004,2017-10-03,深圳,35,0,2017-10-03 10:20:22,11,6,6
OSS上的数据data.csv
1,2017-10-01,北京,20,0,2017-10-01 06:00:00,20,10,10
1,2017-10-01,北京,20,0,2017-10-01 07:00:00,15,2,2
10001,2017-10-01,北京,30,1,2017-10-01 17:05:45,2,22,22
10002,2017-10-02,上海,20,1,2017-10-02 12:59:12,200,5,5
10003,2017-10-02,广州,32,0,2017-10-02 11:20:00,30,11,11
10004,2017-10-01,深圳,35,0,2017-10-01 10:00:15,100,3,3
10004,2017-10-03,深圳,35,0,2017-10-03 10:20:22,11,6,6
10005,2017-10-03,深圳,35,0,2017-10-03 10:20:22,11,6,6
导入语句
LOAD LABEL demo.load_oss_label_1
(
DATA INFILE("oss://otsosstest/doris/data.csv")
INTO TABLE example_tbl
COLUMNS TERMINATED BY ","
)
WITH BROKER "broker_za"
(
"fs.oss.endpoint" = "https://x";,
"fs.oss.accessKeyId" = "x",
"fs.oss.accessKeySecret"="x"
)
执行结果：符合预期
1.前端执行：
2.Load任务
3.数据检查
2 Export with Borker
用例：将刚才的7条数据导入到OSS上
EXPORT TABLE demo.example_tbl
TO "oss://otsosstest/doris/export_broker/01/"
PROPERTIES
(
"label" = "export_from_doris_18",
"column_separator"=",",
"timeout" = "3600"
)
WITH BROKER "broker_za"
(
"fs.oss.endpoint" = "https://oss-cn-hangzhou.aliyuncs.com";,
"fs.oss.accessKeyId" = "x",
"fs.oss.accessKeySecret"="x"
);
测试结果符合预期
执行结果
后台执行结果
oss上导出的文件：
oss上导出的文件内容：
3 Outfile export with Broker
select * from demo.example_tbl
into outfile "oss://streamoss-1/doris/newbroker/01"
FORMAT AS CSV
PROPERTIES
(
"broker.name" = "broker_za",
"broker.fs.oss.endpoint" = "https://oss-cn-hangzhou.aliyuncs.com";,
"broker.fs.oss.accessKeyId" = "xx",
"broker.fs.oss.accessKeySecret"="xxx",
"column_separator" = ",",
"line_delimiter" = "\n",
检查结果：符合预期
[GitHub] [doris] freemandealer opened a new pull request, #14014: [enhancement](load) shrink reserved buffer for page builder (#14012)
freemandealer opened a new pull request, #14014: URL: https://github.com/apache/doris/pull/14014 For table with hundreds of text type columns, flushing its memtable may cost huge memory. These memory are consumed when initializing page builder, as it reserves 1MB for each column. So memory consumption grows in proportion with column number. Shrinking the reservation may reduce memory substantially in load process. Signed-off-by: freemandealer # Proposed changes Issue Number: close #14012 For table with hundreds of text type columns, flushing its memtable may cost huge memory. These memory are consumed when initializing page builder, as it reserves 1MB for each column. So memory consumption grows in proportion with column number. Shrinking the reservation may reduce memory substantially in load process, as shown in the test results listed below. Shrinking will not limit the size of dict page size -- it could still grow when needed. Thus, we can make the most of dict encoding while achive reasonable memory consumption. No performance degrading is observed in the tests, proving buffer growth brings little overheads. We believe this commit will make Doris more capable for larger load concurrency. Test config: | CPU | Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz 96 core| | | | | RAM | 375G | | SSD | Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO | | Data | GithubEvent, 3.1G single json, 664 columns, dict size will trigger buffer growth | Test results: - single load | | Resv 1M (original) | Resv 64K + Grow | | | --- | --- | | LoadTime:| 99.4s | 94.7s (-5%) | | Peak ProcessMem: | 12G | 11G (-1G) | | Peak LoadMem:| 17G | 13G (-4G) | | Peak Orphan: | 11G | 8G (-3G)| - multiple loads (concurrency = 10) | | Resv 1M (original) | Resv 64K + Grow | | | --- | --- | | LoadTime:| 183.2s | 182.9s (-1%)| | Peak ProcessMem: | 80G | 73G (-7G) | | Peak LoadMem:| 55G | 51G (-4G) | | Peak Orphan: | 57G | 54G (-4G) | ## Checklist(Required) 1. Does it affect the original behavior: - [X] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [X] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [X] No Need 4. Does it need to update dependencies: - [ ] Yes - [X] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [X] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
