(doris) branch master updated: [Fix](TrinoConnector) it will return error when query `information_schema.tables` within `trino-connector` catalog (#49912)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 7710b179288 [Fix](TrinoConnector) it will return error when query
`information_schema.tables` within `trino-connector` catalog (#49912)
7710b179288 is described below
commit 7710b1792883bcf119f04258b9ee3884c9922aa0
Author: Tiewei Fang
AuthorDate: Mon Apr 14 09:10:14 2025 +0800
[Fix](TrinoConnector) it will return error when query
`information_schema.tables` within `trino-connector` catalog (#49912)
Fix that it will return error when query `information_schema.tables`
within `trino-connector` catalog
---
.../java/org/apache/doris/catalog/TableIf.java | 1 +
.../test_trinoconnector_information_schema.groovy | 44 ++
2 files changed, 45 insertions(+)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
index 2885c427319..96fd8da54af 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
@@ -461,6 +461,7 @@ public interface TableIf {
case ICEBERG_EXTERNAL_TABLE:
case PAIMON_EXTERNAL_TABLE:
case MATERIALIZED_VIEW:
+case TRINO_CONNECTOR_EXTERNAL_TABLE:
return "BASE TABLE";
default:
return null;
diff --git
a/regression-test/suites/external_table_p0/trino_connector/test_trinoconnector_information_schema.groovy
b/regression-test/suites/external_table_p0/trino_connector/test_trinoconnector_information_schema.groovy
new file mode 100644
index 000..76465e4cd78
--- /dev/null
+++
b/regression-test/suites/external_table_p0/trino_connector/test_trinoconnector_information_schema.groovy
@@ -0,0 +1,44 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_trinoconnector_information_schema",
"p0,external,hive,external_docker,external_docker_hive") {
+String enabled = context.config.otherConfigs.get("enableHiveTest")
+if (enabled != null && enabled.equalsIgnoreCase("true")) {
+def host_ips = new ArrayList()
+String[][] backends = sql """ show backends """
+for (def b in backends) {
+host_ips.add(b[1])
+}
+String [][] frontends = sql """ show frontends """
+for (def f in frontends) {
+host_ips.add(f[1])
+}
+dispatchTrinoConnectors(host_ips.unique())
+String hms_port = context.config.otherConfigs.get("hive2HmsPort")
+String catalog_name = "test_trinoconnector_information_schema"
+String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+
+sql """drop catalog if exists ${catalog_name}"""
+sql """create catalog if not exists ${catalog_name} properties (
+"type"="trino-connector",
+"trino.connector.name"="hive",
+'trino.hive.metastore.uri' =
'thrift://${externalEnvIp}:${hms_port}'
+);"""
+
+sql """ select * from ${catalog_name}.information_schema.tables limit
10""";
+}
+}
\ No newline at end of file
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [Fix](TrinoConnector) it will return error when query `information_schema.tables` within `trino-connector` catalog #49912 [doris]
hello-stephen commented on PR #50006: URL: https://github.com/apache/doris/pull/50006#issuecomment-2800245819 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-49912-branch-3.0 updated (c5e18ef5520 -> 1659cec7923)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch auto-pick-49912-branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
from c5e18ef5520 branch-3.0: [fix](multi-catalog) Fix multi-thread issue in
hive/iceberg writer commit meta-info to fe. #49842 (#49863)
add 1659cec7923 [Fix](TrinoConnector) it will return error when query
`information_schema.tables` within `trino-connector` catalog (#49912)
No new revisions were added by this update.
Summary of changes:
.../java/org/apache/doris/catalog/TableIf.java | 1 +
... test_trinoconnector_information_schema.groovy} | 27 +-
2 files changed, 12 insertions(+), 16 deletions(-)
copy
regression-test/suites/external_table_p0/trino_connector/{hive/test_trino_hive_serde_prop.groovy
=> test_trinoconnector_information_schema.groovy} (68%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] Reapply "[enhancement](hive)Initial support for Hive org.openx.data.jsonserde.JsonSerDe" (#49928) [doris]
morningman commented on PR #49958: URL: https://github.com/apache/doris/pull/49958#issuecomment-2800251504 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [Fix](TrinoConnector) it will return error when query `information_schema.tables` within `trino-connector` catalog #49912 [doris]
dataroaring closed pull request #50006: branch-3.0: [Fix](TrinoConnector) it will return error when query `information_schema.tables` within `trino-connector` catalog #49912 URL: https://github.com/apache/doris/pull/50006 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](test)add some be ut for orc/parquet reader (#49418) [doris]
github-actions[bot] commented on PR #49948: URL: https://github.com/apache/doris/pull/49948#issuecomment-2800252782 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](executor) Add the time-sharing executor framework and use it in the scanner section. [doris]
morningman commented on code in PR #49671:
URL: https://github.com/apache/doris/pull/49671#discussion_r2041244856
##
be/src/vec/exec/executor/time_sharing/time_sharing_task_executor.h:
##
@@ -0,0 +1,183 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "bvar/latency_recorder.h"
+#include "util/threadpool.h"
+#include "vec/exec/executor/listenable_future.h"
+#include "vec/exec/executor/task_executor.h"
+#include "vec/exec/executor/ticker.h"
+#include "vec/exec/executor/time_sharing/multilevel_split_queue.h"
+#include "vec/exec/executor/time_sharing/prioritized_split_runner.h"
+
+namespace doris {
+namespace vectorized {
+
+/**
+ * ThreadSafe
+ */
+class TimeSharingTaskExecutor : public TaskExecutor {
+ENABLE_FACTORY_CREATOR(TimeSharingTaskExecutor);
+
+public:
+struct ThreadConfig {
+std::string thread_name;
+int max_thread_num;
+int min_thread_num;
+int max_queue_size = 0;
+std::weak_ptr cgroup_cpu_ctl;
+};
+
+TimeSharingTaskExecutor(ThreadConfig config, int min_concurrency,
+int guaranteed_concurrency_per_task, int
max_concurrency_per_task,
+std::shared_ptr ticker,
+std::chrono::milliseconds
stuck_split_warning_threshold =
+std::chrono::milliseconds(6),
+std::shared_ptr split_queue
= nullptr);
+
+~TimeSharingTaskExecutor() override;
+
+Status init() override;
+
+Status start() override;
+void stop() override;
+
+Result> create_task(
+const TaskId& task_id, std::function
utilization_supplier,
+int initial_split_concurrency,
+std::chrono::nanoseconds split_concurrency_adjust_frequency,
+std::optional max_concurrency_per_task) override;
+
+Status add_task(const TaskId& task_id, std::shared_ptr
task_handle) override;
Review Comment:
when to use `add_task` and when to use `create_task`?
##
be/src/vec/exec/executor/time_sharing/split_concurrency_controller.h:
##
@@ -0,0 +1,88 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#pragma once
+
+#include
+#include
+#include
+#include
+
+#include "common/factory_creator.h"
+#include "common/status.h"
+
+namespace doris::vectorized {
+
+class SplitConcurrencyController {
+ENABLE_FACTORY_CREATOR(SplitConcurrencyController);
+
+public:
+SplitConcurrencyController(int initial_concurrency,
+ std::chrono::nanoseconds adjustment_interval)
+: _adjustment_interval_nanos(adjustment_interval),
+ _target_concurrency(initial_concurrency) {}
+
+void update(uint64_t nanos, double utilization, int current_concurrency) {
+_validate_args(nanos, utilization, current_concurrency);
+_thread_nanos_since_adjust += nanos;
+if (_should_increase_concurrency(current_concurrency, utilization)) {
+_reset_adjust_counter();
+++_target_concurrency;
+}
+}
+
+int target_concurrency() const { return _target_concurrency; }
+
+void split_finished(uint64_t split_nanos, double utilization, int
current_concurrency) {
Review Comment:
Looks like the `utilization` is always `0.0` for now
[PR] branch-3.0: [Fix](TrinoConnector) it will return error when query `information_schema.tables` within `trino-connector` catalog #49912 [doris]
github-actions[bot] opened a new pull request, #50006: URL: https://github.com/apache/doris/pull/50006 Cherry-picked from #49912 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](TrinoConnector) it will return error when query `information_schema.tables` within `trino-connector` catalog [doris]
morningman merged PR #49912: URL: https://github.com/apache/doris/pull/49912 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-49912-branch-3.0 created (now c5e18ef5520)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-49912-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git at c5e18ef5520 branch-3.0: [fix](multi-catalog) Fix multi-thread issue in hive/iceberg writer commit meta-info to fe. #49842 (#49863) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](parquet)fix when hive_parquet_use_column_names=false && read partition tb cause be core. [doris]
morningman merged PR #49966: URL: https://github.com/apache/doris/pull/49966 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](parquet)fix when hive_parquet_use_column_names=false && read partition tb cause be core. (#49966)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 68cfef6d67c [fix](parquet)fix when hive_parquet_use_column_names=false
&& read partition tb cause be core. (#49966)
68cfef6d67c is described below
commit 68cfef6d67ce588c2b079c53a2204a471c9acf93
Author: daidai
AuthorDate: Mon Apr 14 09:22:04 2025 +0800
[fix](parquet)fix when hive_parquet_use_column_names=false && read
partition tb cause be core. (#49966)
### What problem does this PR solve?
related pr : #38432
Problem Summary:
when you query hive parquet format partition table, and `set
hive_parquet_use_column_names = false`, maybe you will get :
```
*** SIGABRT unknown detail explain (@0x2f59de) received by PID 3103198 (TID
3110278 OR 0x7f51c8e63640) from PID 3103198; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int,
siginfo_t*, void*) at
/home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:421
1# 0x7F55DFB45520 in /lib/x86_64-linux-gnu/libc.so.6
2# pthread_kill at ./nptl/pthread_kill.c:89
3# raise at ../sysdeps/posix/raise.c:27
4# abort at ./stdlib/abort.c:81
5# __gnu_cxx::__verbose_terminate_handler() [clone .cold] at
../../../../libstdc++-v3/libsupc++/vterminate.cc:75
6# __cxxabiv1::__terminate(void (*)()) at
../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
7# 0x55C8BD4E2041 in
/mnt/disk1/doris-clusters/doris-master/output/be/lib/doris_be
8# 0x55C8BD4E2194 in
/mnt/disk1/doris-clusters/doris-master/output/be/lib/doris_be
9# 0x55C8BD4E2586 in
/mnt/disk1/doris-clusters/doris-master/output/be/lib/doris_be
10# std::__cxx11::basic_string,
std::allocator >::_M_assign(std::__cxx11::basic_string, std::allocator > const&) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.tcc:265
11#
doris::vectorized::ParquetReader::get_next_block(doris::vectorized::Block*,
unsigned long*, bool*) at
/home/zcp/repo_center/doris_master/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:586
The reason is that when `get_next_block` replaces the column name, data
out of bounds occurs.
---
be/src/vec/exec/format/parquet/vparquet_reader.cpp| 3 ++-
.../hive/test_external_catalog_hive_partition.out | Bin 2711 -> 4455 bytes
.../hive/test_external_catalog_hive_partition.groovy | 9 +
3 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/be/src/vec/exec/format/parquet/vparquet_reader.cpp
b/be/src/vec/exec/format/parquet/vparquet_reader.cpp
index b91a7c21cbd..3083fd61ab0 100644
--- a/be/src/vec/exec/format/parquet/vparquet_reader.cpp
+++ b/be/src/vec/exec/format/parquet/vparquet_reader.cpp
@@ -561,6 +561,7 @@ Status ParquetReader::get_next_block(Block* block, size_t*
read_rows, bool* eof)
return Status::OK();
}
+std::vector original_block_column_name = block->get_names();
if (!_hive_use_column_names) {
for (auto i = 0; i < block->get_names().size(); i++) {
auto& col = block->get_by_position(i);
@@ -584,7 +585,7 @@ Status ParquetReader::get_next_block(Block* block, size_t*
read_rows, bool* eof)
if (!_hive_use_column_names) {
for (auto i = 0; i < block->columns(); i++) {
-block->get_by_position(i).name = (*_column_names)[i];
+block->get_by_position(i).name = original_block_column_name[i];
}
block->initialize_index_by_name();
}
diff --git
a/regression-test/data/external_table_p0/hive/test_external_catalog_hive_partition.out
b/regression-test/data/external_table_p0/hive/test_external_catalog_hive_partition.out
index aa1e48a439d..0402feef40e 100644
Binary files
a/regression-test/data/external_table_p0/hive/test_external_catalog_hive_partition.out
and
b/regression-test/data/external_table_p0/hive/test_external_catalog_hive_partition.out
differ
diff --git
a/regression-test/suites/external_table_p0/hive/test_external_catalog_hive_partition.groovy
b/regression-test/suites/external_table_p0/hive/test_external_catalog_hive_partition.groovy
index 32b80f5650d..d34467c4c56 100644
---
a/regression-test/suites/external_table_p0/hive/test_external_catalog_hive_partition.groovy
+++
b/regression-test/suites/external_table_p0/hive/test_external_catalog_hive_partition.groovy
@@ -65,9 +65,18 @@ suite("test_external_catalog_hive_partition",
"p0,external,hive,external_docker,
qt_q06 """ select * from multi_catalog.text_partitioned_columns
where t_int is not null order by t_float """
}
sql """ use `multi_catalog`; """
+sql """ set hive_parquet_use_column_names = true; """
+sql """ set hive_orc_use_column_names = true"""
+
q01_pa
(doris) branch auto-pick-49966-branch-3.0 created (now c5e18ef5520)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-49966-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git at c5e18ef5520 branch-3.0: [fix](multi-catalog) Fix multi-thread issue in hive/iceberg writer commit meta-info to fe. #49842 (#49863) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-2.1: [fix](parquet)fix when hive_parquet_use_column_names=false && read partition tb cause be core. #49966 [doris]
Thearas commented on PR #50008: URL: https://github.com/apache/doris/pull/50008#issuecomment-2800256394 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-2.1: [fix](parquet)fix when hive_parquet_use_column_names=false && read partition tb cause be core. #49966 [doris]
dataroaring closed pull request #50008: branch-2.1: [fix](parquet)fix when hive_parquet_use_column_names=false && read partition tb cause be core. #49966 URL: https://github.com/apache/doris/pull/50008 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] branch-2.1: [fix](parquet)fix when hive_parquet_use_column_names=false && read partition tb cause be core. #49966 [doris]
github-actions[bot] opened a new pull request, #50008: URL: https://github.com/apache/doris/pull/50008 Cherry-picked from #49966 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](nereids)support show functions [doris]
github-actions[bot] commented on PR #49893: URL: https://github.com/apache/doris/pull/49893#issuecomment-2800259161 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-49966-branch-3.0 updated (c5e18ef5520 -> 336c71a9988)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-49966-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git from c5e18ef5520 branch-3.0: [fix](multi-catalog) Fix multi-thread issue in hive/iceberg writer commit meta-info to fe. #49842 (#49863) add 336c71a9988 [fix](parquet)fix when hive_parquet_use_column_names=false && read partition tb cause be core. (#49966) No new revisions were added by this update. Summary of changes: be/src/vec/exec/format/parquet/vparquet_reader.cpp| 3 ++- .../hive/test_external_catalog_hive_partition.out | Bin 2711 -> 4455 bytes .../hive/test_external_catalog_hive_partition.groovy | 9 + 3 files changed, 11 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [fix](parquet)fix when hive_parquet_use_column_names=false && read partition tb cause be core. #49966 [doris]
hello-stephen commented on PR #50007: URL: https://github.com/apache/doris/pull/50007#issuecomment-2800255662 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] Reapply "[enhancement](hive)Initial support for Hive org.openx.data.jsonserde.JsonSerDe" (#49928) [doris]
doris-robot commented on PR #49958: URL: https://github.com/apache/doris/pull/49958#issuecomment-2800266295 TeamCity cloud ut coverage result: Function Coverage: 83.07% (1089/1311) Line Coverage: 66.05% (18156/27489) Region Coverage: 65.50% (8945/13657) Branch Coverage: 55.36% (4821/8708) Coverage Report: http://coverage.selectdb-in.cc/coverage/185fe999cbb402e3301a2b9a518e2ef0c18b5282_185fe999cbb402e3301a2b9a518e2ef0c18b5282_cloud/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](inverted index) Downgrade ICU version to support lower OS versions #47664 [doris]
github-actions[bot] commented on PR #50005: URL: https://github.com/apache/doris/pull/50005#issuecomment-2800228364 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](inverted index) Downgrade ICU version to support lower OS versions #47664 [doris]
github-actions[bot] commented on PR #50005: URL: https://github.com/apache/doris/pull/50005#issuecomment-2800228409 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hive/iceberg) rectify the fs name if path already contains fs [doris]
github-actions[bot] commented on PR #49998: URL: https://github.com/apache/doris/pull/49998#issuecomment-2800272182 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](replica) skip missing version should care catchup [doris]
yujun777 commented on code in PR #4: URL: https://github.com/apache/doris/pull/4#discussion_r2041290940 ## fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java: ## @@ -314,12 +314,7 @@ public List getQueryableReplicas(long visibleVersion, Map
Re: [PR] [fix](replica) skip missing version should care catchup [doris]
yujun777 commented on code in PR #4:
URL: https://github.com/apache/doris/pull/4#discussion_r2041291553
##
fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java:
##
@@ -314,12 +314,7 @@ public List getQueryableReplicas(long
visibleVersion, Map 0 && !allowFailedVersion) {
-continue;
-}
-
-if (!replica.checkVersionCatchUp(visibleVersion, false)) {
Review Comment:
change to : if (!replica.checkVersionCatchUp() && !allowMissingVersions) {
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](replica) skip missing version should care catchup [doris]
yujun777 commented on code in PR #4:
URL: https://github.com/apache/doris/pull/4#discussion_r2041291553
##
fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java:
##
@@ -314,12 +314,7 @@ public List getQueryableReplicas(long
visibleVersion, Map 0 && !allowFailedVersion) {
-continue;
-}
-
-if (!replica.checkVersionCatchUp(visibleVersion, false)) {
Review Comment:
change to : if (!replica.checkVersionCatchUp() && !allowMissingVersions) {
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](inverted index) add ngram tokenizer [doris]
zzzxl1993 commented on code in PR #49472:
URL: https://github.com/apache/doris/pull/49472#discussion_r2041319637
##
be/src/olap/rowset/segment_v2/inverted_index/token_filter/ascii_folding_filter.cpp:
##
@@ -0,0 +1,2014 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "ascii_folding_filter.h"
+
+#include
+
+namespace doris::segment_v2::inverted_index {
+
+ASCIIFoldingFilter::ASCIIFoldingFilter(const TokenStreamPtr& in, bool
preserve_original)
+: DorisTokenFilter(in), _preserve_original(preserve_original),
_output(512, 0) {}
+
+Token* ASCIIFoldingFilter::next(Token* t) {
+if (!_state.empty()) {
+assert(_preserve_original);
+t->set(_state.data(), 0, _state.size());
+t->setPositionIncrement(0);
+return t;
+}
+if (_in->next(t)) {
+const char* buffer = t->termBuffer();
+int32_t length = t->termLength();
+for (int32_t i = 0; i < length;) {
+UChar32 c = U_UNASSIGNED;
+U8_NEXT(buffer, i, length, c);
+if (c < 0) {
+continue;
+}
+if (c >= 0x0080) {
Review Comment:
ASCII
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] opt compaction [doris]
Thearas commented on PR #50021: URL: https://github.com/apache/doris/pull/50021#issuecomment-2800632177 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] opt compaction [doris]
eldenmoon opened a new pull request, #50021: URL: https://github.com/apache/doris/pull/50021 ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [test](UT) add scan normalize predicate BEUT test [doris]
Mryange commented on PR #49971: URL: https://github.com/apache/doris/pull/49971#issuecomment-2800632175 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [fix](iceberg) Fix the failure of creating a table in OBS #49983 [doris]
doris-robot commented on PR #50018: URL: https://github.com/apache/doris/pull/50018#issuecomment-2800597913 TPC-DS: Total hot run time: 197077 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit b4fa6bb30c4fc279471f0b588d98ba6c3a845a14, data reload: false query1 1294919 882 882 query2 6242202219981998 query3 10794 426742834267 query4 61177 28453 23553 23553 query5 5077459 447 447 query6 392 171 172 171 query7 5371326 307 307 query8 289 215 220 215 query9 8148263526022602 query10 451 288 259 259 query11 17097 15165 15620 15165 query12 165 102 102 102 query13 1378472 435 435 query14 10047 699978376999 query15 197 189 179 179 query16 7143467 493 467 query17 1156549 571 549 query18 1779341 321 321 query19 209 164 151 151 query20 115 109 109 109 query21 200 101 104 101 query22 4736443145934431 query23 34782 34317 34176 34176 query24 6169299228792879 query25 539 432 435 432 query26 655 180 177 177 query27 1780376 364 364 query28 4309249324402440 query29 710 478 477 477 query30 255 167 170 167 query31 962 835 862 835 query32 70 61 60 60 query33 452 315 305 305 query34 918 506 533 506 query35 843 754 736 736 query36 1095971 970 970 query37 121 73 67 67 query38 4078393539433935 query39 1528146614871466 query40 203 103 106 103 query41 54 52 50 50 query42 119 103 105 103 query43 543 486 485 485 query44 1206831 805 805 query45 188 175 170 170 query46 1157730 739 730 query47 2013189118741874 query48 485 389 382 382 query49 736 420 410 410 query50 869 460 428 428 query51 7373711972237119 query52 106 97 91 91 query53 279 201 194 194 query54 653 501 481 481 query55 79 81 75 75 query56 273 250 239 239 query57 1260114310971097 query58 218 210 210 210 query59 3030282729022827 query60 295 263 276 263 query61 108 107 130 107 query62 811 676 644 644 query63 230 193 190 190 query64 1917694 647 647 query65 3277320431683168 query66 720 296 303 296 query67 15956 15666 15474 15474 query68 3227599 598 598 query69 444 274 276 274 query70 1155104410541044 query71 402 268 261 261 query72 6419422439953995 query73 753 366 358 358 query74 10276 921091459145 query75 3360260826772608 query76 2091115011011101 query77 592 267 276 267 query78 10676 964296009600 query79 1834595 622 595 query80 1195436 419 419 query81 538 243 239 239 query82 118491 88 88 query83 171 145 140 140 query84 287 80 82 80 query85 1124303 289 289 query86 405 282 308 282 query87 4392425242624252 query88 3919237624592376 query89 419 293 288 288 query90 1895190 190 190 query91 197 148 151 148 query92 61 50 51 50 query93 2481565 558 558 query94 747 303 302 302 query95 352 270 263 263 query96 645 290 281 281 query97 3265317931153115 query98 214 206 195 195 query99 1582131213131312 Total cold run time: 313183 ms Total hot run time: 197077 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t
Re: [PR] [enhance](mtmv) Create a materialized view that only includes the partitions in the query conditions. [doris]
KeeProMise commented on PR #49852: URL: https://github.com/apache/doris/pull/49852#issuecomment-2800626736 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](spill) Refine logics in pipeline task [doris]
doris-robot commented on PR #50010: URL: https://github.com/apache/doris/pull/50010#issuecomment-2800641747 TPC-H: Total hot run time: 34273 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 8969bdacc57357eb18554a045b53aaadef20b330, data reload: false -- Round 1 -- q1 25894 513449734973 q2 2062291 181 181 q3 10382 1252687 687 q4 10238 1033542 542 q5 7530234923662349 q6 196 170 140 140 q7 940 751 617 617 q8 9328130111921192 q9 6887518250875087 q10 6873232218821882 q11 474 279 280 279 q12 353 360 227 227 q13 17790 364131043104 q14 237 222 228 222 q15 542 522 487 487 q16 661 615 595 595 q17 608 856 358 358 q18 7672737171707170 q19 1767954 537 537 q20 333 324 217 217 q21 4308337724672467 q22 1061998 960 960 Total cold run time: 116136 ms Total hot run time: 34273 ms - Round 2, with runtime_filter_mode=off - q1 5215508951275089 q2 237 320 240 240 q3 2159269623312331 q4 1425185414471447 q5 4469441543674367 q6 215 164 134 134 q7 1985192517551755 q8 2577273724732473 q9 7300713771737137 q10 2997319727222722 q11 581 522 505 505 q12 705 748 628 628 q13 3491389733663366 q14 275 279 283 279 q15 530 484 485 484 q16 655 696 668 668 q17 1140153314051405 q18 7711752874567456 q19 829 824 818 818 q20 1957194818591859 q21 5211485347614761 q22 10831062985 985 Total cold run time: 52747 ms Total hot run time: 50909 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch hello-stephen-patch-6 created (now 24fda7535cb)
This is an automated email from the ASF dual-hosted git repository. hellostephen pushed a change to branch hello-stephen-patch-6 in repository https://gitbox.apache.org/repos/asf/doris.git at 24fda7535cb [fix](ci) remove required check 'Build Third Party Libraries (macOS)' on branch-3.0 This branch includes the following new commits: new 24fda7535cb [fix](ci) remove required check 'Build Third Party Libraries (macOS)' on branch-3.0 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch hello-stephen-patch-3 created (now 5972f263a46)
This is an automated email from the ASF dual-hosted git repository. hellostephen pushed a change to branch hello-stephen-patch-3 in repository https://gitbox.apache.org/repos/asf/doris.git at 5972f263a46 [fix](ci) remove required check 'Build Third Party Libraries (macOS)' on branch-3.0 This branch includes the following new commits: new 5972f263a46 [fix](ci) remove required check 'Build Third Party Libraries (macOS)' on branch-3.0 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 01/01: [fix](ci) remove required check 'Build Third Party Libraries (macOS)' on branch-3.0
This is an automated email from the ASF dual-hosted git repository. hellostephen pushed a commit to branch hello-stephen-patch-3 in repository https://gitbox.apache.org/repos/asf/doris.git commit 5972f263a462628569316c4719e10da67e231e9a Author: Dongyang Li AuthorDate: Mon Apr 14 14:50:46 2025 +0800 [fix](ci) remove required check 'Build Third Party Libraries (macOS)' on branch-3.0 --- .asf.yaml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.asf.yaml b/.asf.yaml index 3bc6e4cfceb..1f656a3494b 100644 --- a/.asf.yaml +++ b/.asf.yaml @@ -75,7 +75,6 @@ github: - CheckStyle - Build Broker - Build Third Party Libraries (Linux) - - Build Third Party Libraries (macOS) - FE UT (Doris FE UT) - BE UT (Doris BE UT) - Cloud UT (Doris Cloud UT) @@ -133,4 +132,4 @@ github: - doris-robot - LiBinfeng-01 notifications: - pullrequests_status: commits@doris.apache.org \ No newline at end of file + pullrequests_status: commits@doris.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 01/01: [fix](ci) remove required check 'Build Third Party Libraries (macOS)' on branch-3.0
This is an automated email from the ASF dual-hosted git repository. hellostephen pushed a commit to branch hello-stephen-patch-6 in repository https://gitbox.apache.org/repos/asf/doris.git commit 24fda7535cbbacc8fe4b6725d6513de2c9e0136b Author: Dongyang Li AuthorDate: Mon Apr 14 14:51:41 2025 +0800 [fix](ci) remove required check 'Build Third Party Libraries (macOS)' on branch-3.0 --- .asf.yaml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.asf.yaml b/.asf.yaml index 3bc6e4cfceb..1f656a3494b 100644 --- a/.asf.yaml +++ b/.asf.yaml @@ -75,7 +75,6 @@ github: - CheckStyle - Build Broker - Build Third Party Libraries (Linux) - - Build Third Party Libraries (macOS) - FE UT (Doris FE UT) - BE UT (Doris BE UT) - Cloud UT (Doris Cloud UT) @@ -133,4 +132,4 @@ github: - doris-robot - LiBinfeng-01 notifications: - pullrequests_status: commits@doris.apache.org \ No newline at end of file + pullrequests_status: commits@doris.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](ci) remove required check 'Build Third Party Libraries (macOS)'ā¦ [doris]
hello-stephen commented on PR #50023: URL: https://github.com/apache/doris/pull/50023#issuecomment-2800645551 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Enhancement] (nereids)implement DropResourceCommand in nereids [doris]
starocean999 closed issue #42620: [Enhancement] (nereids)implement DropResourceCommand in nereids URL: https://github.com/apache/doris/issues/42620 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Enhancement] (nereids)implement DropResourceCommand in nereids [doris]
starocean999 closed issue #42620: [Enhancement] (nereids)implement DropResourceCommand in nereids URL: https://github.com/apache/doris/issues/42620 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] (nereids) implement dropResource/RowPolicyCommand in nereids [doris]
starocean999 merged PR #49038: URL: https://github.com/apache/doris/pull/49038 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] impl external be [doris]
HappenLee commented on code in PR #49944:
URL: https://github.com/apache/doris/pull/49944#discussion_r2041160508
##
be/src/vec/exec/format/orc/vorc_reader.cpp:
##
@@ -1176,6 +1178,9 @@ Status OrcReader::set_fill_columns(
}
_row_reader = _reader->createRowReader(_row_reader_options,
_orc_filter.get(),
_string_dict_filter.get());
+if (_read_on_line_mode) {
Review Comment:
the if check better in `_seek_to_read_one_line`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch asf-site updated (42f6616ddd4 -> 30d9a5c11ce)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/doris-website.git discard 42f6616ddd4 Automated deployment with doris branch @ c48e670323787beecea9f8b93ef1ea5009ebd600 new 30d9a5c11ce Automated deployment with doris branch @ c48e670323787beecea9f8b93ef1ea5009ebd600 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (42f6616ddd4) \ N -- N -- N refs/heads/asf-site (30d9a5c11ce) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/2.0/search-index.json | 2 +- docs/3.0/search-index.json | 2 +- docs/dev/search-index.json | 2 +- search-index.json| 2 +- zh-CN/docs/2.0/search-index.json | 2 +- zh-CN/docs/3.0/search-index.json | 2 +- zh-CN/docs/dev/search-index.json | 2 +- zh-CN/search-index.json | 2 +- 8 files changed, 8 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) fix hint existing using problems [doris]
LiBinfeng-01 commented on PR #49987: URL: https://github.com/apache/doris/pull/49987#issuecomment-2800020322 run cloud_p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](mtmv) Create a materialized view that only includes the partitions in the query conditions. [doris]
KeeProMise commented on PR #49852: URL: https://github.com/apache/doris/pull/49852#issuecomment-2800033789 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch asf-site updated (30d9a5c11ce -> 516b41fed36)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/doris-website.git discard 30d9a5c11ce Automated deployment with doris branch @ c48e670323787beecea9f8b93ef1ea5009ebd600 new 516b41fed36 Automated deployment with doris branch @ c48e670323787beecea9f8b93ef1ea5009ebd600 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (30d9a5c11ce) \ N -- N -- N refs/heads/asf-site (516b41fed36) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/2.0/search-index.json | 2 +- docs/3.0/search-index.json | 2 +- docs/dev/search-index.json | 2 +- search-index.json| 2 +- zh-CN/docs/2.0/search-index.json | 2 +- zh-CN/docs/3.0/search-index.json | 2 +- zh-CN/docs/dev/search-index.json | 2 +- zh-CN/search-index.json | 2 +- 8 files changed, 8 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](mtmv) Create a materialized view that only includes the partitions in the query conditions. [doris]
doris-robot commented on PR #49852: URL: https://github.com/apache/doris/pull/49852#issuecomment-2800047805 TPC-H: Total hot run time: 33895 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 84072d9a40ca9b0b2fca760170bc71603ef51b07, data reload: false -- Round 1 -- q1 26176 504049064906 q2 2061264 183 183 q3 10418 1206684 684 q4 10221 990 538 538 q5 7526235825162358 q6 183 164 128 128 q7 902 724 606 606 q8 9311123210381038 q9 6978503151565031 q10 6830230518581858 q11 483 275 272 272 q12 339 344 223 223 q13 17776 363030743074 q14 244 232 211 211 q15 539 483 479 479 q16 638 622 581 581 q17 574 845 350 350 q18 7462721673267216 q19 1239950 570 570 q20 335 331 222 222 q21 4478348423992399 q22 10161028968 968 Total cold run time: 115729 ms Total hot run time: 33895 ms - Round 2, with runtime_filter_mode=off - q1 5086508250815081 q2 238 327 235 235 q3 2148260523042304 q4 1409180114261426 q5 4407436143974361 q6 214 169 126 126 q7 2012189217851785 q8 2576249625732496 q9 7294717171577157 q10 2964310927392739 q11 579 511 485 485 q12 678 787 616 616 q13 3463383633243324 q14 296 306 262 262 q15 514 466 479 466 q16 665 668 663 663 q17 1169150814331433 q18 7706753373697369 q19 797 828 991 828 q20 1912195718461846 q21 5230483546914691 q22 1080108010231023 Total cold run time: 52437 ms Total hot run time: 50716 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] impl external be [doris]
HappenLee commented on code in PR #49944:
URL: https://github.com/apache/doris/pull/49944#discussion_r2041174660
##
be/src/vec/exec/scan/vfile_scanner.cpp:
##
@@ -1156,41 +1087,216 @@ Status VFileScanner::_get_next_reader() {
return Status::InternalError("failed to init reader, err: {}",
init_status.to_string());
}
-_name_to_col_type.clear();
-_missing_cols.clear();
-RETURN_IF_ERROR(_cur_reader->get_columns(&_name_to_col_type,
&_missing_cols));
_cur_reader->set_push_down_agg_type(_get_push_down_agg_type());
-RETURN_IF_ERROR(_generate_missing_columns());
-RETURN_IF_ERROR(_cur_reader->set_fill_columns(_partition_col_descs,
_missing_col_descs));
-if (VLOG_NOTICE_IS_ON && !_missing_cols.empty() && _is_load) {
-fmt::memory_buffer col_buf;
-for (auto& col : _missing_cols) {
-fmt::format_to(col_buf, " {}", col);
-}
-VLOG_NOTICE << fmt::format("Unknown columns:{} in file {}",
fmt::to_string(col_buf),
- range.path);
-}
-
-_source_file_col_names.clear();
-_source_file_col_types.clear();
-_source_file_col_name_types.clear();
-if (_state->query_options().truncate_char_or_varchar_columns &&
need_to_get_parsed_schema) {
-Status status =
_cur_reader->get_parsed_schema(&_source_file_col_names,
-
&_source_file_col_types);
-if (!status.ok() && status.code() !=
TStatusCode::NOT_IMPLEMENTED_ERROR) {
-return status;
-}
-DCHECK(_source_file_col_names.size() ==
_source_file_col_types.size());
-for (int i = 0; i < _source_file_col_names.size(); ++i) {
-_source_file_col_name_types[_source_file_col_names[i]] =
&_source_file_col_types[i];
-}
-}
+
RETURN_IF_ERROR(_set_fill_or_truncate_columns(need_to_get_parsed_schema));
_cur_reader_eof = false;
break;
}
return Status::OK();
}
+Status VFileScanner::_init_parquet_reader(std::unique_ptr&&
parquet_reader) {
+const TFileRangeDesc& range = _current_range;
+Status init_status = Status::OK();
+
+if (range.__isset.table_format_params &&
+range.table_format_params.table_format_type == "iceberg") {
+std::unique_ptr iceberg_reader =
+IcebergParquetReader::create_unique(std::move(parquet_reader),
_profile,
+_state, *_params, range,
_kv_cache,
+_io_ctx.get());
+init_status = iceberg_reader->init_reader(
+_file_col_names, _col_id_name_map, _colname_to_value_range,
+_push_down_conjuncts, _real_tuple_desc,
_default_val_row_desc.get(),
+_col_name_to_slot_id, &_not_single_slot_filter_conjuncts,
+&_slot_id_to_filter_conjuncts);
+_cur_reader = std::move(iceberg_reader);
+} else if (range.__isset.table_format_params &&
+ range.table_format_params.table_format_type == "paimon") {
+std::vector place_holder;
+init_status = parquet_reader->init_reader(
+_file_col_names, place_holder, _colname_to_value_range,
+_push_down_conjuncts, _real_tuple_desc,
_default_val_row_desc.get(),
+_col_name_to_slot_id, &_not_single_slot_filter_conjuncts,
+&_slot_id_to_filter_conjuncts);
+std::unique_ptr paimon_reader =
+PaimonParquetReader::create_unique(std::move(parquet_reader),
_profile,
+ _state, *_params, range,
_io_ctx.get());
+RETURN_IF_ERROR(paimon_reader->init_row_filters());
+_cur_reader = std::move(paimon_reader);
+} else {
+bool hive_parquet_use_column_names = true;
+
+if (range.__isset.table_format_params &&
+range.table_format_params.table_format_type == "hive" && _state !=
nullptr)
+[[likely]] {
+hive_parquet_use_column_names =
+_state->query_options().hive_parquet_use_column_names;
+}
+
+std::vector place_holder;
+init_status = parquet_reader->init_reader(
+_file_col_names, place_holder, _colname_to_value_range,
+_push_down_conjuncts, _real_tuple_desc,
_default_val_row_desc.get(),
+_col_name_to_slot_id, &_not_single_slot_filter_conjuncts,
+&_slot_id_to_filter_conjuncts, true,
hive_parquet_use_column_names);
+_cur_reader = std::move(parquet_reader);
+}
+return init_status;
+}
+
+
+Status VFileScanner::_init_orc_reader(std::unique_ptr&& orc_reader)
{
+const TFileRangeDesc& range = _current_range;
+Status init_status = Status::OK();
+
+if (range.__i
Re: [PR] [enhance](mtmv) Create a materialized view that only includes the partitions in the query conditions. [doris]
doris-robot commented on PR #49852: URL: https://github.com/apache/doris/pull/49852#issuecomment-2800052022 TPC-DS: Total hot run time: 193362 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 84072d9a40ca9b0b2fca760170bc71603ef51b07, data reload: false query1 1410109510611061 query2 6235186818911868 query3 11002 453244574457 query4 52155 24597 23527 23527 query5 5131560 445 445 query6 349 214 203 203 query7 4934500 280 280 query8 317 239 250 239 query9 5585262625872587 query10 424 328 263 263 query11 15123 14971 14865 14865 query12 163 111 106 106 query13 1057496 385 385 query14 9986621564516215 query15 207 196 176 176 query16 7081631 515 515 query17 1082716 584 584 query18 1506385 302 302 query19 192 190 161 161 query20 127 129 124 124 query21 209 133 104 104 query22 4535443844424438 query23 34169 33545 33604 33545 query24 6563245524252425 query25 466 455 411 411 query26 668 271 157 157 query27 2268506 335 335 query28 3143246224392439 query29 587 558 448 448 query30 284 220 191 191 query31 863 880 779 779 query32 74 66 65 65 query33 442 355 308 308 query34 759 850 532 532 query35 801 840 777 777 query36 933 1010916 916 query37 124 95 77 77 query38 4116426543284265 query39 1520146414221422 query40 216 127 106 106 query41 55 55 54 54 query42 126 109 115 109 query43 497 531 499 499 query44 1334828 827 827 query45 179 179 167 167 query46 829 1012640 640 query47 1841189218001800 query48 387 419 314 314 query49 712 504 426 426 query50 647 692 417 417 query51 4252429142664266 query52 139 103 101 101 query53 232 263 183 183 query54 602 575 514 514 query55 86 85 81 81 query56 340 301 291 291 query57 1169116411171117 query58 260 262 280 262 query59 2680278126652665 query60 328 333 328 328 query61 148 122 125 122 query62 748 735 665 665 query63 225 201 200 200 query64 15791145814 814 query65 4397422842414228 query66 746 416 320 320 query67 15874 15437 15273 15273 query68 7256873 519 519 query69 532 333 287 287 query70 1205107511011075 query71 481 362 288 288 query72 5801474750274747 query73 1317649 348 348 query74 8909930388548854 query75 3705318826682668 query76 41031173753 753 query77 662 469 276 276 query78 10061 10254 91889188 query79 2564810 564 564 query80 627 504 444 444 query81 487 254 220 220 query82 436 134 101 101 query83 310 248 233 233 query84 291 102 85 85 query85 782 407 310 310 query86 361 305 299 299 query87 4517450543674367 query88 343722202220 query89 406 301 283 283 query90 1830215 210 210 query91 144 141 113 113 query92 73 58 57 57 query93 1879896 572 572 query94 721 418 304 304 query95 363 294 281 281 query96 475 564 274 274 query97 3130322831433143 query98 232 206 199 199 query99 1468140412941294 Total cold run time: 295783 ms Total hot run time: 193362 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] impl external be [doris]
HappenLee commented on code in PR #49944:
URL: https://github.com/apache/doris/pull/49944#discussion_r2041162318
##
be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:
##
@@ -751,6 +755,38 @@ Status RowGroupReader::_read_empty_batch(size_t
batch_size, size_t* read_rows, b
return Status::OK();
}
+
+Status RowGroupReader::_get_current_batch_row_id(size_t read_rows,
std::vector& row_ids) {
+row_ids.resize(read_rows);
+int64_t idx = 0;
+int64_t read_range_rows = 0;
+for (auto& range : _read_ranges) {
+if (idx > read_rows) {
+break;
+}
+if (read_range_rows >= _total_read_rows) {
+for (auto x = range.first_row; x < range.last_row; x ++) {
+row_ids[idx++] = (rowid_t) (x +
_current_row_group_idx.first_row);
+}
+}
+read_range_rows += range.last_row - range.first_row;
+}
+return Status::OK();
+}
+
+Status RowGroupReader::_fill_row_id_columns(Block* block,size_t read_rows) {
+if (_row_id_column_iterator_pair.first != nullptr) {
+vector row_ids;
Review Comment:
the function could be call multi thread? if not, `row_ids` better be a
membership of `vparquet_group_reader`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](mtmv) Create a materialized view that only includes the partitions in the query conditions. [doris]
doris-robot commented on PR #49852: URL: https://github.com/apache/doris/pull/49852#issuecomment-2800054064 ClickBench: Total hot run time: 30.83 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 84072d9a40ca9b0b2fca760170bc71603ef51b07, data reload: false query1 0.040.040.03 query2 0.140.100.12 query3 0.260.190.20 query4 1.590.210.19 query5 0.600.580.59 query6 1.210.700.72 query7 0.020.010.02 query8 0.040.030.03 query9 0.570.510.51 query10 0.560.560.55 query11 0.160.110.11 query12 0.140.120.11 query13 0.610.590.60 query14 2.682.702.85 query15 0.900.860.86 query16 0.380.380.40 query17 1.021.030.99 query18 0.210.190.19 query19 1.871.861.83 query20 0.010.010.02 query21 15.34 0.860.54 query22 0.761.240.65 query23 14.92 1.370.64 query24 6.612.730.69 query25 0.480.210.10 query26 0.660.150.14 query27 0.050.050.06 query28 10.37 0.860.41 query29 12.55 3.893.26 query30 0.250.090.07 query31 2.820.580.38 query32 3.220.540.46 query33 3.023.033.12 query34 15.73 5.064.48 query35 4.484.544.48 query36 0.660.500.49 query37 0.090.060.06 query38 0.050.040.03 query39 0.030.020.02 query40 0.160.130.12 query41 0.090.030.02 query42 0.030.020.02 query43 0.040.030.03 Total cold run time: 105.42 s Total hot run time: 30.83 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch asf-site updated (516b41fed36 -> d3f8057c7a5)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/doris-website.git discard 516b41fed36 Automated deployment with doris branch @ c48e670323787beecea9f8b93ef1ea5009ebd600 new d3f8057c7a5 Automated deployment with doris branch @ c48e670323787beecea9f8b93ef1ea5009ebd600 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (516b41fed36) \ N -- N -- N refs/heads/asf-site (d3f8057c7a5) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/2.0/search-index.json | 2 +- docs/3.0/search-index.json | 2 +- docs/dev/search-index.json | 2 +- search-index.json| 2 +- zh-CN/docs/2.0/search-index.json | 2 +- zh-CN/docs/3.0/search-index.json | 2 +- zh-CN/docs/dev/search-index.json | 2 +- zh-CN/search-index.json | 2 +- 8 files changed, 8 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [fix](parquet)fix when hive_parquet_use_column_names=false && read partition tb cause be core. #49966 [doris]
dataroaring closed pull request #50007: branch-3.0: [fix](parquet)fix when hive_parquet_use_column_names=false && read partition tb cause be core. #49966 URL: https://github.com/apache/doris/pull/50007 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](inverted index) Downgrade ICU version to support lower OS versions #47664 [doris]
github-actions[bot] commented on PR #50003: URL: https://github.com/apache/doris/pull/50003#issuecomment-2800277473 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](replica) skip missing version should care catchup [doris]
yujun777 commented on code in PR #4: URL: https://github.com/apache/doris/pull/4#discussion_r2041290940 ## fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java: ## @@ -314,12 +314,7 @@ public List getQueryableReplicas(long visibleVersion, Map
[PR] branch-3.0: [fix](parquet)fix when hive_parquet_use_column_names=false && read partition tb cause be core. #49966 [doris]
github-actions[bot] opened a new pull request, #50007: URL: https://github.com/apache/doris/pull/50007 Cherry-picked from #49966 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](replica) skip missing version should care catchup [doris]
yujun777 commented on code in PR #4:
URL: https://github.com/apache/doris/pull/4#discussion_r2041293626
##
fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java:
##
@@ -314,12 +314,7 @@ public List getQueryableReplicas(long
visibleVersion, Map 0 && !allowFailedVersion) {
-continue;
-}
-
-if (!replica.checkVersionCatchUp(visibleVersion, false)) {
Review Comment:
if be not found rowset [1, visible version], be maybe also throw a error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [Fix](TrinoConnector) it will return error when query `information_schema.tables` within `trino-connector` catalog #49912 [doris]
doris-robot commented on PR #50006: URL: https://github.com/apache/doris/pull/50006#issuecomment-2800280811 TPC-H: Total hot run time: 40055 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 1659cec792392c848efdb3f30812e315566bf115, data reload: false -- Round 1 -- q1 17598 670365716571 q2 2057179 168 168 q3 10615 106111481061 q4 10565 734 704 704 q5 7732280128172801 q6 219 137 133 133 q7 964 604 619 604 q8 9366194719901947 q9 6620637563976375 q10 7004224922962249 q11 462 262 268 262 q12 398 212 207 207 q13 17871 298030862980 q14 243 214 213 213 q15 513 462 477 462 q16 674 593 604 593 q17 977 624 567 567 q18 7443693867586758 q19 1393104411381044 q20 470 210 202 202 q21 4031322431633163 q22 1125991 992 991 Total cold run time: 108340 ms Total hot run time: 40055 ms - Round 2, with runtime_filter_mode=off - q1 6581652665566526 q2 329 229 231 229 q3 2829275428882754 q4 2033184017831783 q5 5727575856975697 q6 215 132 132 132 q7 2173183017791779 q8 3340354235433542 q9 8753893288308830 q10 3549351435063506 q11 583 487 494 487 q12 784 602 586 586 q13 9434322031553155 q14 307 267 262 262 q15 508 475 459 459 q16 704 644 663 644 q17 1834163316041604 q18 8236786176797679 q19 1650158615671567 q20 2102190518761876 q21 5437533054255330 q22 1129105410091009 Total cold run time: 68237 ms Total hot run time: 59436 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refine](SetOperator) refine some SetOperator code. [doris]
Mryange commented on PR #49772: URL: https://github.com/apache/doris/pull/49772#issuecomment-2800281116 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [test](ut) add cases about hash join [doris]
github-actions[bot] commented on PR #49803: URL: https://github.com/apache/doris/pull/49803#issuecomment-2800287900 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [test](ut) add cases about hash join [doris]
github-actions[bot] commented on PR #49803: URL: https://github.com/apache/doris/pull/49803#issuecomment-2800287936 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](mtmv)fix when compatible fail, will throw NPE [doris]
zddr commented on PR #49875: URL: https://github.com/apache/doris/pull/49875#issuecomment-2800289974 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](executor) Add the time-sharing executor framework and use it in the scanner section. [doris]
kaka11chen commented on code in PR #49671:
URL: https://github.com/apache/doris/pull/49671#discussion_r2041300767
##
be/src/vec/exec/executor/time_sharing/time_sharing_task_executor.h:
##
@@ -0,0 +1,183 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "bvar/latency_recorder.h"
+#include "util/threadpool.h"
+#include "vec/exec/executor/listenable_future.h"
+#include "vec/exec/executor/task_executor.h"
+#include "vec/exec/executor/ticker.h"
+#include "vec/exec/executor/time_sharing/multilevel_split_queue.h"
+#include "vec/exec/executor/time_sharing/prioritized_split_runner.h"
+
+namespace doris {
+namespace vectorized {
+
+/**
+ * ThreadSafe
+ */
+class TimeSharingTaskExecutor : public TaskExecutor {
+ENABLE_FACTORY_CREATOR(TimeSharingTaskExecutor);
+
+public:
+struct ThreadConfig {
+std::string thread_name;
+int max_thread_num;
+int min_thread_num;
+int max_queue_size = 0;
+std::weak_ptr cgroup_cpu_ctl;
+};
+
+TimeSharingTaskExecutor(ThreadConfig config, int min_concurrency,
+int guaranteed_concurrency_per_task, int
max_concurrency_per_task,
+std::shared_ptr ticker,
+std::chrono::milliseconds
stuck_split_warning_threshold =
+std::chrono::milliseconds(6),
+std::shared_ptr split_queue
= nullptr);
+
+~TimeSharingTaskExecutor() override;
+
+Status init() override;
+
+Status start() override;
+void stop() override;
+
+Result> create_task(
+const TaskId& task_id, std::function
utilization_supplier,
+int initial_split_concurrency,
+std::chrono::nanoseconds split_concurrency_adjust_frequency,
+std::optional max_concurrency_per_task) override;
+
+Status add_task(const TaskId& task_id, std::shared_ptr
task_handle) override;
+
+Status remove_task(std::shared_ptr task_handle) override;
+
+Result>> enqueue_splits(
+std::shared_ptr task_handle, bool intermediate,
+const std::vector>& splits) override;
+
+size_t waiting_splits_size() const { return _waiting_splits->size(); }
+
+size_t intermediate_splits_size() const {
+std::lock_guard guard(_mutex);
+return _intermediate_splits.size();
+}
+
+size_t running_splits_size() const { return _running_splits.size(); }
+
+size_t blocked_splits_size() const { return _blocked_splits.size(); }
+
+size_t total_splits_size() const {
+std::lock_guard guard(_mutex);
+return _all_splits.size();
+}
+
+size_t tasks_size() const {
+std::lock_guard guard(_mutex);
+return _tasks.size();
+}
+
+int64_t completed_tasks_level0() const { return
_completed_tasks_per_level[0]; }
+
+int64_t completed_tasks_level1() const { return
_completed_tasks_per_level[1]; }
+
+int64_t completed_tasks_level2() const { return
_completed_tasks_per_level[2]; }
+
+int64_t completed_tasks_level3() const { return
_completed_tasks_per_level[3]; }
+
+int64_t completed_tasks_level4() const { return
_completed_tasks_per_level[4]; }
+
+int64_t completed_splits_level0() const { return
_completed_splits_per_level[0]; }
+
+int64_t completed_splits_level1() const { return
_completed_splits_per_level[1]; }
+
+int64_t completed_splits_level2() const { return
_completed_splits_per_level[2]; }
+
+int64_t completed_splits_level3() const { return
_completed_splits_per_level[3]; }
+
+int64_t completed_splits_level4() const { return
_completed_splits_per_level[4]; }
+
+int64_t running_tasks_level0() const { return
_get_running_tasks_for_level(0); }
+
+int64_t running_tasks_level1() const { return
_get_running_tasks_for_level(1); }
+
+int64_t running_tasks_level2() const { return
_get_running_tasks_for_level(2); }
+
+int64_t running_tasks_level3() const { return
_get_running_tasks_for_level(3); }
+
+int64_t running_tasks_level4() const { return
_
Re: [PR] branch-3.0: [fix](parquet)fix when hive_parquet_use_column_names=false && read partition tb cause be core. #49966 [doris]
doris-robot commented on PR #50007: URL: https://github.com/apache/doris/pull/50007#issuecomment-2800292082 TPC-H: Total hot run time: 39911 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 336c71a9988f85050563383abcf7d3120b9f241e, data reload: false -- Round 1 -- q1 17565 669865876587 q2 2046169 189 169 q3 10923 108111781081 q4 10511 723 750 723 q5 7767283828672838 q6 228 133 133 133 q7 973 618 607 607 q8 9351191819871918 q9 6587638465076384 q10 6995226023242260 q11 468 265 266 265 q12 393 206 211 206 q13 17787 296929572957 q14 246 208 210 208 q15 512 462 462 462 q16 683 594 592 592 q17 965 540 599 540 q18 7192661967096619 q19 1398110710121012 q20 496 206 200 200 q21 4037317731553155 q22 1120995 1001995 Total cold run time: 108243 ms Total hot run time: 39911 ms - Round 2, with runtime_filter_mode=off - q1 6625657765786577 q2 336 242 231 231 q3 2859275128992751 q4 2005176618251766 q5 5808575957565756 q6 212 122 127 122 q7 2212186018541854 q8 3350358035003500 q9 8868882588848825 q10 3535349835033498 q11 598 493 480 480 q12 806 626 601 601 q13 9643320432123204 q14 307 263 268 263 q15 511 461 468 461 q16 692 659 666 659 q17 1849163216301630 q18 8255788477127712 q19 1646154014501450 q20 2081184518801845 q21 5478550852465246 q22 1129102810451028 Total cold run time: 68805 ms Total hot run time: 59459 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](executor) Add the time-sharing executor framework and use it in the scanner section. [doris]
kaka11chen commented on code in PR #49671:
URL: https://github.com/apache/doris/pull/49671#discussion_r2041300767
##
be/src/vec/exec/executor/time_sharing/time_sharing_task_executor.h:
##
@@ -0,0 +1,183 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "bvar/latency_recorder.h"
+#include "util/threadpool.h"
+#include "vec/exec/executor/listenable_future.h"
+#include "vec/exec/executor/task_executor.h"
+#include "vec/exec/executor/ticker.h"
+#include "vec/exec/executor/time_sharing/multilevel_split_queue.h"
+#include "vec/exec/executor/time_sharing/prioritized_split_runner.h"
+
+namespace doris {
+namespace vectorized {
+
+/**
+ * ThreadSafe
+ */
+class TimeSharingTaskExecutor : public TaskExecutor {
+ENABLE_FACTORY_CREATOR(TimeSharingTaskExecutor);
+
+public:
+struct ThreadConfig {
+std::string thread_name;
+int max_thread_num;
+int min_thread_num;
+int max_queue_size = 0;
+std::weak_ptr cgroup_cpu_ctl;
+};
+
+TimeSharingTaskExecutor(ThreadConfig config, int min_concurrency,
+int guaranteed_concurrency_per_task, int
max_concurrency_per_task,
+std::shared_ptr ticker,
+std::chrono::milliseconds
stuck_split_warning_threshold =
+std::chrono::milliseconds(6),
+std::shared_ptr split_queue
= nullptr);
+
+~TimeSharingTaskExecutor() override;
+
+Status init() override;
+
+Status start() override;
+void stop() override;
+
+Result> create_task(
+const TaskId& task_id, std::function
utilization_supplier,
+int initial_split_concurrency,
+std::chrono::nanoseconds split_concurrency_adjust_frequency,
+std::optional max_concurrency_per_task) override;
+
+Status add_task(const TaskId& task_id, std::shared_ptr
task_handle) override;
+
+Status remove_task(std::shared_ptr task_handle) override;
+
+Result>> enqueue_splits(
+std::shared_ptr task_handle, bool intermediate,
+const std::vector>& splits) override;
+
+size_t waiting_splits_size() const { return _waiting_splits->size(); }
+
+size_t intermediate_splits_size() const {
+std::lock_guard guard(_mutex);
+return _intermediate_splits.size();
+}
+
+size_t running_splits_size() const { return _running_splits.size(); }
+
+size_t blocked_splits_size() const { return _blocked_splits.size(); }
+
+size_t total_splits_size() const {
+std::lock_guard guard(_mutex);
+return _all_splits.size();
+}
+
+size_t tasks_size() const {
+std::lock_guard guard(_mutex);
+return _tasks.size();
+}
+
+int64_t completed_tasks_level0() const { return
_completed_tasks_per_level[0]; }
+
+int64_t completed_tasks_level1() const { return
_completed_tasks_per_level[1]; }
+
+int64_t completed_tasks_level2() const { return
_completed_tasks_per_level[2]; }
+
+int64_t completed_tasks_level3() const { return
_completed_tasks_per_level[3]; }
+
+int64_t completed_tasks_level4() const { return
_completed_tasks_per_level[4]; }
+
+int64_t completed_splits_level0() const { return
_completed_splits_per_level[0]; }
+
+int64_t completed_splits_level1() const { return
_completed_splits_per_level[1]; }
+
+int64_t completed_splits_level2() const { return
_completed_splits_per_level[2]; }
+
+int64_t completed_splits_level3() const { return
_completed_splits_per_level[3]; }
+
+int64_t completed_splits_level4() const { return
_completed_splits_per_level[4]; }
+
+int64_t running_tasks_level0() const { return
_get_running_tasks_for_level(0); }
+
+int64_t running_tasks_level1() const { return
_get_running_tasks_for_level(1); }
+
+int64_t running_tasks_level2() const { return
_get_running_tasks_for_level(2); }
+
+int64_t running_tasks_level3() const { return
_get_running_tasks_for_level(3); }
+
+int64_t running_tasks_level4() const { return
_
(doris-website) branch asf-site updated (e0493d61b52 -> 2ddc0384546)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/doris-website.git discard e0493d61b52 Automated deployment with doris branch @ c48e670323787beecea9f8b93ef1ea5009ebd600 new 2ddc0384546 Automated deployment with doris branch @ c48e670323787beecea9f8b93ef1ea5009ebd600 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (e0493d61b52) \ N -- N -- N refs/heads/asf-site (2ddc0384546) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/2.0/search-index.json | 2 +- docs/3.0/search-index.json | 2 +- docs/dev/search-index.json | 2 +- search-index.json| 2 +- zh-CN/docs/2.0/search-index.json | 2 +- zh-CN/docs/3.0/search-index.json | 2 +- zh-CN/docs/dev/search-index.json | 2 +- zh-CN/search-index.json | 2 +- 8 files changed, 8 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [test](ut) add cases about hash join [doris]
BiteThet commented on code in PR #49803:
URL: https://github.com/apache/doris/pull/49803#discussion_r2041297258
##
be/src/pipeline/exec/hashjoin_probe_operator.h:
##
@@ -155,7 +155,8 @@ class HashJoinProbeOperatorX MOCK_REMOVE(final)
bool need_finalize_variant_column() const { return
_need_finalize_variant_column; }
bool is_lazy_materialized_column(int column_id) const {
-return _have_other_join_conjunct &&
!_other_conjunct_refer_column_ids.contains(column_id);
+return _have_other_join_conjunct &&
Review Comment:
what if _have_other_join_conjunct is false and mark join is true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](inverted_index) add ik analyzer for chinese language [doris]
Ryan19929 commented on code in PR #49519:
URL: https://github.com/apache/doris/pull/49519#discussion_r2041302169
##
be/CMakeLists.txt:
##
@@ -239,6 +239,11 @@ install(DIRECTORY
${BASE_DIR}/dict/icu/uax29
DESTINATION ${OUTPUT_DIR}/dict/icu)
+
+install(DIRECTORY
+${BASE_DIR}/dict/ik
+DESTINATION ${OUTPUT_DIR}/dict)
Review Comment:
> DESTINATION ${OUTPUT_DIR}/dict/ik
All dict files are in the `${BASE_DIR}/dict/ik`
directory(`${BASE_DIR}/dict/ik/*.dict`). Setting `DESTINATION to
${OUTPUT_DIR}/dict/ik` would result in the path /dict/ik/ik/*.dict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [Fix](TrinoConnector) it will return error when query `information_schema.tables` within `trino-connector` catalog #49912 [doris]
doris-robot commented on PR #50006: URL: https://github.com/apache/doris/pull/50006#issuecomment-2800298426 ClickBench: Total hot run time: 32.38 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 1659cec792392c848efdb3f30812e315566bf115, data reload: false query1 0.040.040.03 query2 0.070.030.03 query3 0.230.070.06 query4 1.640.100.10 query5 0.520.530.48 query6 1.170.720.73 query7 0.020.020.02 query8 0.040.020.03 query9 0.560.520.50 query10 0.560.560.56 query11 0.160.110.13 query12 0.140.110.11 query13 0.610.590.59 query14 2.822.712.72 query15 0.900.820.83 query16 0.380.380.40 query17 0.981.081.06 query18 0.240.220.21 query19 2.031.901.97 query20 0.010.010.02 query21 15.35 0.610.59 query22 2.932.312.55 query23 16.96 1.000.76 query24 3.320.731.80 query25 0.310.070.21 query26 0.370.140.14 query27 0.040.040.05 query28 9.630.520.47 query29 12.57 3.253.22 query30 0.250.060.05 query31 2.870.390.37 query32 3.260.470.46 query33 2.953.083.02 query34 16.94 4.524.47 query35 4.514.534.49 query36 0.660.470.48 query37 0.080.060.06 query38 0.050.040.03 query39 0.030.020.02 query40 0.160.130.12 query41 0.090.020.03 query42 0.030.020.02 query43 0.040.040.03 Total cold run time: 106.52 s Total hot run time: 32.38 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](stream loader) do not redirect to https for stream load [doris]
hello-stephen commented on PR #50009: URL: https://github.com/apache/doris/pull/50009#issuecomment-2800301346 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](stream loader) do not redirect to https for stream load [doris]
englefly opened a new pull request, #50009: URL: https://github.com/apache/doris/pull/50009 ### What problem does this PR solve? auditlog stream load plugin does not work when https is enabled. The reason is when https is enabled, LoadAction will add an extra redirection(from http to https), which breaks the process of audit plugin. Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-2.1: [fix](parquet)fix when hive_parquet_use_column_names=false && read partition tb cause be core. #49966 [doris]
doris-robot commented on PR #50008: URL: https://github.com/apache/doris/pull/50008#issuecomment-2800293319 # BE UT Coverage Report Increment line coverage `50.00% (1/2)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/c5f044af5d85d2a4b318d5f5837aa9ff6ce406b6_c5f044af5d85d2a4b318d5f5837aa9ff6ce406b6/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/c5f044af5d85d2a4b318d5f5837aa9ff6ce406b6_c5f044af5d85d2a4b318d5f5837aa9ff6ce406b6/report/index.html) | Category | Coverage | |---|| | Function Coverage | 38.49% (10146/26357) | | Line Coverage | 29.48% (83730/283979) | | Region Coverage | 28.16% (43100/153077) | | Branch Coverage | 24.76% (21857/88268) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [fix](parquet)fix when hive_parquet_use_column_names=false && read partition tb cause be core. #49966 [doris]
doris-robot commented on PR #50007: URL: https://github.com/apache/doris/pull/50007#issuecomment-2800305854 TPC-DS: Total hot run time: 197435 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 336c71a9988f85050563383abcf7d3120b9f241e, data reload: false query1 1270911 925 911 query2 6260210920512051 query3 10818 413743254137 query4 60915 29277 23607 23607 query5 5164473 468 468 query6 392 185 196 185 query7 5466323 317 317 query8 314 230 224 224 query9 8527261625952595 query10 474 263 252 252 query11 17899 15122 15653 15122 query12 174 102 110 102 query13 1411440 436 436 query14 10768 781975787578 query15 205 174 185 174 query16 7083458 487 458 query17 1200586 602 586 query18 1773348 329 329 query19 222 166 171 166 query20 118 119 111 111 query21 206 103 105 103 query22 4592424047764240 query23 34461 34248 33919 33919 query24 6134288129202881 query25 532 442 437 437 query26 657 181 181 181 query27 1883364 369 364 query28 4058249024172417 query29 708 469 486 469 query30 243 167 164 164 query31 981 826 876 826 query32 68 61 58 58 query33 447 314 312 312 query34 900 524 515 515 query35 865 759 732 732 query36 1079968 956 956 query37 122 71 67 67 query38 4111408139973997 query39 1548166514741474 query40 202 99 99 99 query41 48 47 50 47 query42 109 101 104 101 query43 546 496 497 496 query44 1171814 833 814 query45 186 167 168 167 query46 1137714 724 714 query47 2002189319061893 query48 485 397 380 380 query49 727 404 393 393 query50 867 430 431 430 query51 7431726372627262 query52 111 91 88 88 query53 278 182 184 182 query54 581 477 477 477 query55 85 84 81 81 query56 269 260 256 256 query57 1245117211491149 query58 226 220 211 211 query59 3458294930982949 query60 283 252 255 252 query61 129 123 111 111 query62 762 675 670 670 query63 216 185 190 185 query64 1388687 651 651 query65 3252318831963188 query66 714 294 292 292 query67 15711 15571 15673 15571 query68 4235585 571 571 query69 435 263 263 263 query70 1155113111151115 query71 340 261 250 250 query72 6341404940354035 query73 759 342 364 342 query74 10275 906989488948 query75 3337264827052648 query76 2068986 1117986 query77 489 274 277 274 query78 10499 967695019501 query79 2027590 588 588 query80 1027432 430 430 query81 530 239 236 236 query82 185 88 87 87 query83 160 141 157 141 query84 289 77 73 73 query85 959 332 287 287 query86 403 310 280 280 query87 4448427142934271 query88 4494239023582358 query89 410 291 292 291 query90 1946185 187 185 query91 180 149 149 149 query92 56 51 51 51 query93 2359561 575 561 query94 797 292 301 292 query95 361 256 261 256 query96 624 291 280 280 query97 3296323331813181 query98 213 200 204 200 query99 1527129912751275 Total cold run time: 313891 ms Total hot run time: 197435 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] branch-3.0: [fix](parquet)fix when hive_parquet_use_column_names=false && read partition tb cause be core. #49966 [doris]
doris-robot commented on PR #50007: URL: https://github.com/apache/doris/pull/50007#issuecomment-2800311438 ClickBench: Total hot run time: 32.62 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 336c71a9988f85050563383abcf7d3120b9f241e, data reload: false query1 0.040.030.02 query2 0.070.030.03 query3 0.240.060.06 query4 1.620.100.10 query5 0.540.520.51 query6 1.140.730.73 query7 0.020.020.02 query8 0.040.030.03 query9 0.560.520.51 query10 0.550.550.55 query11 0.140.100.13 query12 0.150.120.11 query13 0.610.600.59 query14 2.842.832.85 query15 0.890.840.82 query16 0.400.380.38 query17 1.011.061.04 query18 0.240.210.22 query19 1.921.851.99 query20 0.010.010.01 query21 15.37 0.580.57 query22 2.513.361.99 query23 16.85 1.060.78 query24 3.331.101.08 query25 0.290.260.18 query26 0.290.140.13 query27 0.040.050.04 query28 10.08 0.500.49 query29 12.60 3.183.20 query30 0.250.060.06 query31 2.880.380.38 query32 3.250.460.45 query33 2.993.053.03 query34 17.06 4.474.50 query35 4.524.524.54 query36 0.680.470.48 query37 0.090.060.06 query38 0.050.040.03 query39 0.030.020.02 query40 0.160.120.12 query41 0.080.020.02 query42 0.030.020.02 query43 0.030.030.03 Total cold run time: 106.49 s Total hot run time: 32.62 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [refactor](spill) Refine logics in pipeline task [doris]
Gabriel39 opened a new pull request, #50010: URL: https://github.com/apache/doris/pull/50010 ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (68cfef6d67c -> 424d2a5eb66)
This is an automated email from the ASF dual-hosted git repository.
gabriellee pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 68cfef6d67c [fix](parquet)fix when hive_parquet_use_column_names=false
&& read partition tb cause be core. (#49966)
add 424d2a5eb66 [UT](low memory) Test low memory mode in pipeline task
(#49992)
No new revisions were added by this update.
Summary of changes:
be/src/pipeline/pipeline_task.cpp | 2 +-
be/src/runtime/memory/thread_mem_tracker_mgr.h | 5 +-
be/src/runtime/runtime_state.h | 2 +-
be/test/pipeline/pipeline_task_test.cpp| 141 -
be/test/pipeline/thrift_builder.h | 4 +
be/test/testutil/mock/mock_runtime_state.h | 7 +
.../testutil/mock/mock_thread_mem_tracker_mgr.h} | 27 ++--
7 files changed, 161 insertions(+), 27 deletions(-)
copy be/{src/service/brpc_service.h =>
test/testutil/mock/mock_thread_mem_tracker_mgr.h} (67%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids)show user friendly error message when meet unsupported subquery [doris]
github-actions[bot] commented on PR #49319: URL: https://github.com/apache/doris/pull/49319#issuecomment-2800566814 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids)show user friendly error message when meet unsupported subquery [doris]
github-actions[bot] commented on PR #49319: URL: https://github.com/apache/doris/pull/49319#issuecomment-2800566763 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](mtmv) Create a materialized view that only includes the partitions in the query conditions. [doris]
doris-robot commented on PR #49852: URL: https://github.com/apache/doris/pull/49852#issuecomment-2800320697 TPC-H: Total hot run time: 35266 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 84072d9a40ca9b0b2fca760170bc71603ef51b07, data reload: false -- Round 1 -- q1 26391 511451225114 q2 2077289 194 194 q3 10390 1262743 743 q4 10250 1029549 549 q5 7500247423972397 q6 191 168 138 138 q7 915 737 621 621 q8 9312131011281128 q9 6791519551285128 q10 6835230119031903 q11 485 297 276 276 q12 357 361 232 232 q13 17765 370031093109 q14 238 220 220 220 q15 526 499 515 499 q16 623 621 590 590 q17 589 876 366 366 q18 7594726570717071 q19 1206952 583 583 q20 348 321 219 219 q21 4516351732053205 q22 1035990 981 981 Total cold run time: 115934 ms Total hot run time: 35266 ms - Round 2, with runtime_filter_mode=off - q1 5165513551175117 q2 237 337 229 229 q3 2178267923162316 q4 1490198114941494 q5 4582453643474347 q6 210 161 124 124 q7 2035193817491749 q8 2662262225602560 q9 7151718270887088 q10 3016319227752775 q11 584 494 485 485 q12 700 774 605 605 q13 3576401732703270 q14 278 293 259 259 q15 546 491 504 491 q16 641 703 658 658 q17 1179158213811381 q18 7778742575707425 q19 865 852 999 852 q20 2013198518211821 q21 5340469546904690 q22 10861053981 981 Total cold run time: 53312 ms Total hot run time: 50717 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] (nereids) implement dropResource/RowPolicyCommand in nereids [doris]
github-actions[bot] commented on PR #49038: URL: https://github.com/apache/doris/pull/49038#issuecomment-2800570873 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [only test now] [doris]
Mryange closed pull request #49980: [only test now] URL: https://github.com/apache/doris/pull/49980 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] (nereids) implement dropResource/RowPolicyCommand in nereids [doris]
github-actions[bot] commented on PR #49038: URL: https://github.com/apache/doris/pull/49038#issuecomment-2800570934 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) fix initcap of utf-8 charset [doris]
LiBinfeng-01 commented on PR #49988: URL: https://github.com/apache/doris/pull/49988#issuecomment-2800576374 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) fix initcap of utf-8 charset [doris]
LiBinfeng-01 commented on PR #49988: URL: https://github.com/apache/doris/pull/49988#issuecomment-2800576534 run cloud_p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](inverted index) reset memory after inverted index write_v1 for each dir [doris]
xiaokang commented on code in PR #50013:
URL: https://github.com/apache/doris/pull/50013#discussion_r2041475660
##
be/src/olap/rowset/segment_v2/inverted_index_file_writer.cpp:
##
@@ -345,6 +345,8 @@ Status InvertedIndexFileWriter::write_v1() {
FINALLY({
FINALLY_CLOSE(output);
FINALLY_CLOSE(out_dir);
+output = nullptr;
Review Comment:
Is setting ptr to nullptr necessary for other FINALLY_CLOSE(ptr)? If prt
should be set to nullptr after FINALLY_CLOSE, can it be set inside
FINALLY_CLOSE?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] (nereids) enhance like add escape syntax [doris]
feiniaofeiafei commented on code in PR #47993:
URL: https://github.com/apache/doris/pull/47993#discussion_r2041477923
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/LikeToEqualRewrite.java:
##
@@ -69,4 +77,18 @@ private static Expression rewriteLikeToEqual(Like like) {
}
return new EqualTo(left, new VarcharLiteral(sb.toString()));
}
+
+private static Like replaceEscapeCharInLike(Like oldLike, char escapeChar,
char replaceChar) {
+Expression left = oldLike.getLeft();
+Expression right = oldLike.getRight();
+Expression escape = oldLike.getEscape().get();
+
+if (right instanceof VarcharLiteral && escape instanceof
VarcharLiteral) {
+String rightValue = ((VarcharLiteral)
right).value.replace(escapeChar, replaceChar);
+right = new VarcharLiteral(rightValue);
+String escapeValue = ((VarcharLiteral)
escape).value.replace(escapeChar, replaceChar);
+escape = new VarcharLiteral(escapeValue);
+}
+return new Like(left, right, escape);
+}
Review Comment:
LikeToEqualRewrite is an optimize rule which can be skipped, but this
"replaceEscapeCharInLike" replace is required and cannot be skipped. Should we
put replaceEscapeCharInLike in LogicalPlanBuilder or other place?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [only test] [doris]
Mryange commented on PR #50020: URL: https://github.com/apache/doris/pull/50020#issuecomment-2800580746 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [only test] [doris]
Thearas commented on PR #50020: URL: https://github.com/apache/doris/pull/50020#issuecomment-2800580628 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-c108335-hive-sql updated: [tmp] use table to get hive schema
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-c108335-hive-sql
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-c108335-hive-sql by
this push:
new be6f138729f [tmp] use table to get hive schema
be6f138729f is described below
commit be6f138729f58aa9a6baf44c13c846bc85597997
Author: morningman
AuthorDate: Mon Apr 14 14:19:04 2025 +0800
[tmp] use table to get hive schema
---
.../doris/datasource/hive/HMSExternalTable.java| 24 +-
1 file changed, 23 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java
index b17493989fd..efd1c7c762a 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java
@@ -61,6 +61,7 @@ import org.apache.doris.thrift.TTableDescriptor;
import org.apache.doris.thrift.TTableType;
import com.google.common.collect.BiMap;
+import com.google.common.collect.ImmutableList;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
import com.google.common.collect.Sets;
@@ -77,6 +78,7 @@ import
org.apache.hadoop.hive.metastore.api.DoubleColumnStatsData;
import org.apache.hadoop.hive.metastore.api.FieldSchema;
import org.apache.hadoop.hive.metastore.api.LongColumnStatsData;
import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
import org.apache.hadoop.hive.metastore.api.StringColumnStatsData;
import org.apache.hadoop.hive.ql.io.AcidUtils;
import org.apache.hudi.common.table.HoodieTableMetaClient;
@@ -592,7 +594,7 @@ public class HMSExternalTable extends ExternalTable
implements MTMVRelatedTableI
} else if (dlaType.equals(DLAType.HUDI)) {
return getHudiSchema();
} else {
-return getHiveSchema();
+return getHiveSchema2();
}
}
@@ -618,6 +620,26 @@ public class HMSExternalTable extends ExternalTable
implements MTMVRelatedTableI
return Optional.of(hudiSchemaCacheValue);
}
+private static List getSchemaFromTable(Table table) {
+ImmutableList.Builder schema = ImmutableList.builder();
+schema.addAll(table.getSd().getCols());
+schema.addAll(table.getPartitionKeys());
+return schema.build();
+}
+
+private Optional getHiveSchema2() {
+List schema = getSchemaFromTable(this.remoteTable);
+List columns = Lists.newArrayListWithCapacity(schema.size());
+for (FieldSchema field : schema) {
+String fieldName = field.getName().toLowerCase(Locale.ROOT);
+columns.add(new Column(fieldName,
+
HiveMetaStoreClientHelper.hiveTypeToDorisType(field.getType()), true, null,
+true, null, field.getComment(), true, -1));
+}
+List partitionColumns = initPartitionColumns(columns);
+return Optional.of(new HMSSchemaCacheValue(columns, partitionColumns));
+}
+
private Optional getHiveSchema() {
HMSCachedClient client = ((HMSExternalCatalog) catalog).getClient();
List schema = client.getSchema(dbName, name);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
doris.git: Error while running github feature from branch-c108335-hive-sql:.asf.yaml
An error occurred while processing the github feature in .asf.yaml: while parsing a mapping in "doris.git/.asf.yaml::github", line 111, column 1: - Build Broker ^ (line: 111) unexpected key not in schema 'required_pull_request_reviews' in "doris.git/.asf.yaml::github", line 114, column 1: - COMPILE (DORIS_COMPILE) ^ (line: 114) --- With regards, ASF Infra. For further information, please see the .asf.yaml documentation at: https://github.com/apache/infrastructure-asfyaml/blob/main/README.md - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [only test] [doris]
Mryange opened a new pull request, #50020: URL: https://github.com/apache/doris/pull/50020 ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](resource) Fix `S3FileWriter` thread attach task [doris]
doris-robot commented on PR #50017: URL: https://github.com/apache/doris/pull/50017#issuecomment-2800598679 # BE UT Coverage Report Increment line coverage `0.00% (0/3)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/c9da497dcd30f08386d29a52c2969baacd14a834_c9da497dcd30f08386d29a52c2969baacd14a834/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/c9da497dcd30f08386d29a52c2969baacd14a834_c9da497dcd30f08386d29a52c2969baacd14a834/report/index.html) | Category | Coverage | |---|| | Function Coverage | 52.49% (14111/26883) | | Line Coverage | 41.29% (122010/295479) | | Region Coverage | 40.05% (62132/155146) | | Branch Coverage | 34.69% (31103/89662) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](nereids)support show functions [doris]
starocean999 merged PR #49893: URL: https://github.com/apache/doris/pull/49893 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Enhancement](nereids)support show functions (#49893)
This is an automated email from the ASF dual-hosted git repository.
starocean999 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 5d2698924fc [Enhancement](nereids)support show functions (#49893)
5d2698924fc is described below
commit 5d2698924fc0a39db6bf5d18f0d6727d3f9b35ea
Author: lsy3993 <110876560+lsy3...@users.noreply.github.com>
AuthorDate: Mon Apr 14 14:28:34 2025 +0800
[Enhancement](nereids)support show functions (#49893)
---
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 4 +-
.../org/apache/doris/catalog/FunctionRegistry.java | 17 ++
.../doris/nereids/parser/LogicalPlanBuilder.java | 26 +++
.../apache/doris/nereids/trees/plans/PlanType.java | 1 +
.../trees/plans/commands/ShowFunctionsCommand.java | 236 +
.../trees/plans/visitor/CommandVisitor.java| 5 +
.../plans/commands/ShowFunctionsCommandTest.java | 190 +
.../show/test_nereids_show_functions.groovy| 92
8 files changed, 569 insertions(+), 2 deletions(-)
diff --git a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
index bf480cd099c..77a28ea6f8a 100644
--- a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
+++ b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
@@ -285,6 +285,8 @@ supportedShowStatement
| SHOW LAST INSERT
#showLastInsert
| SHOW ((CHAR SET) | CHARSET)
#showCharset
| SHOW DELETE ((FROM | IN) database=multipartIdentifier)?
#showDelete
+| SHOW FULL? BUILTIN? FUNCTIONS
+((FROM | IN) database=multipartIdentifier)? (LIKE STRING_LITERAL)?
#showFunctions
| SHOW ALL? GRANTS
#showGrants
| SHOW GRANTS FOR userIdentify
#showGrantsForUser
| SHOW SYNC JOB ((FROM | IN) database=multipartIdentifier)?
#showSyncJob
@@ -415,8 +417,6 @@ unsupportedShowStatement
wildWhere? sortClause? limitClause?
#showPartitions
| SHOW RESOURCES wildWhere? sortClause? limitClause?
#showResources
| SHOW WORKLOAD GROUPS wildWhere?
#showWorkloadGroups
-| SHOW FULL? BUILTIN? FUNCTIONS
-((FROM | IN) database=multipartIdentifier)? wildWhere?
#showFunctions
| SHOW GLOBAL FULL? FUNCTIONS wildWhere?
#showGlobalFunctions
| SHOW TYPECAST ((FROM | IN) database=multipartIdentifier)?
#showTypeCast
| SHOW (KEY | KEYS | INDEX | INDEXES)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionRegistry.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionRegistry.java
index dc11b5d02be..74fe1e38753 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionRegistry.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionRegistry.java
@@ -74,6 +74,18 @@ public class FunctionRegistry {
afterRegisterBuiltinFunctions(name2BuiltinBuilders);
}
+public Map> getName2BuiltinBuilders() {
+return name2BuiltinBuilders;
+}
+
+public String getGlobalFunctionDbName() {
+return GLOBAL_FUNCTION;
+}
+
+public Map>>
getName2UdfBuilders() {
+return name2UdfBuilders;
+}
+
// this function is used to test.
// for example, you can create child class of FunctionRegistry and clear
builtin functions or add more functions
// in this method
@@ -284,6 +296,11 @@ public class FunctionRegistry {
Map> builders =
name2UdfBuilders.getOrDefault(dbName, ImmutableMap.of());
builders.getOrDefault(name, Lists.newArrayList())
.removeIf(builder -> ((UdfBuilder)
builder).getArgTypes().equals(argTypes));
+
+// the name will be used when show functions, so remove the name
when it's dropped
+if (builders.getOrDefault(name, Lists.newArrayList()).isEmpty()) {
+builders.remove(name);
+}
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
index 8e8944035f7..f722344fa7f 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
@@ -329,6 +329,7 @@ import
org.apache.doris.nereids.DorisParser.ShowDynamicPartitionContext;
import org.apache.dor
[PR] [Improve] add exception throws if record's topic is modified before put [doris-kafka-connector]
chuang-wang-pre opened a new pull request, #68:
URL: https://github.com/apache/doris-kafka-connector/pull/68
Currentlly, if using `io.confluent.connect.transforms.ExtractTopic` or
`org.apache.kafka.connect.transforms.RegexRouter` or other Transformers have
changed the topic of the record, according to the current implementation,
**`doris-kafka-connector` will not successfully submit the consumer offset.**
because in `org.apache.kafka.connect.sink.SinkTask#preCommit`, the topic
corresponds to the original topic before applying any transformations. If the
connector is restarted, it will be consumed repeatedly.
More seriously, by default, **`two_phase_commit` is enabled, data will not
be written to doris** because transactions were not successfully committed in
`org.apache.kafka.connect.sink.SinkTask#preCommit`. But at this point, both the
connector and task are in a normal RUNNING state, and the user is not aware of
this abnormal situation.
Although after version 3.6, Kafka supports originalTopic and can maintain
the mapping between the original topic and offset, lower versions of Kafka do
not support it. So personally, I think it would be better to throw an exception
directly in this situation
For example, for the following configured connector:
> {
> "name":"test_connector",
> "config":{
>
"connector.class":"org.apache.doris.kafka.connector.DorisSinkConnector",
> "topics":"original_topic",
> "tasks.max":"1",
> "doris.topic2table.map":
"original_topic:test_table,transformed_original_topic:test_table",
> "buffer.count.records":"2",
> "buffer.flush.time":"11",
> "buffer.size.bytes":"1000",
> "doris.urls":"127.0.0.1",
> "doris.user":"root",
> "doris.password":"",
> "doris.http.port":"8030",
> "doris.query.port":"9030",
> "doris.database":"transforms_msg",
> "load.model":"stream_load",
> "transforms": "renameTopic",
> "transforms": "AddPrefix",
>"transforms.AddPrefix.type":
"org.apache.kafka.connect.transforms.RegexRouter",
>"transforms.AddPrefix.regex": ".*",
>"transforms.AddPrefix.replacement": "transformed_$0"
> }
> }
errors will occur when consuming dataļ¼
`Unexpected topic: [transformed_original_topic] in SinkRecord. This may be
caused by a Single Message Transform (SMT) modifying the topic. Please check
your connector configuration.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](spill) Refine logics in pipeline task [doris]
github-actions[bot] commented on PR #50010: URL: https://github.com/apache/doris/pull/50010#issuecomment-2800591010 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](spill) Refine logics in pipeline task [doris]
github-actions[bot] commented on PR #50010: URL: https://github.com/apache/doris/pull/50010#issuecomment-2800591101 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [fix](iceberg) Fix the failure of creating a table in OBS #49983 [doris]
doris-robot commented on PR #50018: URL: https://github.com/apache/doris/pull/50018#issuecomment-2800607255 ClickBench: Total hot run time: 32.06 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit b4fa6bb30c4fc279471f0b588d98ba6c3a845a14, data reload: false query1 0.040.030.04 query2 0.060.030.04 query3 0.230.070.07 query4 1.620.100.10 query5 0.530.510.50 query6 1.140.720.73 query7 0.020.020.01 query8 0.040.030.04 query9 0.570.490.51 query10 0.560.560.57 query11 0.150.100.10 query12 0.140.110.11 query13 0.610.590.60 query14 2.752.742.77 query15 0.900.820.83 query16 0.390.410.38 query17 1.040.971.06 query18 0.240.210.20 query19 1.891.852.00 query20 0.010.030.02 query21 15.38 0.570.60 query22 2.712.462.29 query23 16.81 0.900.83 query24 3.340.970.38 query25 0.240.270.07 query26 0.310.140.14 query27 0.050.040.04 query28 10.84 0.520.50 query29 12.61 3.263.25 query30 0.240.050.06 query31 2.860.400.39 query32 3.230.460.45 query33 2.972.993.01 query34 17.30 4.464.46 query35 4.574.554.55 query36 0.680.510.48 query37 0.090.060.06 query38 0.040.040.04 query39 0.030.030.02 query40 0.150.120.12 query41 0.080.020.02 query42 0.030.020.02 query43 0.040.030.02 Total cold run time: 107.53 s Total hot run time: 32.06 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch asf-site updated (95ccb9098d1 -> fb6489e6d9e)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/doris-website.git discard 95ccb9098d1 Automated deployment with doris branch @ c48e670323787beecea9f8b93ef1ea5009ebd600 new fb6489e6d9e Automated deployment with doris branch @ c48e670323787beecea9f8b93ef1ea5009ebd600 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (95ccb9098d1) \ N -- N -- N refs/heads/asf-site (fb6489e6d9e) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/2.0/search-index.json | 2 +- docs/3.0/search-index.json | 2 +- docs/dev/search-index.json | 2 +- search-index.json| 2 +- zh-CN/docs/2.0/search-index.json | 2 +- zh-CN/docs/3.0/search-index.json | 2 +- zh-CN/docs/dev/search-index.json | 2 +- zh-CN/search-index.json | 2 +- 8 files changed, 8 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] chore: disable download md5sum check by default [doris]
w41ter commented on PR #50012: URL: https://github.com/apache/doris/pull/50012#issuecomment-2800611564 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
