[PR] [refine](pipelineX) refine dep in MultiCast/LocalExchange [doris]
Mryange opened a new pull request, #26463: URL: https://github.com/apache/doris/pull/26463 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refine](pipelineX) refine dep in MultiCast/LocalExchange [doris]
Mryange commented on PR #26463: URL: https://github.com/apache/doris/pull/26463#issuecomment-1794265677 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Refactor](Tvf) delete some unused code of tvf and add doc for `queries` tvf [doris]
doris-robot commented on PR #26460: URL: https://github.com/apache/doris/pull/26460#issuecomment-1794269476 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.22 seconds stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.0 seconds inserted 1000 Rows, about 344K ops/s storage size: 17162175641 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refine](pipelineX) refine dep in MultiCast/LocalExchange [doris]
github-actions[bot] commented on code in PR #26463:
URL: https://github.com/apache/doris/pull/26463#discussion_r1382907232
##
be/src/pipeline/exec/multi_cast_data_streamer.cpp:
##
@@ -87,10 +92,39 @@ Status MultiCastDataStreamer::push(RuntimeState* state,
doris::vectorized::Block
for (int i = 0; i < _sender_pos_to_read.size(); ++i) {
if (_sender_pos_to_read[i] == _multi_cast_blocks.end()) {
_sender_pos_to_read[i] = end;
+_set_ready_for_read(i);
}
}
_eos = eos;
return Status::OK();
}
+void MultiCastDataStreamer::_set_ready_for_read(int sender_idx) {
Review Comment:
warning: method '_set_ready_for_read' can be made const
[readability-make-member-function-const]
```suggestion
void MultiCastDataStreamer::_set_ready_for_read(int sender_idx) const {
```
be/src/pipeline/exec/multi_cast_data_streamer.h:79:
```diff
- void _set_ready_for_read(int sender_idx);
+ void _set_ready_for_read(int sender_idx) const;
```
##
be/src/pipeline/exec/multi_cast_data_streamer.cpp:
##
@@ -87,10 +92,39 @@
for (int i = 0; i < _sender_pos_to_read.size(); ++i) {
if (_sender_pos_to_read[i] == _multi_cast_blocks.end()) {
_sender_pos_to_read[i] = end;
+_set_ready_for_read(i);
}
}
_eos = eos;
return Status::OK();
}
+void MultiCastDataStreamer::_set_ready_for_read(int sender_idx) {
+if (!_has_dependencys) {
+return;
+}
+auto* dep = _dependencys[sender_idx];
+DCHECK(dep);
+dep->set_ready_for_read();
+}
+
+void MultiCastDataStreamer::_set_ready_for_read() {
+if (!_has_dependencys) {
+return;
+}
+for (auto* dep : _dependencys) {
+DCHECK(dep);
+dep->set_ready_for_read();
+}
+}
+
+void MultiCastDataStreamer::_block_reading(int sender_idx) {
Review Comment:
warning: method '_block_reading' can be made const
[readability-make-member-function-const]
```suggestion
void MultiCastDataStreamer::_block_reading(int sender_idx) const {
```
be/src/pipeline/exec/multi_cast_data_streamer.h:81:
```diff
- void _block_reading(int sender_idx);
+ void _block_reading(int sender_idx) const;
```
##
be/src/pipeline/exec/multi_cast_data_streamer.cpp:
##
@@ -87,10 +92,39 @@
for (int i = 0; i < _sender_pos_to_read.size(); ++i) {
if (_sender_pos_to_read[i] == _multi_cast_blocks.end()) {
_sender_pos_to_read[i] = end;
+_set_ready_for_read(i);
}
}
_eos = eos;
return Status::OK();
}
+void MultiCastDataStreamer::_set_ready_for_read(int sender_idx) {
+if (!_has_dependencys) {
+return;
+}
+auto* dep = _dependencys[sender_idx];
+DCHECK(dep);
+dep->set_ready_for_read();
+}
+
+void MultiCastDataStreamer::_set_ready_for_read() {
Review Comment:
warning: method '_set_ready_for_read' can be made const
[readability-make-member-function-const]
```suggestion
void MultiCastDataStreamer::_set_ready_for_read() const {
```
be/src/pipeline/exec/multi_cast_data_streamer.h:80:
```diff
- void _set_ready_for_read();
+ void _set_ready_for_read() const;
```
##
be/src/pipeline/pipeline_x/dependency.h:
##
@@ -885,7 +877,30 @@ struct LocalExchangeSharedState {
public:
ENABLE_FACTORY_CREATOR(LocalExchangeSharedState);
std::vector> data_queue;
+std::vector source_dependencies;
std::atomic running_sink_operators = 0;
+void add_running_sink_operators() { running_sink_operators++; }
+void sub_running_sink_operators() {
+running_sink_operators--;
+if (running_sink_operators == 0) {
+_set_ready_for_read();
+}
+}
+void _set_ready_for_read() {
+for (auto* dep : source_dependencies) {
+DCHECK(dep);
+dep->set_ready_for_read();
+}
+}
+void set_dep_by_channel_id(Dependency* dep, int channel_id) {
Review Comment:
warning: method 'set_dep_by_channel_id' can be made static
[readability-convert-member-functions-to-static]
```suggestion
static void set_dep_by_channel_id(Dependency* dep, int channel_id) {
```
##
be/src/pipeline/pipeline_x/dependency.h:
##
@@ -885,7 +877,30 @@
public:
ENABLE_FACTORY_CREATOR(LocalExchangeSharedState);
std::vector> data_queue;
+std::vector source_dependencies;
std::atomic running_sink_operators = 0;
+void add_running_sink_operators() { running_sink_operators++; }
+void sub_running_sink_operators() {
+running_sink_operators--;
+if (running_sink_operators == 0) {
+_set_ready_for_read();
+}
+}
+void _set_ready_for_read() {
+for (auto* dep : source_dependencies) {
+DCHECK(dep);
+dep->set_ready_for_read();
+}
+
Re: [PR] [feature](tvf)(jni-avro)jni-avro scanner add complex data types [doris]
doris-robot commented on PR #26236: URL: https://github.com/apache/doris/pull/26236#issuecomment-1794280939 TeamCity be ut coverage result: Function Coverage: 37.01% (8393/22676) Line Coverage: 29.44% (67927/230738) Region Coverage: 28.12% (35156/125020) Branch Coverage: 24.91% (17957/72094) Coverage Report: http://coverage.selectdb-in.cc/coverage/5833735cd7adda181f32245c06da87445b5c69ee_5833735cd7adda181f32245c06da87445b5c69ee/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](regression) fix regression test case failure [doris]
jacktengg commented on PR #26457: URL: https://github.com/apache/doris/pull/26457#issuecomment-1794284773 duplicate to #26453 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](regression) fix regression test case failure [doris]
jacktengg closed pull request #26457: [fix](regression) fix regression test case failure URL: https://github.com/apache/doris/pull/26457 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bug](bitmap) fix bitmap value copy operator not call reset [doris]
doris-robot commented on PR #26451: URL: https://github.com/apache/doris/pull/26451#issuecomment-1794286522 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.43 seconds stream load tsv: 575 seconds loaded 74807831229 Bytes, about 124 MB/s stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.8 seconds inserted 1000 Rows, about 347K ops/s storage size: 17162414063 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) estimate join cost when col stats are not available [doris]
doris-robot commented on PR #26086: URL: https://github.com/apache/doris/pull/26086#issuecomment-1794286990 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.13 seconds stream load tsv: 554 seconds loaded 74807831229 Bytes, about 128 MB/s stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.8 seconds inserted 1000 Rows, about 347K ops/s storage size: 17162077021 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [ci](perf) add new pipeline of tpch-sf100 [doris]
github-actions[bot] commented on PR #26334: URL: https://github.com/apache/doris/pull/26334#issuecomment-1794289293 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](tvf)(jni-avro)jni-avro scanner add complex data types [doris]
doris-robot commented on PR #26236: URL: https://github.com/apache/doris/pull/26236#issuecomment-1794292938 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.55 seconds stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 35 seconds loaded 861443392 Bytes, about 23 MB/s insert into select: 29.1 seconds inserted 1000 Rows, about 343K ops/s storage size: 17162301092 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve] add multi table sink to DorisBatchSink [doris-flink-connector]
zy-kkk merged PR #223: URL: https://github.com/apache/doris-flink-connector/pull/223 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](pipeline) simplify runtime state ctor (#25995) [doris]
doris-robot commented on PR #26461: URL: https://github.com/apache/doris/pull/26461#issuecomment-1794301421 TeamCity be ut coverage result: Function Coverage: 37.01% (8393/22678) Line Coverage: 29.44% (67921/230735) Region Coverage: 28.11% (35144/125019) Branch Coverage: 24.90% (17950/72090) Coverage Report: http://coverage.selectdb-in.cc/coverage/784db58f56a4cbc9d2df7454264d2c7be0d6986a_784db58f56a4cbc9d2df7454264d2c7be0d6986a/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-flink-connector) branch master updated: add multisink to DorisBatchSink (#223)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-flink-connector.git
The following commit(s) were added to refs/heads/master by this push:
new 32fd54e add multisink to DorisBatchSink (#223)
32fd54e is described below
commit 32fd54e3c1595897d02a188b00badebd0b9ef4e2
Author: wudi <676366...@qq.com>
AuthorDate: Mon Nov 6 16:27:08 2023 +0800
add multisink to DorisBatchSink (#223)
Support multi-table writing on DorisBatchSink
Example：
```java
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1);
DorisBatchSink.Builder builder =
DorisBatchSink.builder();
final DorisReadOptions.Builder readOptionBuilder =
DorisReadOptions.builder();
Properties properties = new Properties();
properties.setProperty("column_separator", ",");
properties.setProperty("line_delimiter", "\n");
properties.setProperty("format", "csv");
DorisOptions.Builder dorisBuilder = DorisOptions.builder();
dorisBuilder.setFenodes("127.0.0.1:8030")
.setTableIdentifier("")
.setUsername("root")
.setPassword("");
DorisExecutionOptions.Builder executionBuilder =
DorisExecutionOptions.builder();
executionBuilder.setLabelPrefix("label")
.setStreamLoadProp(properties)
.setDeletable(false)
.setBufferFlushMaxBytes(8 * 1024)
.setBufferFlushMaxRows(10)
.setBufferFlushIntervalMs(1000 * 10);
builder.setDorisReadOptions(readOptionBuilder.build())
.setDorisExecutionOptions(executionBuilder.build())
.setDorisOptions(dorisBuilder.build());
//Multiple table writing
RecordWithMeta record = new RecordWithMeta("test",
"test_flink_tmp1", "wangwu,1");
RecordWithMeta record1 = new RecordWithMeta("test",
"test_flink_tmp", "wangwu,1");
DataStreamSource source =
env.fromCollection(Arrays.asList(record, record1));
source.sinkTo(builder.build());
```
---
.../doris/flink/sink/batch/BatchRecordBuffer.java | 25 +
.../doris/flink/sink/batch/DorisBatchSink.java | 1 -
.../flink/sink/batch/DorisBatchStreamLoad.java | 109 ++---
.../doris/flink/sink/batch/DorisBatchWriter.java | 29 +-
.../doris/flink/sink/batch/RecordWithMeta.java | 62
.../doris/flink/sink/writer/LabelGenerator.java| 4 +
.../doris/flink/DorisSinkMultiTableExample.java| 101 +++
7 files changed, 271 insertions(+), 60 deletions(-)
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/BatchRecordBuffer.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/BatchRecordBuffer.java
index 5fa601d..1de6253 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/BatchRecordBuffer.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/BatchRecordBuffer.java
@@ -36,6 +36,8 @@ public class BatchRecordBuffer {
private int numOfRecords = 0;
private int bufferSizeBytes = 0;
private boolean loadBatchFirstRecord = true;
+private String database;
+private String table;
public BatchRecordBuffer(){}
@@ -45,6 +47,14 @@ public class BatchRecordBuffer {
this.buffer = ByteBuffer.allocate(bufferSize);
}
+public BatchRecordBuffer(String database, String table, byte[]
lineDelimiter, int bufferSize) {
+super();
+this.database = database;
+this.table = table;
+this.lineDelimiter = lineDelimiter;
+this.buffer = ByteBuffer.allocate(bufferSize);
+}
+
public void insert(byte[] record) {
ensureCapacity(record.length);
if(loadBatchFirstRecord){
@@ -141,4 +151,19 @@ public class BatchRecordBuffer {
this.bufferSizeBytes = bufferSizeBytes;
}
+public String getDatabase() {
+return database;
+}
+
+public void setDatabase(String database) {
+this.database = database;
+}
+
+public String getTable() {
+return table;
+}
+
+public void setTable(String table) {
+this.table = table;
+}
}
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/DorisBatchSink.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/DorisBatchSink.java
index 2c578d4..37d3973 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/DorisBatchSink.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/DorisBatchSink.java
@@ -87,7 +87,6 @@ public class DorisB
Re: [PR] [only test] [doris]
Mryange commented on PR #26452: URL: https://github.com/apache/doris/pull/26452#issuecomment-1794302191 run pipelinex_p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore] Print FE version even if it has been started [doris]
doris-robot commented on PR #26427: URL: https://github.com/apache/doris/pull/26427#issuecomment-1794303696 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.23 seconds stream load tsv: 554 seconds loaded 74807831229 Bytes, about 128 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.8 seconds inserted 1000 Rows, about 347K ops/s storage size: 17162232614 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](stats) merge insert action to reduce trasaction volume [doris]
doris-robot commented on PR #26163: URL: https://github.com/apache/doris/pull/26163#issuecomment-1794308121 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.68 seconds stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.9 seconds inserted 1000 Rows, about 346K ops/s storage size: 17162280498 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refine](pipelineX) refine dep in MultiCast/LocalExchange [doris]
doris-robot commented on PR #26463: URL: https://github.com/apache/doris/pull/26463#issuecomment-1794309353 TeamCity be ut coverage result: Function Coverage: 36.99% (8390/22682) Line Coverage: 29.42% (67889/230777) Region Coverage: 28.09% (35124/125024) Branch Coverage: 24.89% (17940/72090) Coverage Report: http://coverage.selectdb-in.cc/coverage/e80506ec276e8b15800ec1adef1cba2fd21073bf_e80506ec276e8b15800ec1adef1cba2fd21073bf/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [improve] add multi table sink to DorisSink [doris-flink-connector]
JNSimba opened a new pull request, #224: URL: https://github.com/apache/doris-flink-connector/pull/224 # Proposed changes ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: Revert "[fix](auto-partition) Fix auto partition concurrent conflict (#26166)" (#26448)
This is an automated email from the ASF dual-hosted git repository.
panxiaolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 1a83a39aec7 Revert "[fix](auto-partition) Fix auto partition
concurrent conflict (#26166)" (#26448)
1a83a39aec7 is described below
commit 1a83a39aec7fe4acf537bb1cab4232e2672b781d
Author: zclllyybb
AuthorDate: Mon Nov 6 16:39:19 2023 +0800
Revert "[fix](auto-partition) Fix auto partition concurrent conflict
(#26166)" (#26448)
This reverts commit f22611769944e78c28f1b0a1eeb7b7414a16e8db.
---
be/src/runtime/tablets_channel.cpp | 8 +-
be/src/runtime/tablets_channel.h | 1 -
docs/en/docs/admin-manual/config/fe-config.md | 8 +-
docs/zh-CN/docs/admin-manual/config/fe-config.md | 10 +--
.../main/java/org/apache/doris/common/Config.java | 7 --
.../apache/doris/analysis/PartitionExprUtil.java | 33 ++--
.../java/org/apache/doris/catalog/Database.java| 23 --
.../main/java/org/apache/doris/catalog/Env.java| 7 +-
.../java/org/apache/doris/catalog/OlapTable.java | 4 -
.../org/apache/doris/catalog/PartitionInfo.java| 95 --
.../apache/doris/datasource/InternalCatalog.java | 33 +++-
.../apache/doris/service/FrontendServiceImpl.java | 71
12 files changed, 41 insertions(+), 259 deletions(-)
diff --git a/be/src/runtime/tablets_channel.cpp
b/be/src/runtime/tablets_channel.cpp
index 68a35ccc109..91294135a06 100644
--- a/be/src/runtime/tablets_channel.cpp
+++ b/be/src/runtime/tablets_channel.cpp
@@ -84,7 +84,6 @@ void TabletsChannel::_init_profile(RuntimeProfile* profile) {
_slave_replica_timer = ADD_TIMER(_profile, "SlaveReplicaTime");
_add_batch_timer = ADD_TIMER(_profile, "AddBatchTime");
_write_block_timer = ADD_TIMER(_profile, "WriteBlockTime");
-_incremental_open_timer = ADD_TIMER(_profile, "IncrementalOpenTabletTime");
_memory_usage_counter = memory_usage->AddHighWaterMarkCounter("Total",
TUnit::BYTES);
_write_memory_usage_counter =
memory_usage->AddHighWaterMarkCounter("Write", TUnit::BYTES);
_flush_memory_usage_counter =
memory_usage->AddHighWaterMarkCounter("Flush", TUnit::BYTES);
@@ -121,14 +120,13 @@ Status TabletsChannel::open(const
PTabletWriterOpenRequest& request) {
}
Status TabletsChannel::incremental_open(const PTabletWriterOpenRequest&
params) {
-SCOPED_TIMER(_incremental_open_timer);
if (_state == kInitialized) { // haven't opened
return open(params);
}
std::lock_guard l(_lock);
std::vector* index_slots = nullptr;
int32_t schema_hash = 0;
-for (const auto& index : _schema->indexes()) {
+for (auto& index : _schema->indexes()) {
if (index->index_id == _index_id) {
index_slots = &index->slots;
schema_hash = index->schema_hash;
@@ -139,12 +137,14 @@ Status TabletsChannel::incremental_open(const
PTabletWriterOpenRequest& params)
return Status::InternalError("unknown index id, key={}",
_key.to_string());
}
// update tablets
+std::vector tablet_ids;
+tablet_ids.reserve(params.tablets_size());
size_t incremental_tablet_num = 0;
std::stringstream ss;
ss << "LocalTabletsChannel txn_id: " << _txn_id << " load_id: " <<
print_id(params.id())
<< " incremental open delta writer: ";
-for (const auto& tablet : params.tablets()) {
+for (auto& tablet : params.tablets()) {
if (_tablet_writers.find(tablet.tablet_id()) != _tablet_writers.end())
{
continue;
}
diff --git a/be/src/runtime/tablets_channel.h b/be/src/runtime/tablets_channel.h
index 4dca9050331..fe9c226829d 100644
--- a/be/src/runtime/tablets_channel.h
+++ b/be/src/runtime/tablets_channel.h
@@ -196,7 +196,6 @@ private:
RuntimeProfile::Counter* _slave_replica_timer = nullptr;
RuntimeProfile::Counter* _add_batch_timer = nullptr;
RuntimeProfile::Counter* _write_block_timer = nullptr;
-RuntimeProfile::Counter* _incremental_open_timer = nullptr;
};
template
diff --git a/docs/en/docs/admin-manual/config/fe-config.md
b/docs/en/docs/admin-manual/config/fe-config.md
index 11eab194221..bb54a4fe692 100644
--- a/docs/en/docs/admin-manual/config/fe-config.md
+++ b/docs/en/docs/admin-manual/config/fe-config.md
@@ -167,7 +167,7 @@ Default：100
the max txn number which bdbje can rollback when trying to rejoin the group
- `grpc_threadmgr_threads_nums`
+### `grpc_threadmgr_threads_nums`
Default: 4096
@@ -2763,9 +2763,3 @@ Forbid LocalDeployManager drop nodes to prevent errors in
the cluster.info file
Default: mysql
To ensure compatibility with the MySQL ecosystem, Doris includes a built-in
database called mysql. If this database conflicts with a user's own database,
please modify this field to replace the name of the Doris built-in MyS
Re: [PR] Revert "[fix](auto-partition) Fix auto partition concurrent conflict" [doris]
BiteThet merged PR #26448: URL: https://github.com/apache/doris/pull/26448 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-flink-connector) branch master updated: [feature] multiple tables to one for DatabaseSync (#208)
This is an automated email from the ASF dual-hosted git repository.
diwu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-flink-connector.git
The following commit(s) were added to refs/heads/master by this push:
new a4b4bdf [feature] multiple tables to one for DatabaseSync (#208)
a4b4bdf is described below
commit a4b4bdfc92bb8fecefffb6f4f81b0a8f577d142e
Author: Antg <57290855+codea...@users.noreply.github.com>
AuthorDate: Mon Nov 6 16:43:46 2023 +0800
[feature] multiple tables to one for DatabaseSync (#208)
---
.../org/apache/doris/flink/tools/cdc/CdcTools.java | 4 +-
.../apache/doris/flink/tools/cdc/DatabaseSync.java | 70 --
.../flink/tools/cdc/CdcMysqlSyncDatabaseCase.java | 10 +++-
.../tools/cdc/CdcOraclelSyncDatabaseCase.java | 6 +-
.../tools/cdc/CdcPostgresSyncDatabaseCase.java | 6 +-
.../tools/cdc/CdcSqlServerSyncDatabaseCase.java| 10 ++--
.../doris/flink/tools/cdc/DatabaseSyncTest.java| 40 +
7 files changed, 128 insertions(+), 18 deletions(-)
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/CdcTools.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/CdcTools.java
index 6a390ea..8a8b3db 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/CdcTools.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/CdcTools.java
@@ -103,6 +103,8 @@ public class CdcTools {
String tableSuffix = params.get("table-suffix");
String includingTables = params.get("including-tables");
String excludingTables = params.get("excluding-tables");
+String multiToOneOrigin = params.get("multi-to-one-origin");
+String multiToOneTarget = params.get("multi-to-one-target");
boolean createTableOnly = params.has("create-table-only");
boolean ignoreDefaultValue = params.has("ignore-default-value");
boolean useNewSchemaChange = params.has("use-new-schema-change");
@@ -112,7 +114,7 @@ public class CdcTools {
Configuration sinkConfig = Configuration.fromMap(sinkMap);
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
-databaseSync.create(env, database, config, tablePrefix, tableSuffix,
includingTables, excludingTables, ignoreDefaultValue, sinkConfig, tableMap,
createTableOnly, useNewSchemaChange);
+databaseSync.create(env, database, config, tablePrefix, tableSuffix,
includingTables, excludingTables,multiToOneOrigin,multiToOneTarget,
ignoreDefaultValue, sinkConfig, tableMap, createTableOnly, useNewSchemaChange);
databaseSync.build();
if(StringUtils.isNullOrWhitespaceOnly(jobName)){
jobName = String.format("%s-Doris Sync Database: %s", type,
config.getString("database-name","db"));
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DatabaseSync.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DatabaseSync.java
index fcd0f4c..99c45eb 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DatabaseSync.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DatabaseSync.java
@@ -33,6 +33,7 @@ import
org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.OutputTag;
import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -58,6 +59,7 @@ public abstract class DatabaseSync {
protected TableNameConverter converter;
protected Pattern includingPattern;
protected Pattern excludingPattern;
+protected Map multiToOneRulesPattern;
protected Map tableConfig;
protected Configuration sinkConfig;
protected boolean ignoreDefaultValue;
@@ -67,6 +69,8 @@ public abstract class DatabaseSync {
private boolean newSchemaChange;
protected String includingTables;
protected String excludingTables;
+protected String multiToOneOrigin;
+protected String multiToOneTarget;
public abstract void registerDriver() throws SQLException;
@@ -82,16 +86,19 @@ public abstract class DatabaseSync {
public void create(StreamExecutionEnvironment env, String database,
Configuration config,
String tablePrefix, String tableSuffix, String
includingTables,
- String excludingTables, boolean ignoreDefaultValue,
Configuration sinkConfig,
+ String excludingTables,String multiToOneOrigin,String
multiToOneTarget, boolean ignoreDefaultValue, Configuration sinkConfig,
Map tableConfig, boolean createTableOnly, boolean
useNewSchemaChange) {
this.env = env;
this.config = config;
this.databas
Re: [PR] [feature] multiple tables to one [doris-flink-connector]
JNSimba merged PR #208: URL: https://github.com/apache/doris-flink-connector/pull/208 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](Nereids): add Unit Test for PushdownTopNThroughJoin [doris]
github-actions[bot] commented on PR #26447: URL: https://github.com/apache/doris/pull/26447#issuecomment-1794328036 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Improvement](pipelineX) Improve local exchange on pipelineX engine [doris]
Gabriel39 opened a new pull request, #26464: URL: https://github.com/apache/doris/pull/26464 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](pipelineX) Improve local exchange on pipelineX engine [doris]
Gabriel39 commented on PR #26464: URL: https://github.com/apache/doris/pull/26464#issuecomment-1794335786 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](Nereids): add Unit Test for PushdownTopNThroughJoin [doris]
jackwener merged PR #26447: URL: https://github.com/apache/doris/pull/26447 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [enhancement](Nereids): add Unit Test for PushdownTopNThroughJoin (#26447)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 64ea450ac72 [enhancement](Nereids): add Unit Test for
PushdownTopNThroughJoin (#26447)
64ea450ac72 is described below
commit 64ea450ac72f72d3e72aec3652e709ad8e640547
Author: jakevin
AuthorDate: Mon Nov 6 16:54:36 2023 +0800
[enhancement](Nereids): add Unit Test for PushdownTopNThroughJoin (#26447)
---
.../rules/rewrite/PushdownTopNThroughJoinTest.java | 26 ++
1 file changed, 26 insertions(+)
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushdownTopNThroughJoinTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushdownTopNThroughJoinTest.java
index c033ca46bc9..8532f119298 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushdownTopNThroughJoinTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushdownTopNThroughJoinTest.java
@@ -131,6 +131,32 @@ class PushdownTopNThroughJoinTest extends
TestWithFeService implements MemoPatte
);
}
+@Test
+void testTwoJoinSql() {
+PlanChecker.from(connectContext)
+.analyze(
+"SELECT\n"
++ " V.*\n"
++ "FROM\n"
++ " (\n"
++ "SELECT t1.*\n"
++ "FROM t1 LEFT JOIN t1 t2 ON t1.k1 =
t2.k1\n"
++ "WHERE t1.k2 >= 20221001 AND t2.k2 = 1\n"
++ "ORDER BY t1.k2 ASC LIMIT 10\n"
++ " ) V\n"
++ " LEFT JOIN t1 O ON V.k1 = O.k1\n"
++ "ORDER BY k2 ASC LIMIT 10;")
+.rewrite()
+.matches(
+logicalJoin(
+logicalTopN(
+logicalProject(logicalJoin())
+),
+logicalProject(logicalOlapScan())
+)
+);
+}
+
@Test
void testProjectSql() {
PlanChecker.from(connectContext)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve] add multi table sink to DorisSink [doris-flink-connector]
zy-kkk merged PR #224: URL: https://github.com/apache/doris-flink-connector/pull/224 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-flink-connector) branch master updated: [improve] add multi table sink to DorisSink (#224)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-flink-connector.git
The following commit(s) were added to refs/heads/master by this push:
new 0b2a14a [improve] add multi table sink to DorisSink (#224)
0b2a14a is described below
commit 0b2a14a93d3b65115c25fe59d84d6b3375da517d
Author: wudi <676366...@qq.com>
AuthorDate: Mon Nov 6 16:56:04 2023 +0800
[improve] add multi table sink to DorisSink (#224)
DorisSink supports multi-table import.
Example:
```java
Configuration config = new Configuration();
//
config.setString("execution.savepoint.path","/tmp/checkpoint/chk-6");
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment(config);
env.setParallelism(1);
env.getCheckpointConfig().setCheckpointStorage("file:///tmp/checkpoint/");
env.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
env.setRestartStrategy(RestartStrategies.fixedDelayRestart(5,
Time.milliseconds(1)));
env.enableCheckpointing(1);
DorisSink.Builder builder = DorisSink.builder();
final DorisReadOptions.Builder readOptionBuilder =
DorisReadOptions.builder();
Properties properties = new Properties();
properties.setProperty("column_separator", ",");
properties.setProperty("line_delimiter", "\n");
properties.setProperty("format", "csv");
DorisOptions.Builder dorisBuilder = DorisOptions.builder();
dorisBuilder.setFenodes("127.0.0.1:8030")
.setTableIdentifier("")
.setUsername("root")
.setPassword("");
DorisExecutionOptions.Builder executionBuilder =
DorisExecutionOptions.builder();
executionBuilder.setLabelPrefix("xxx12")
.setStreamLoadProp(properties)
.setDeletable(false).enable2PC();
builder.setDorisReadOptions(readOptionBuilder.build())
.setDorisExecutionOptions(executionBuilder.build())
.setDorisOptions(dorisBuilder.build());
RecordWithMeta record = new RecordWithMeta("test",
"test_flink_tmp1", "wangwu,1");
RecordWithMeta record1 = new RecordWithMeta("test",
"test_flink_tmp", "wangwu,1");
DataStreamSource stringDataStreamSource =
env.fromCollection(
Arrays.asList(record, record1));
stringDataStreamSource.sinkTo(builder.build());
```
For details, please refer to
`org.apache.doris.flink.DorisSinkStreamMultiTableExample.java`
---
.../org/apache/doris/flink/cfg/DorisOptions.java | 4 +
.../org/apache/doris/flink/sink/BackendUtil.java | 12 ++
.../org/apache/doris/flink/sink/DorisSink.java | 1 -
.../doris/flink/sink/writer/DorisStreamLoad.java | 12 +-
.../doris/flink/sink/writer/DorisWriter.java | 211 +++--
.../doris/flink/sink/writer/DorisWriterState.java | 32 +++-
.../sink/writer/DorisWriterStateSerializer.java| 17 +-
.../doris/flink/sink/writer/LabelGenerator.java| 19 +-
.../flink/DorisSinkStreamMultiTableExample.java| 100 ++
.../org/apache/doris/flink/sink/HttpTestUtil.java | 41
.../flink/sink/writer/TestDorisStreamLoad.java | 12 +-
.../doris/flink/sink/writer/TestDorisWriter.java | 12 +-
.../writer/TestDorisWriterStateSerializer.java | 4 +-
13 files changed, 394 insertions(+), 83 deletions(-)
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/DorisOptions.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/DorisOptions.java
index f560eae..6391e91 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/DorisOptions.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/DorisOptions.java
@@ -52,6 +52,10 @@ public class DorisOptions extends DorisConnectionOptions {
return tableIdentifier;
}
+public void setTableIdentifier(String tableIdentifier) {
+this.tableIdentifier = tableIdentifier;
+}
+
public String save() throws IllegalArgumentException {
Properties copy = new Properties();
return IOUtils.propsToString(copy);
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/BackendUtil.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/BackendUtil.java
index 0d45e2f..954bdd0 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/BackendUtil.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/BackendUtil.java
@@ -17,7 +17,11 @@
package org.apache.doris.flink.sink;
+import org.apache.commons.lang3.StringUtils;
+impor
Re: [PR] [Improvement](pipelineX) Improve local exchange on pipelineX engine [doris]
github-actions[bot] commented on PR #26464: URL: https://github.com/apache/doris/pull/26464#issuecomment-1794352816 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](auditlog) add column catalog for audit log and audit log table [doris]
Yulei-Yang commented on PR #26403: URL: https://github.com/apache/doris/pull/26403#issuecomment-1794353212 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refine](pipelineX) refine dep in MultiCast/LocalExchange [doris]
doris-robot commented on PR #26463: URL: https://github.com/apache/doris/pull/26463#issuecomment-1794355078 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.43 seconds stream load tsv: 552 seconds loaded 74807831229 Bytes, about 129 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.2 seconds inserted 1000 Rows, about 342K ops/s storage size: 17162240740 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](planner)isnull predicate can't be safely constant folded in inlineview [doris]
starocean999 commented on PR #25377: URL: https://github.com/apache/doris/pull/25377#issuecomment-1794359698 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](export)build_key_ranges_and_filters should not change original range after executing extend_scan_key function [doris]
github-actions[bot] commented on PR #26462: URL: https://github.com/apache/doris/pull/26462#issuecomment-1794363235 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](export)build_key_ranges_and_filters should not change original range after executing extend_scan_key function [doris]
github-actions[bot] commented on PR #26462: URL: https://github.com/apache/doris/pull/26462#issuecomment-1794363319 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [ci](perf) add new pipeline of tpch-sf100 [doris]
doris-robot commented on PR #26334: URL: https://github.com/apache/doris/pull/26334#issuecomment-1794373681 Tpch sf1 test resutl on commit 63363d994c1692ec988cb521e30d34e0418d1a2c run tpch-sf1 query with default conf and session variables q1 382 204 203 203 q2 428 203 211 203 q3 126 102 103 102 q4 117 102 85 85 q5 180 158 155 155 q6 44 43 40 40 q7 198 153 157 153 q8 197 209 188 188 q9 300 271 247 247 q10 119 123 118 118 q11 124 117 129 117 q12 80 69 69 69 q13 110 108 96 96 q14 45 65 44 44 q15 81 65 68 65 q16 117 231 109 109 q17 122 109 112 109 q18 185 218 185 185 q19 66 58 60 58 q20 115 120 124 120 q21 265 250 266 250 q22 66 61 64 61 Total cold run time: 3467 ms Total hot run time: 2777 ms run tpch-sf1 query with default conf and set session variable runtime_filter_mode=off q1 160 177 156 156 q2 124 132 133 132 q3 97 92 100 92 q4 76 76 69 69 q5 138 145 144 144 q6 40 41 32 32 q7 146 130 133 130 q8 161 165 152 152 q9 244 226 207 207 q10 100 101 108 101 q11 96 101 108 101 q12 62 68 68 68 q13 114 108 103 103 q14 39 38 42 38 q15 59 54 59 54 q16 83 97 83 83 q17 98 90 92 90 q18 178 176 226 176 q19 59 54 60 54 q20 112 125 110 110 q21 247 226 235 226 q22 69 59 60 59 Total cold run time: 2502 ms Total hot run time: 2377 ms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [opt](expr) common expr pushdown supports unique key MoR and agg key table if all slot is key column. [doris]
xinyiZzz opened a new pull request, #26465: URL: https://github.com/apache/doris/pull/26465 ## Proposed changes common expr pushdown in dup key table, unique key MoW table, unique key MoR table and agg key table must satisfy all expr slot is key column. ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [ci](perf) add new pipeline of tpch-sf100 [doris]
doris-robot commented on PR #26334: URL: https://github.com/apache/doris/pull/26334#issuecomment-1794381539 Tpch sf1 test resutl on commit 63363d994c1692ec988cb521e30d34e0418d1a2c run tpch-sf1 query with default conf and session variables q1 382 204 203 203 q2 428 203 211 203 q3 126 102 103 102 q4 117 102 85 85 q5 180 158 155 155 q6 44 43 40 40 q7 198 153 157 153 q8 197 209 188 188 q9 300 271 247 247 q10 119 123 118 118 q11 124 117 129 117 q12 80 69 69 69 q13 110 108 96 96 q14 45 65 44 44 q15 81 65 68 65 q16 117 231 109 109 q17 122 109 112 109 q18 185 218 185 185 q19 66 58 60 58 q20 115 120 124 120 q21 265 250 266 250 q22 66 61 64 61 Total cold run time: 3467 ms Total hot run time: 2777 ms run tpch-sf1 query with default conf and set session variable runtime_filter_mode=off q1 160 177 156 156 q2 124 132 133 132 q3 97 92 100 92 q4 76 76 69 69 q5 138 145 144 144 q6 40 41 32 32 q7 146 130 133 130 q8 161 165 152 152 q9 244 226 207 207 q10 100 101 108 101 q11 96 101 108 101 q12 62 68 68 68 q13 114 108 103 103 q14 39 38 42 38 q15 59 54 59 54 q16 83 97 83 83 q17 98 90 92 90 q18 178 176 226 176 q19 59 54 60 54 q20 112 125 110 110 q21 247 226 235 226 q22 69 59 60 59 Total cold run time: 2502 ms Total hot run time: 2377 ms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bug](pipelineX) Fix potential bug using broadcast shuffle [doris]
github-actions[bot] commented on PR #26458: URL: https://github.com/apache/doris/pull/26458#issuecomment-1794384171 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bug](pipelineX) Fix potential bug using broadcast shuffle [doris]
github-actions[bot] commented on PR #26458: URL: https://github.com/apache/doris/pull/26458#issuecomment-1794384244 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](transaction) Fix concurrent schema change and txn cause dead lock [doris]
deardeng commented on code in PR #26428:
URL: https://github.com/apache/doris/pull/26428#discussion_r1382944155
##
regression-test/suites/schema_change/test_schema_change_concurrent_with_txn.groovy:
##
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.apache.doris.regression.suite.ClusterOptions
+import org.apache.doris.regression.util.NodeType
+
+suite('test_schema_change_concurrent_with_txn') {
+def options = new ClusterOptions()
+options.enableDebugPoints()
+options.feConfigs.add('publish_wait_time_second=-1')
+docker(options) {
+sql 'SET GLOBAL insert_visible_timeout_ms = 2000'
+
+def result = sql 'SELECT DATABASE()'
+def dbName = result[0][0]
+
+sql 'CREATE TABLE tbl_1 (k1 INT, k2 INT) PROPERTIES (
"light_schema_change" = "false")'
+sql 'INSERT INTO tbl_1 VALUES (1, 10)'
+sql 'INSERT INTO tbl_1 VALUES (2, 20)'
+order_qt_select_1_1 'SELECT * FROM tbl_1'
+
+sql 'CREATE TABLE tbl_2 AS SELECT * FROM tbl_1'
+order_qt_select_2_1 'SELECT * FROM tbl_1'
Review Comment:
tbl_2
##
fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java:
##
@@ -,8 +1114,28 @@ public void finishTransaction(long transactionId) throws
UserException {
LOG.info("finish transaction {} successfully, publish result: {}",
transactionState, publishResult.name());
}
-private void checkReplicaContinuousVersionSucc(long tabletId, Replica
replica, long version,
-PublishVersionTask backendPublishTask, Set errorReplicaIds,
List tabletSuccReplicas,
+private boolean isCheckTxnIgnoreAlterReplica(long transactionId, OlapTable
table) {
Review Comment:
function name publishSkipAlterReplica ？
##
fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java:
##
@@ -,8 +1114,28 @@ public void finishTransaction(long transactionId) throws
UserException {
LOG.info("finish transaction {} successfully, publish result: {}",
transactionState, publishResult.name());
}
-private void checkReplicaContinuousVersionSucc(long tabletId, Replica
replica, long version,
-PublishVersionTask backendPublishTask, Set errorReplicaIds,
List tabletSuccReplicas,
+private boolean isCheckTxnIgnoreAlterReplica(long transactionId, OlapTable
table) {
Review Comment:
function name publishSkipAlterReplica ？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [ci](perf) add new pipeline of tpch-sf100 [doris]
doris-robot commented on PR #26334: URL: https://github.com/apache/doris/pull/26334#issuecomment-1794388401 succ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](expr) common expr pushdown supports unique key MoR and agg key table if all slot is key column. [doris]
github-actions[bot] commented on PR #26465: URL: https://github.com/apache/doris/pull/26465#issuecomment-1794388833 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](expr) common expr pushdown supports unique key MoR and agg key table if all slot is key column. [doris]
github-actions[bot] commented on PR #26465: URL: https://github.com/apache/doris/pull/26465#issuecomment-1794389287 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [ci](perf) add new pipeline of tpch-sf100 [doris]
doris-robot commented on PR #26334: URL: https://github.com/apache/doris/pull/26334#issuecomment-1794389482 Tpch sf1 test resutl on commit 63363d994c1692ec988cb521e30d34e0418d1a2c run tpch-sf1 query with default conf and session variables q1 382 204 203 203 q2 428 203 211 203 q3 126 102 103 102 q4 117 102 85 85 q5 180 158 155 155 q6 44 43 40 40 q7 198 153 157 153 q8 197 209 188 188 q9 300 271 247 247 q10 119 123 118 118 q11 124 117 129 117 q12 80 69 69 69 q13 110 108 96 96 q14 45 65 44 44 q15 81 65 68 65 q16 117 231 109 109 q17 122 109 112 109 q18 185 218 185 185 q19 66 58 60 58 q20 115 120 124 120 q21 265 250 266 250 q22 66 61 64 61 Total cold run time: 3467 ms Total hot run time: 2777 ms run tpch-sf1 query with default conf and set session variable runtime_filter_mode=off q1 160 177 156 156 q2 124 132 133 132 q3 97 92 100 92 q4 76 76 69 69 q5 138 145 144 144 q6 40 41 32 32 q7 146 130 133 130 q8 161 165 152 152 q9 244 226 207 207 q10 100 101 108 101 q11 96 101 108 101 q12 62 68 68 68 q13 114 108 103 103 q14 39 38 42 38 q15 59 54 59 54 q16 83 97 83 83 q17 98 90 92 90 q18 178 176 226 176 q19 59 54 60 54 q20 112 125 110 110 q21 247 226 235 226 q22 69 59 60 59 Total cold run time: 2502 ms Total hot run time: 2377 ms= -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [ci](perf) add new pipeline of tpch-sf100 [doris]
doris-robot commented on PR #26334: URL: https://github.com/apache/doris/pull/26334#issuecomment-1794389631 Tpch sf1 test resutl on commit 63363d994c1692ec988cb521e30d34e0418d1a2c run tpch-sf1 query with default conf and session variables q1 382 204 203 203 q2 428 203 211 203 q3 126 102 103 102 q4 117 102 85 85 q5 180 158 155 155 q6 44 43 40 40 q7 198 153 157 153 q8 197 209 188 188 q9 300 271 247 247 q10 119 123 118 118 q11 124 117 129 117 q12 80 69 69 69 q13 110 108 96 96 q14 45 65 44 44 q15 81 65 68 65 q16 117 231 109 109 q17 122 109 112 109 q18 185 218 185 185 q19 66 58 60 58 q20 115 120 124 120 q21 265 250 266 250 q22 66 61 64 61 Total cold run time: 3467 ms Total hot run time: 2777 ms run tpch-sf1 query with default conf and set session variable runtime_filter_mode=off q1 160 177 156 156 q2 124 132 133 132 q3 97 92 100 92 q4 76 76 69 69 q5 138 145 144 144 q6 40 41 32 32 q7 146 130 133 130 q8 161 165 152 152 q9 244 226 207 207 q10 100 101 108 101 q11 96 101 108 101 q12 62 68 68 68 q13 114 108 103 103 q14 39 38 42 38 q15 59 54 59 54 q16 83 97 83 83 q17 98 90 92 90 q18 178 176 226 176 q19 59 54 60 54 q20 112 125 110 110 q21 247 226 235 226 q22 69 59 60 59 Total cold run time: 2502 ms Total hot run time: 2377 ms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [feature](pipelineX) add time unit when slow_dependency need to log [doris]
Mryange opened a new pull request, #26466: URL: https://github.com/apache/doris/pull/26466 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](pipelineX) add time unit when slow_dependency need to log [doris]
Mryange commented on PR #26466: URL: https://github.com/apache/doris/pull/26466#issuecomment-1794391606 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refine](pipelineX) refine dep in MultiCast/LocalExchange [doris]
Mryange commented on PR #26463: URL: https://github.com/apache/doris/pull/26463#issuecomment-1794396534 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [ci](perf) add new pipeline of tpch-sf100 [doris]
doris-robot commented on PR #26334: URL: https://github.com/apache/doris/pull/26334#issuecomment-1794396549 Tpch sf1 test resutl on commit 63363d994c1692ec988cb521e30d34e0418d1a2c run tpch-sf1 query with default conf and session variables q1 382 204 203 203 q2 428 203 211 203 q3 126 102 103 102 q4 117 102 85 85 q5 180 158 155 155 q6 44 43 40 40 q7 198 153 157 153 q8 197 209 188 188 q9 300 271 247 247 q10 119 123 118 118 q11 124 117 129 117 q12 80 69 69 69 q13 110 108 96 96 q14 45 65 44 44 q15 81 65 68 65 q16 117 231 109 109 q17 122 109 112 109 q18 185 218 185 185 q19 66 58 60 58 q20 115 120 124 120 q21 265 250 266 250 q22 66 61 64 61 Total cold run time: 3467 ms Total hot run time: 2777 ms run tpch-sf1 query with default conf and set session variable runtime_filter_mode=off q1 160 177 156 156 q2 124 132 133 132 q3 97 92 100 92 q4 76 76 69 69 q5 138 145 144 144 q6 40 41 32 32 q7 146 130 133 130 q8 161 165 152 152 q9 244 226 207 207 q10 100 101 108 101 q11 96 101 108 101 q12 62 68 68 68 q13 114 108 103 103 q14 39 38 42 38 q15 59 54 59 54 q16 83 97 83 83 q17 98 90 92 90 q18 178 176 226 176 q19 59 54 60 54 q20 112 125 110 110 q21 247 226 235 226 q22 69 59 60 59 Total cold run time: 2502 ms Total hot run time: 2377 ms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [improvement](agg) speed up serialization of aggregation function collect_list/collect_set [doris]
mrhhsg opened a new pull request, #26467: URL: https://github.com/apache/doris/pull/26467 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](agg) speed up serialization of aggregation function collect_list/collect_set [doris]
mrhhsg commented on PR #26467: URL: https://github.com/apache/doris/pull/26467#issuecomment-1794403189 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](pipelineX) add time unit when slow_dependency need to log [doris]
github-actions[bot] commented on PR #26466: URL: https://github.com/apache/doris/pull/26466#issuecomment-1794403877 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](pipelineX) Improve local exchange on pipelineX engine [doris]
doris-robot commented on PR #26464: URL: https://github.com/apache/doris/pull/26464#issuecomment-1794404850 TeamCity be ut coverage result: Function Coverage: 37.00% (8391/22677) Line Coverage: 29.42% (67887/230734) Region Coverage: 28.10% (35124/125014) Branch Coverage: 24.89% (17941/72090) Coverage Report: http://coverage.selectdb-in.cc/coverage/440ce4adc1d131a9e0d6c7561559b340946000f7_440ce4adc1d131a9e0d6c7561559b340946000f7/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] fix profile and change timer profile [doris]
HappenLee opened a new pull request, #26468: URL: https://github.com/apache/doris/pull/26468 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](auditlog) add column catalog for audit log and audit log table [doris]
doris-robot commented on PR #26403: URL: https://github.com/apache/doris/pull/26403#issuecomment-1794410706 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.71 seconds stream load tsv: 552 seconds loaded 74807831229 Bytes, about 129 MB/s stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.8 seconds inserted 1000 Rows, about 347K ops/s storage size: 17162248635 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch new_join updated: fix profile and change timer profile (#26468)
This is an automated email from the ASF dual-hosted git repository. panxiaolei pushed a commit to branch new_join in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/new_join by this push: new 20b3070b17d fix profile and change timer profile (#26468) 20b3070b17d is described below commit 20b3070b17de5c1f717c1bda6f455806fbdc5a83 Author: HappenLee AuthorDate: Mon Nov 6 17:32:28 2023 +0800 fix profile and change timer profile (#26468) --- be/src/pipeline/exec/hashjoin_build_sink.cpp | 7 --- be/src/pipeline/exec/hashjoin_build_sink.h | 8 be/src/pipeline/exec/join_build_sink_operator.cpp | 8 be/src/pipeline/exec/join_build_sink_operator.h| 4 ++-- .../exec/nested_loop_join_build_operator.h | 8 ++-- be/src/vec/common/hash_table/hash_map.h| 5 + be/src/vec/exec/join/vhash_join_node.cpp | 8 be/src/vec/exec/join/vhash_join_node.h | 22 +++--- be/src/vec/exec/join/vjoin_node_base.cpp | 4 ++-- be/src/vec/exec/join/vjoin_node_base.h | 4 ++-- be/src/vec/exec/join/vnested_loop_join_node.cpp| 4 ++-- be/src/vec/exec/join/vnested_loop_join_node.h | 8 ++-- 12 files changed, 36 insertions(+), 54 deletions(-) diff --git a/be/src/pipeline/exec/hashjoin_build_sink.cpp b/be/src/pipeline/exec/hashjoin_build_sink.cpp index 0cdb34605e0..c7af4f89ba4 100644 --- a/be/src/pipeline/exec/hashjoin_build_sink.cpp +++ b/be/src/pipeline/exec/hashjoin_build_sink.cpp @@ -92,17 +92,10 @@ Status HashJoinBuildSinkLocalState::init(RuntimeState* state, LocalSinkStateInfo _build_side_merge_block_timer = ADD_TIMER(profile(), "BuildSideMergeBlockTime"); _build_table_insert_timer = ADD_TIMER(record_profile, "BuildTableInsertTime"); _build_expr_call_timer = ADD_TIMER(record_profile, "BuildExprCallTime"); -_build_table_expanse_timer = ADD_TIMER(record_profile, "BuildTableExpanseTime"); -_build_table_convert_timer = ADD_TIMER(record_profile, "BuildTableConvertToPartitionedTime"); _build_side_compute_hash_timer = ADD_TIMER(record_profile, "BuildSideHashComputingTime"); -_build_runtime_filter_timer = ADD_TIMER(record_profile, "BuildRuntimeFilterTime"); _allocate_resource_timer = ADD_TIMER(profile(), "AllocateResourceTime"); -_build_buckets_counter = ADD_COUNTER(profile(), "BuildBuckets", TUnit::UNIT); -_build_buckets_fill_counter = ADD_COUNTER(profile(), "FilledBuckets", TUnit::UNIT); - -_build_collisions_counter = ADD_COUNTER(profile(), "BuildCollisions", TUnit::UNIT); // Hash Table Init _hash_table_init(state); diff --git a/be/src/pipeline/exec/hashjoin_build_sink.h b/be/src/pipeline/exec/hashjoin_build_sink.h index 16b58adf5fc..dc047e39848 100644 --- a/be/src/pipeline/exec/hashjoin_build_sink.h +++ b/be/src/pipeline/exec/hashjoin_build_sink.h @@ -107,16 +107,8 @@ protected: RuntimeProfile::Counter* _build_table_timer; RuntimeProfile::Counter* _build_expr_call_timer; RuntimeProfile::Counter* _build_table_insert_timer; -RuntimeProfile::Counter* _build_table_expanse_timer; -RuntimeProfile::Counter* _build_table_convert_timer; -RuntimeProfile::Counter* _build_buckets_counter; -RuntimeProfile::Counter* _build_buckets_fill_counter; - RuntimeProfile::Counter* _build_side_compute_hash_timer; RuntimeProfile::Counter* _build_side_merge_block_timer; -RuntimeProfile::Counter* _build_runtime_filter_timer; - -RuntimeProfile::Counter* _build_collisions_counter; RuntimeProfile::Counter* _allocate_resource_timer; diff --git a/be/src/pipeline/exec/join_build_sink_operator.cpp b/be/src/pipeline/exec/join_build_sink_operator.cpp index 2ed4ebfb0d3..280a35a0049 100644 --- a/be/src/pipeline/exec/join_build_sink_operator.cpp +++ b/be/src/pipeline/exec/join_build_sink_operator.cpp @@ -35,10 +35,10 @@ Status JoinBuildSinkLocalState::init(RuntimeState* stat _build_rows_counter = ADD_COUNTER(PipelineXSinkLocalState::profile(), "BuildRows", TUnit::UNIT); -_push_down_timer = ADD_TIMER(PipelineXSinkLocalState::profile(), - "PublishRuntimeFilterTime"); -_push_compute_timer = -ADD_TIMER(PipelineXSinkLocalState::profile(), "PushDownComputeTime"); +_publish_runtime_filter_timer = ADD_TIMER(PipelineXSinkLocalState::profile(), + "PublishRuntimeFilterTime"); +_runtime_filter_compute_timer = ADD_TIMER(PipelineXSinkLocalState::profile(), + "RuntimeFilterComputeTime"); return Status::OK(); } diff --git a/be/src/pipeline/exec/join_build_sink_operator.h b/be/src/pipeline/exec/join_build_sink_operator.h index 9034057658a..369b1548cfd 100644 --- a/be/src/pipeline/exec/join_build_sink_operator.h +++ b/be/src/pipeline/ex
Re: [PR] [enhancement](Nereids): when the DPhyper failed, roll back to cascades without join reorder [doris]
XieJiann commented on PR #26390: URL: https://github.com/apache/doris/pull/26390#issuecomment-1794408730 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [ci](perf) add new pipeline of tpch-sf100 [doris]
github-actions[bot] commented on PR #26334: URL: https://github.com/apache/doris/pull/26334#issuecomment-1794409990 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] fix profile and change timer profile [doris]
BiteThet merged PR #26468: URL: https://github.com/apache/doris/pull/26468 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [ci](perf) add new pipeline of tpch-sf100 [doris]
doris-robot commented on PR #26334: URL: https://github.com/apache/doris/pull/26334#issuecomment-1794411218 Tpch sf1 test resutl on commit e6a7d60e8113529baca30359d9c62391dd86d7ed run tpch-sf1 query with default conf and session variables q1 393 226 197 197 q2 470 197 209 197 q3 134 116 105 105 q4 110 83 81 81 q5 201 164 162 162 q6 45 42 42 42 q7 192 160 159 159 q8 214 192 182 182 q9 292 298 264 264 q10 111 115 122 115 q11 139 125 129 125 q12 77 67 70 67 q13 113 107 103 103 q14 48 45 45 45 q15 84 73 70 70 q16 114 107 98 98 q17 119 109 103 103 q18 210 216 185 185 q19 68 63 57 57 q20 119 123 127 123 q21 253 266 251 251 q22 91 71 78 71 Total cold run time: 3597 ms Total hot run time: 2802 ms run tpch-sf1 query with default conf and set session variable runtime_filter_mode=off q1 202 195 161 161 q2 122 144 126 126 q3 99 93 103 93 q4 87 76 75 75 q5 154 142 135 135 q6 32 33 33 33 q7 133 139 143 139 q8 158 154 158 154 q9 252 243 223 223 q10 110 103 109 103 q11 100 122 103 103 q12 62 61 57 57 q13 99 98 101 98 q14 42 42 40 40 q15 69 56 64 56 q16 82 87 85 85 q17 95 93 89 89 q18 229 185 176 176 q19 59 57 54 54 q20 115 111 123 111 q21 232 234 232 232 q22 66 59 62 59 Total cold run time: 2599 ms Total hot run time: 2402 ms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bug](pipelineX) Fix potential bug using broadcast shuffle [doris]
Gabriel39 merged PR #26458: URL: https://github.com/apache/doris/pull/26458 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](transaction) Fix concurrent schema change and txn cause dead lock [doris]
deardeng commented on code in PR #26428:
URL: https://github.com/apache/doris/pull/26428#discussion_r1383006786
##
be/src/olap/task/engine_publish_version_task.cpp:
##
@@ -235,7 +236,19 @@ Status EnginePublishVersionTask::finish() {
} else {
// check if the version exist, if not exist, then set publish
failed
if (_error_tablet_ids->find(tablet_id) ==
_error_tablet_ids->end()) {
-if (tablet->check_version_exist(version)) {
+bool exist_version = tablet->check_version_exist(version);
+bool is_converting = false;
+if (!exist_version && tablet->tablet_state() ==
TabletState::TABLET_NOTREADY) {
+bool is_mow = tablet->keys_type() ==
KeysType::UNIQUE_KEYS &&
Review Comment:
add some log?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (64ea450ac72 -> bd89028306d)
This is an automated email from the ASF dual-hosted git repository. gabriellee pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 64ea450ac72 [enhancement](Nereids): add Unit Test for PushdownTopNThroughJoin (#26447) add bd89028306d [bug](pipelineX) Fix potential bug using broadcast shuffle (#26458) No new revisions were added by this update. Summary of changes: be/src/pipeline/exec/exchange_sink_buffer.cpp | 1 + be/src/pipeline/exec/exchange_sink_operator.cpp | 11 +-- be/src/pipeline/exec/exchange_sink_operator.h | 1 - 3 files changed, 10 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [ci](perf) add new pipeline of tpch-sf100 [doris]
github-actions[bot] commented on PR #26334: URL: https://github.com/apache/doris/pull/26334#issuecomment-1794413887 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](agg) speed up serialization of aggregation function collect_list/collect_set [doris]
github-actions[bot] commented on PR #26467: URL: https://github.com/apache/doris/pull/26467#issuecomment-1794415571 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids)select base index if select mv fails [doris]
github-actions[bot] commented on PR #25715: URL: https://github.com/apache/doris/pull/25715#issuecomment-1794419884 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids)select base index if select mv fails [doris]
github-actions[bot] commented on PR #25715: URL: https://github.com/apache/doris/pull/25715#issuecomment-1794419975 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] fix profile and change timer profile [doris]
github-actions[bot] commented on PR #26468: URL: https://github.com/apache/doris/pull/26468#issuecomment-1794422844 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [ci](perf) add new pipeline of tpch-sf100 [doris]
github-actions[bot] commented on PR #26334: URL: https://github.com/apache/doris/pull/26334#issuecomment-1794423385 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](load) restore load job progress before retry load task [doris]
TsukiokaKogane opened a new pull request, #26469: URL: https://github.com/apache/doris/pull/26469 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](statistics)Improve stats sample strategy. [doris]
Jibing-Li commented on PR #26435: URL: https://github.com/apache/doris/pull/26435#issuecomment-1794428446 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](expr) common expr pushdown supports unique key MoR and agg key table if all slot is key column. [doris]
xinyiZzz commented on PR #26465: URL: https://github.com/apache/doris/pull/26465#issuecomment-1794436421 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](planner)isnull predicate can't be safely constant folded in inlineview [doris]
doris-robot commented on PR #25377: URL: https://github.com/apache/doris/pull/25377#issuecomment-1794439711 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.79 seconds stream load tsv: 553 seconds loaded 74807831229 Bytes, about 129 MB/s stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.5 seconds inserted 1000 Rows, about 350K ops/s storage size: 17161854943 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [ci](perf) add new pipeline of tpch-sf100 [doris]
github-actions[bot] commented on PR #26334: URL: https://github.com/apache/doris/pull/26334#issuecomment-1794439898 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refine](pipelineX) refine dep in MultiCast/LocalExchange [doris]
doris-robot commented on PR #26463: URL: https://github.com/apache/doris/pull/26463#issuecomment-1794440647 TeamCity be ut coverage result: Function Coverage: 37.00% (8392/22682) Line Coverage: 29.43% (67912/230777) Region Coverage: 28.11% (35147/125020) Branch Coverage: 24.90% (17949/72090) Coverage Report: http://coverage.selectdb-in.cc/coverage/724e2a7114a4b332ccfc172d6b77d8c67bb0717a_724e2a7114a4b332ccfc172d6b77d8c67bb0717a/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Bug] result is wrong when using subquery in Not BITMAP_CONTAINS function [doris]
LemonLiTree commented on issue #26405: URL: https://github.com/apache/doris/issues/26405#issuecomment-1794443872 2.0.2 https://github.com/apache/doris/assets/109049295/2482e2b6-3fec-4186-abfd-281a6918e25a";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](pipelineX) add time unit when slow_dependency need to log [doris]
doris-robot commented on PR #26466: URL: https://github.com/apache/doris/pull/26466#issuecomment-1794450527 TeamCity be ut coverage result: Function Coverage: 37.01% (8393/22678) Line Coverage: 29.43% (67918/230741) Region Coverage: 28.11% (35142/125019) Branch Coverage: 24.90% (17951/72092) Coverage Report: http://coverage.selectdb-in.cc/coverage/79707f8ea13794f93d0fb5893571ac8b06e0977e_79707f8ea13794f93d0fb5893571ac8b06e0977e/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](expr) common expr pushdown supports unique key MoR and agg key table if all slot is key column. [doris]
github-actions[bot] commented on PR #26465: URL: https://github.com/apache/doris/pull/26465#issuecomment-1794453910 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [ci](perf) add new pipeline of tpch-sf100 [doris]
doris-robot commented on PR #26334: URL: https://github.com/apache/doris/pull/26334#issuecomment-1794457609 Tpch sf1 test resutl on commit 14676ad6d11bf2839f2b0934705f711ad9c746d0 run tpch-sf1 query with default conf and session variables q1 430 230 187 187 q2 427 240 206 206 q3 123 113 116 113 q4 135 98 109 98 q5 175 167 169 167 q6 46 44 49 44 q7 182 161 157 157 q8 204 197 204 197 q9 274 252 256 252 q10 123 126 121 121 q11 134 134 120 120 q12 85 77 62 62 q13 109 111 105 105 q14 50 45 49 45 q15 71 71 70 70 q16 120 94 91 91 q17 111 110 102 102 q18 201 211 214 211 q19 71 66 63 63 q20 118 115 131 115 q21 279 244 228 228 q22 70 69 66 66 Total cold run time: 3538 ms Total hot run time: 2820 ms run tpch-sf1 query with default conf and set session variable runtime_filter_mode=off q1 167 173 198 173 q2 123 132 123 123 q3 104 94 97 94 q4 83 84 90 84 q5 158 154 156 154 q6 38 36 38 36 q7 150 142 143 142 q8 151 155 165 155 q9 239 225 229 225 q10 99 106 104 104 q11 101 115 101 101 q12 61 64 58 58 q13 102 92 98 92 q14 43 42 40 40 q15 59 57 63 57 q16 83 85 80 80 q17 90 87 91 87 q18 181 169 174 169 q19 67 54 54 54 q20 114 127 115 115 q21 233 245 227 227 q22 66 61 62 61 Total cold run time: 2512 ms Total hot run time: 2431 ms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](load) optimize the performance of row distribution [doris]
zclllyybb commented on PR #25546: URL: https://github.com/apache/doris/pull/25546#issuecomment-1794457643 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](pipelineX) Improve local exchange on pipelineX engine [doris]
doris-robot commented on PR #26464: URL: https://github.com/apache/doris/pull/26464#issuecomment-1794457807 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.24 seconds stream load tsv: 553 seconds loaded 74807831229 Bytes, about 129 MB/s stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.9 seconds inserted 1000 Rows, about 346K ops/s storage size: 17162045014 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [branch-2.0-var](sample) Fix computeSampleTabletIds NullPointerException [doris]
github-actions[bot] commented on PR #26434: URL: https://github.com/apache/doris/pull/26434#issuecomment-1794457782 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [branch-2.0-var](sample) Fix computeSampleTabletIds NullPointerException [doris]
github-actions[bot] commented on PR #26434: URL: https://github.com/apache/doris/pull/26434#issuecomment-1794457871 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [ci](perf) add new pipeline of tpch-sf100 [doris]
doris-robot commented on PR #26334: URL: https://github.com/apache/doris/pull/26334#issuecomment-1794460403 Tpch sf1 test resutl on commit 14676ad6d11bf2839f2b0934705f711ad9c746d0 run tpch-sf1 query with default conf and session variables q1 430 230 187 187 q2 427 240 206 206 q3 123 113 116 113 q4 135 98 109 98 q5 175 167 169 167 q6 46 44 49 44 q7 182 161 157 157 q8 204 197 204 197 q9 274 252 256 252 q10 123 126 121 121 q11 134 134 120 120 q12 85 77 62 62 q13 109 111 105 105 q14 50 45 49 45 q15 71 71 70 70 q16 120 94 91 91 q17 111 110 102 102 q18 201 211 214 211 q19 71 66 63 63 q20 118 115 131 115 q21 279 244 228 228 q22 70 69 66 66 Total cold run time: 3538 ms Total hot run time: 2820 ms run tpch-sf1 query with default conf and set session variable runtime_filter_mode=off q1 167 173 198 173 q2 123 132 123 123 q3 104 94 97 94 q4 83 84 90 84 q5 158 154 156 154 q6 38 36 38 36 q7 150 142 143 142 q8 151 155 165 155 q9 239 225 229 225 q10 99 106 104 104 q11 101 115 101 101 q12 61 64 58 58 q13 102 92 98 92 q14 43 42 40 40 q15 59 57 63 57 q16 83 85 80 80 q17 90 87 91 87 q18 181 169 174 169 q19 67 54 54 54 q20 114 127 115 115 q21 233 245 227 227 q22 66 61 62 61 Total cold run time: 2512 ms Total hot run time: 2431 ms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids)select base index if select mv fails [doris]
starocean999 merged PR #25715: URL: https://github.com/apache/doris/pull/25715 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](nereids)select base index if select mv fails (#25715)
This is an automated email from the ASF dual-hosted git repository.
starocean999 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 85a1db4b6c6 [fix](nereids)select base index if select mv fails (#25715)
85a1db4b6c6 is described below
commit 85a1db4b6c60b96192b01f41adaae5556310f6a1
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Mon Nov 6 17:57:19 2023 +0800
[fix](nereids)select base index if select mv fails (#25715)
---
.../doris/nereids/pattern/PatternDescriptor.java | 19 +++
.../mv/SelectMaterializedIndexWithAggregate.java | 20 ++--
.../mv/SelectMaterializedIndexWithoutAggregate.java | 10 +-
3 files changed, 34 insertions(+), 15 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/pattern/PatternDescriptor.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/pattern/PatternDescriptor.java
index c00f1711c5c..90c143be1aa 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/pattern/PatternDescriptor.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/pattern/PatternDescriptor.java
@@ -21,6 +21,8 @@ import org.apache.doris.nereids.rules.RulePromise;
import org.apache.doris.nereids.trees.plans.Plan;
import com.google.common.collect.ImmutableList;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
import java.util.List;
import java.util.Objects;
@@ -32,6 +34,7 @@ import java.util.function.Predicate;
* It can support pattern generic type to MatchedAction.
*/
public class PatternDescriptor {
+private static final Logger LOG =
LogManager.getLogger(PatternDescriptor.class);
public final Pattern pattern;
public final RulePromise defaultPromise;
@@ -63,6 +66,22 @@ public class PatternDescriptor {
return new PatternMatcher<>(pattern, defaultPromise, matchedAction);
}
+/**
+ * Same as thenApply, but catch all exception and return null
+ */
+public PatternMatcher
thenApplyNoThrow(
+MatchedAction matchedAction) {
+MatchedAction adaptMatchedAction = ctx -> {
+try {
+return matchedAction.apply(ctx);
+} catch (Exception ex) {
+LOG.warn("nereids apply rule failed, because {}",
ex.getMessage(), ex);
+return null;
+}
+};
+return new PatternMatcher<>(pattern, defaultPromise,
adaptMatchedAction);
+}
+
public PatternMatcher
thenMulti(
Function> matchedAction) {
MatchedMultiAction adaptMatchedAction = ctx
-> matchedAction.apply(ctx.root);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/mv/SelectMaterializedIndexWithAggregate.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/mv/SelectMaterializedIndexWithAggregate.java
index b41fdc2d4e8..abc842cb9a8 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/mv/SelectMaterializedIndexWithAggregate.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/mv/SelectMaterializedIndexWithAggregate.java
@@ -105,7 +105,7 @@ public class SelectMaterializedIndexWithAggregate extends
AbstractSelectMaterial
return ImmutableList.of(
// only agg above scan
// Aggregate(Scan)
-
logicalAggregate(logicalOlapScan().when(this::shouldSelectIndexWithAgg)).thenApply(ctx
-> {
+
logicalAggregate(logicalOlapScan().when(this::shouldSelectIndexWithAgg)).thenApplyNoThrow(ctx
-> {
LogicalAggregate agg = ctx.root;
LogicalOlapScan scan = agg.child();
SelectResult result = select(
@@ -140,7 +140,7 @@ public class SelectMaterializedIndexWithAggregate extends
AbstractSelectMaterial
// filter could push down scan.
// Aggregate(Filter(Scan))
logicalAggregate(logicalFilter(logicalOlapScan().when(this::shouldSelectIndexWithAgg)))
-.thenApply(ctx -> {
+.thenApplyNoThrow(ctx -> {
LogicalAggregate>
agg = ctx.root;
LogicalFilter filter =
agg.child();
LogicalOlapScan scan = filter.child();
@@ -191,7 +191,7 @@ public class SelectMaterializedIndexWithAggregate extends
AbstractSelectMaterial
// column pruning or other projections such as alias, etc.
// Aggregate(Project(Scan))
logicalAggregate(logicalProject(logicalOlapScan().when(this::shouldSelectIndexWithAgg)))
-.thenApply(ctx -> {
+.thenApplyNoThrow(ctx -> {
LogicalAggregate>
agg = ctx.root;
Re: [PR] [refactor](jni) unified jni framework for jdbc catalog [doris]
zy-kkk commented on PR #26317: URL: https://github.com/apache/doris/pull/26317#issuecomment-1794467532 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refine](pipelineX) refine dep in MultiCast/LocalExchange [doris]
doris-robot commented on PR #26463: URL: https://github.com/apache/doris/pull/26463#issuecomment-1794474659 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.01 seconds stream load tsv: 573 seconds loaded 74807831229 Bytes, about 124 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.3 seconds inserted 1000 Rows, about 341K ops/s storage size: 17162352047 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](load) opt row_distribution performance in load (part I) [doris]
zclllyybb closed pull request #25978: [opt](load) opt row_distribution performance in load (part I) URL: https://github.com/apache/doris/pull/25978 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](load) fix merged row number miscounting because of race condition [doris]
freemandealer opened a new pull request, #26470: URL: https://github.com/apache/doris/pull/26470 row numbers miscounting because of race condition, will cause load to fail sometimes with warning 'the rows number written doesn't match'. ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](load) opt row_distribution performance in load (part II) [doris]
zclllyybb closed pull request #25979: [opt](load) opt row_distribution performance in load (part II) URL: https://github.com/apache/doris/pull/25979 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](expr) common expr pushdown supports unique key MoR and agg key table if all slot is key column. [doris]
doris-robot commented on PR #26465: URL: https://github.com/apache/doris/pull/26465#issuecomment-1794479214 TeamCity be ut coverage result: Function Coverage: 37.00% (8393/22683) Line Coverage: 29.44% (67923/230753) Region Coverage: 28.11% (35143/125017) Branch Coverage: 24.90% (17952/72098) Coverage Report: http://coverage.selectdb-in.cc/coverage/42f153d07157793a5cbfdfd66bfcd29a8234ac75_42f153d07157793a5cbfdfd66bfcd29a8234ac75/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](agg) speed up serialization of aggregation function collect_list/collect_set [doris]
doris-robot commented on PR #26467: URL: https://github.com/apache/doris/pull/26467#issuecomment-1794481050 TeamCity be ut coverage result: Function Coverage: 37.01% (8393/22677) Line Coverage: 29.41% (67917/230912) Region Coverage: 28.10% (35150/125088) Branch Coverage: 24.90% (17956/72124) Coverage Report: http://coverage.selectdb-in.cc/coverage/03cc5ff4666c168427c2425f72c598bd7ebc8010_03cc5ff4666c168427c2425f72c598bd7ebc8010/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](tvf)(jni-avro)jni-avro scanner add complex data types [doris]
DongLiang-0 commented on PR #26236: URL: https://github.com/apache/doris/pull/26236#issuecomment-1794479695 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](load) fix merged row number miscounting because of race condition [doris]
freemandealer commented on PR #26470: URL: https://github.com/apache/doris/pull/26470#issuecomment-1794481170 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] Join rewrite [doris]
github-actions[bot] commented on code in PR #26138:
URL: https://github.com/apache/doris/pull/26138#discussion_r1383056514
##
be/src/exprs/bloom_filter_func.h:
##
@@ -310,137 +276,199 @@ struct CommonFindOp {
}
return new_size;
}
+};
-void find_batch(const BloomFilterAdaptor& bloom_filter, const char* data,
const uint8* nullmap,
-int number, uint8* results) const {
-for (int i = 0; i < number; i++) {
-results[i] = false;
-if (nullmap != nullptr && nullmap[i]) {
-continue;
+template
+struct CommonFindOp : BaseOp {
+uint16_t find_batch_olap_engine(const BloomFilterAdaptor& bloom_filter,
const char* data,
+const uint8* nullmap, uint16_t* offsets,
int number,
+const bool is_parse_column) {
+return find_batch_olap_engine_with_element_size(bloom_filter, data,
nullmap, offsets,
+number,
is_parse_column, sizeof(T));
+}
+
+void insert_batch(BloomFilterAdaptor& bloom_filter, const
vectorized::ColumnPtr& column,
+ size_t start) const {
+if (column->is_nullable()) {
+const auto* nullable = assert_cast(column.get());
+const auto& col = nullable->get_nested_column();
+const auto& nullmap =
+assert_cast(nullable->get_null_map_column())
+.get_data();
+
+const T* data = (T*)col.get_raw_data().data;
+for (size_t i = start; i < column->size(); i++) {
+if (!nullmap[i]) {
+bloom_filter.add_element(*(data + i));
+}
}
-if (!bloom_filter.test_element(*((T*)data + i))) {
-continue;
+} else {
+const T* data = (T*)column->get_raw_data().data;
+for (size_t i = start; i < column->size(); i++) {
+bloom_filter.add_element(*(data + i));
+}
+}
+}
+
+void find_batch(const BloomFilterAdaptor& bloom_filter, const
vectorized::ColumnPtr& column,
+uint8_t* results) const {
+if (column->is_nullable()) {
+const auto* nullable = assert_cast(column.get());
+const auto& nullmap =
+assert_cast(nullable->get_null_map_column())
+.get_data();
+
+const T* data =
(T*)nullable->get_nested_column().get_raw_data().data;
+for (size_t i = 0; i < column->size(); i++) {
+if (!nullmap[i]) {
+results[i] = bloom_filter.test_element(data[i]);
+} else {
+results[i] = false;
+}
+}
+} else {
+const T* data = (T*)column->get_raw_data().data;
+for (size_t i = 0; i < column->size(); i++) {
+results[i] = bloom_filter.test_element(data[i]);
}
-results[i] = true;
}
}
void insert(BloomFilterAdaptor& bloom_filter, const void* data) const {
bloom_filter.add_bytes((char*)data, sizeof(T));
}
bool find(const BloomFilterAdaptor& bloom_filter, const void* data) const {
-return bloom_filter.test(Slice((char*)data, sizeof(T)));
+return bloom_filter.test_element(((T*)data)[0]);
}
-bool find_olap_engine(const BloomFilterAdaptor& bloom_filter, const void*
data) const {
+bool find_olap_engine(const BloomFilterAdaptor& bloom_filter, const void*
data) const override {
return find(bloom_filter, data);
}
bool find(const BloomFilterAdaptor& bloom_filter, uint32_t data) const {
return bloom_filter.test(data);
}
};
-struct StringFindOp {
-void insert_batch(BloomFilterAdaptor& bloom_filter, const char* data,
const int* offsets,
- int number) const {
-LOG(FATAL) << "StringFindOp does not support insert_batch";
-}
-
-void insert_single(BloomFilterAdaptor& bloom_filter, const char* data)
const {
-LOG(FATAL) << "StringFindOp does not support insert_single";
-}
-
+struct StringFindOp : public BaseOp {
uint16_t find_batch_olap_engine(const BloomFilterAdaptor& bloom_filter,
const char* data,
const uint8* nullmap, uint16_t* offsets,
int number,
-const bool is_parse_column) const {
-LOG(FATAL) << "StringFindOp does not support find_batch_olap_engine";
-return 0;
-}
-
-void find_batch(const BloomFilterAdaptor& bloom_filter, const char* data,
const uint8* nullmap,
-int number, uint8* results) const {
-LOG(FATAL) << "StringFindOp does not support find_batch";
-}
-
-void insert(BloomFilterAdaptor& bloom_filter, const void* data) const {
-const
Re: [PR] [refactor](jni) unified jni framework for jdbc catalog [doris]
github-actions[bot] commented on code in PR #26317:
URL: https://github.com/apache/doris/pull/26317#discussion_r1383057871
##
be/src/vec/exec/vjdbc_connector.cpp:
##
@@ -223,6 +198,158 @@ Status JdbcConnector::query() {
return Status::OK();
}
+Status JdbcConnector::get_next(bool* eos, Block* block, int batch_size) {
+if (!_is_open) {
+return Status::InternalError("get_next before open of jdbc
connector.");
+}
+SCOPED_RAW_TIMER(&_jdbc_statistic._get_data_timer);
+JNIEnv* env = nullptr;
+RETURN_IF_ERROR(JniUtil::GetJNIEnv(&env));
+jboolean has_next =
+env->CallNonvirtualBooleanMethod(_executor_obj, _executor_clazz,
_executor_has_next_id);
+if (has_next != JNI_TRUE) {
+*eos = true;
+return Status::OK();
+}
+
+RETURN_IF_ERROR(JniUtil::GetJniExceptionMsg(env));
+
+auto column_size = _tuple_desc->slots().size();
+auto slots = _tuple_desc->slots();
+
+jobject map = _get_reader_params(block, env, column_size);
+long address =
+env->CallLongMethod(_executor_obj, _executor_get_block_address_id,
batch_size, map);
+RETURN_IF_ERROR(JniUtil::GetJniExceptionMsg(env));
+env->DeleteLocalRef(map);
+
+std::vector all_columns;
+for (size_t i = 0; i < column_size; ++i) {
+all_columns.push_back(i);
+}
+Status fill_block_status = JniConnector::fill_block(block, all_columns,
address);
+if (!fill_block_status) {
+return fill_block_status;
+}
+
+Status cast_status = _cast_string_to_special(block, env, column_size);
+
+if (!cast_status) {
+return cast_status;
+}
+
+return JniUtil::GetJniExceptionMsg(env);
+}
+
+Status JdbcConnector::append(vectorized::Block* block,
+ const vectorized::VExprContextSPtrs&
output_vexpr_ctxs,
+ uint32_t start_send_row, uint32_t* num_rows_sent,
Review Comment:
warning: pointer parameter 'num_rows_sent' can be pointer to const
[readability-non-const-parameter]
```suggestion
uint32_t start_send_row, const uint32_t*
num_rows_sent,
```
##
be/src/vec/exec/vjdbc_connector.cpp:
##
@@ -223,6 +198,158 @@
return Status::OK();
}
+Status JdbcConnector::get_next(bool* eos, Block* block, int batch_size) {
+if (!_is_open) {
+return Status::InternalError("get_next before open of jdbc
connector.");
+}
+SCOPED_RAW_TIMER(&_jdbc_statistic._get_data_timer);
+JNIEnv* env = nullptr;
+RETURN_IF_ERROR(JniUtil::GetJNIEnv(&env));
+jboolean has_next =
+env->CallNonvirtualBooleanMethod(_executor_obj, _executor_clazz,
_executor_has_next_id);
+if (has_next != JNI_TRUE) {
+*eos = true;
+return Status::OK();
+}
+
+RETURN_IF_ERROR(JniUtil::GetJniExceptionMsg(env));
+
+auto column_size = _tuple_desc->slots().size();
+auto slots = _tuple_desc->slots();
+
+jobject map = _get_reader_params(block, env, column_size);
+long address =
+env->CallLongMethod(_executor_obj, _executor_get_block_address_id,
batch_size, map);
+RETURN_IF_ERROR(JniUtil::GetJniExceptionMsg(env));
+env->DeleteLocalRef(map);
+
+std::vector all_columns;
+for (size_t i = 0; i < column_size; ++i) {
+all_columns.push_back(i);
+}
+Status fill_block_status = JniConnector::fill_block(block, all_columns,
address);
+if (!fill_block_status) {
+return fill_block_status;
+}
+
+Status cast_status = _cast_string_to_special(block, env, column_size);
+
+if (!cast_status) {
+return cast_status;
+}
+
+return JniUtil::GetJniExceptionMsg(env);
+}
+
+Status JdbcConnector::append(vectorized::Block* block,
+ const vectorized::VExprContextSPtrs&
output_vexpr_ctxs,
+ uint32_t start_send_row, uint32_t* num_rows_sent,
+ TOdbcTableType::type table_type) {
+RETURN_IF_ERROR(exec_stmt_write(block, output_vexpr_ctxs, num_rows_sent));
+COUNTER_UPDATE(_sent_rows_counter, *num_rows_sent);
+return Status::OK();
+}
+
+Status JdbcConnector::exec_stmt_write(Block* block, const VExprContextSPtrs&
output_vexpr_ctxs,
Review Comment:
warning: method 'exec_stmt_write' can be made static
[readability-convert-member-functions-to-static]
```suggestion
static Status JdbcConnector::exec_stmt_write(Block* block, const
VExprContextSPtrs& output_vexpr_ctxs,
```
##
be/src/vec/exec/vjdbc_connector.cpp:
##
@@ -223,6 +198,158 @@
return Status::OK();
}
+Status JdbcConnector::get_next(bool* eos, Block* block, int batch_size) {
+if (!_is_open) {
+return Status::InternalError("get_next before open of jdbc
connector.");
+}
+SCOPED_RAW_TIMER(&_jdbc_statistic._get_data_timer);
+JNIEnv* env = nullptr;
+RETURN_IF_ERROR(JniUtil::GetJNIEnv(&env));
+jboolea
Re: [PR] [fix](load) restore load job progress before retry failed load task [doris]
TsukiokaKogane commented on PR #26469: URL: https://github.com/apache/doris/pull/26469#issuecomment-1794485711 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
