[GitHub] [doris] zzzzzzzs opened a new pull request, #20044: [Function] Support date function: microsecond()
zzzs opened a new pull request, #20044: URL: https://github.com/apache/doris/pull/20044 # Proposed changes Issue Number: close #https://github.com/apache/doris/issues/19667 ## Problem summary Support date function: microsecond() ## Checklist(Required) * [ ] Does it affect the original behavior * [x] Has unit tests been added * [x] Has document been added or modified * [ ] Does it need to update dependencies * [x] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz opened a new pull request, #20045: [fix](memory) Fix meta_tool start failed
xinyiZzz opened a new pull request, #20045: URL: https://github.com/apache/doris/pull/20045 # Proposed changes Issue Number: close #xxx ## Problem summary exec env is not initialized when meta_tool start  ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #19996: [1.2-lts](pick) add a config for scan queue memory limit
yiguolei merged PR #19996: URL: https://github.com/apache/doris/pull/19996 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-lts updated: [Improvement](scan) add a config for scan queue memory limit (#19439) (#19996)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new 637ecc450c [Improvement](scan) add a config for scan queue memory
limit (#19439) (#19996)
637ecc450c is described below
commit 637ecc450c6665f0cb8ab0c651f9a403f9028dd8
Author: Xinyi Zou
AuthorDate: Thu May 25 15:01:00 2023 +0800
[Improvement](scan) add a config for scan queue memory limit (#19439)
(#19996)
Co-authored-by: Gabriel
---
be/src/runtime/runtime_state.h | 4
be/src/vec/exec/scan/vscan_node.cpp| 2 +-
.../src/main/java/org/apache/doris/analysis/SetVar.java| 4
.../src/main/java/org/apache/doris/qe/SessionVariable.java | 14 ++
gensrc/thrift/PaloInternalService.thrift | 2 ++
5 files changed, 25 insertions(+), 1 deletion(-)
diff --git a/be/src/runtime/runtime_state.h b/be/src/runtime/runtime_state.h
index 95cddb0424..dc4c5f97d1 100644
--- a/be/src/runtime/runtime_state.h
+++ b/be/src/runtime/runtime_state.h
@@ -85,6 +85,10 @@ public:
Status create_load_dir();
const TQueryOptions& query_options() const { return _query_options; }
+int64_t scan_queue_mem_limit() const {
+return _query_options.__isset.scan_queue_mem_limit ?
_query_options.scan_queue_mem_limit
+ :
_query_options.mem_limit / 20;
+}
ObjectPool* obj_pool() const { return _obj_pool.get(); }
std::shared_ptr obj_pool_ptr() const { return _obj_pool; }
diff --git a/be/src/vec/exec/scan/vscan_node.cpp
b/be/src/vec/exec/scan/vscan_node.cpp
index 002229ebb3..e708c22997 100644
--- a/be/src/vec/exec/scan/vscan_node.cpp
+++ b/be/src/vec/exec/scan/vscan_node.cpp
@@ -187,7 +187,7 @@ Status VScanNode::_init_profile() {
Status VScanNode::_start_scanners(const std::list& scanners) {
_scanner_ctx.reset(new ScannerContext(_state, this, _input_tuple_desc,
_output_tuple_desc,
scanners, limit(),
- _state->query_options().mem_limit /
20));
+ _state->scan_queue_mem_limit()));
RETURN_IF_ERROR(_scanner_ctx->init());
RETURN_IF_ERROR(_state->exec_env()->scanner_scheduler()->submit(_scanner_ctx.get()));
return Status::OK();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/SetVar.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/SetVar.java
index ebe38983f9..f880bf00d9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/SetVar.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/SetVar.java
@@ -147,6 +147,10 @@ public class SetVar {
this.value = new
StringLiteral(Long.toString(ParseUtil.analyzeDataVolumn(getValue().getStringValue(;
this.result = (LiteralExpr) this.value;
}
+if
(getVariable().equalsIgnoreCase(SessionVariable.SCAN_QUEUE_MEM_LIMIT)) {
+this.value = new
StringLiteral(Long.toString(ParseUtil.analyzeDataVolumn(getValue().getStringValue(;
+this.result = (LiteralExpr) this.value;
+}
if (getVariable().equalsIgnoreCase("is_report_success")) {
variable = SessionVariable.ENABLE_PROFILE;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
index e1ac24bd55..432e6b4cc1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
@@ -50,6 +50,7 @@ public class SessionVariable implements Serializable,
Writable {
public static final Logger LOG =
LogManager.getLogger(SessionVariable.class);
public static final String EXEC_MEM_LIMIT = "exec_mem_limit";
+public static final String SCAN_QUEUE_MEM_LIMIT = "scan_queue_mem_limit";
public static final String QUERY_TIMEOUT = "query_timeout";
public static final String ENABLE_PROFILE = "enable_profile";
public static final String SQL_MODE = "sql_mode";
@@ -271,6 +272,9 @@ public class SessionVariable implements Serializable,
Writable {
@VariableMgr.VarAttr(name = EXEC_MEM_LIMIT)
public long maxExecMemByte = 2147483648L;
+@VariableMgr.VarAttr(name = SCAN_QUEUE_MEM_LIMIT)
+public long maxScanQueueMemByte = 2147483648L / 20;
+
@VariableMgr.VarAttr(name = ENABLE_SPILLING)
public boolean enableSpilling = false;
@@ -736,6 +740,10 @@ public class SessionVariable implements Serializable,
Writable {
return maxExecMemByte;
}
+public long getMaxScanQueueExecMemByte() {
+return maxScanQueueMemByte;
+}
+
public int getQueryTimeoutS() {
return queryTimeoutS;
[GitHub] [doris] zzzzzzzs commented on pull request #20044: [Function] Support date function: microsecond()
zzzs commented on PR #20044: URL: https://github.com/apache/doris/pull/20044#issuecomment-1562386075 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] airborne12 opened a new issue, #20046: [Bug] indices_size key not found cause single_replica_load BE coredump
airborne12 opened a new issue, #20046: URL: https://github.com/apache/doris/issues/20046 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? streamload enable single_replica_load，PTabletWriteSlaveRequest_IndexSizeMap key not found, which caused coredump in BE. google::protobuf::Map indices_size = request->inverted_indices_size(); F0524 21:58:15.057839 1302291 global.cpp:309] /mnt/disk1/chenkaijie/doris/thirdparty/installed/include/google/protobuf/map.h:1214 CHECK failed: it != end(): key not found: 1 F0524 21:58:15.057839 1302291 global.cpp:309] /mnt/disk1/chenkaijie/doris/thirdparty/installed/include/google/protobuf/map.h:1214 CHECK failed: it != end(): key not found: 1F0524 21:58:15.682101 1302292 global.cpp:309] /mnt/disk1/chenkaijie/doris/thirdparty/installed/include/google/protobuf/map.h:1214 CHECK failed: it != end(): key not found: 0 F0524 21:58:15.057839 1302291 global.cpp:309] /mnt/disk1/chenkaijie/doris/thirdparty/installed/include/google/protobuf/map.h:1214 CHECK failed: it != end(): key not found: 1F0524 21:58:15.682101 1302292 global.cpp:309] /mnt/disk1/chenkaijie/doris/thirdparty/installed/include/google/protobuf/map.h:1214 CHECK failed: it != end(): key not found: 0F0524 21:58:16.073918 1302297 global.cpp:309] /mnt/disk1/chenkaijie/doris/thirdparty/installed/include/google/protobuf/map.h:1214 CHECK failed: it != end(): key not found: 0 F0524 21:58:15.057839 1302291 global.cpp:309] /mnt/disk1/chenkaijie/doris/thirdparty/installed/include/google/protobuf/map.h:1214 CHECK failed: it != end(): key not found: 1F0524 21:58:15.682101 1302292 global.cpp:309] /mnt/disk1/chenkaijie/doris/thirdparty/installed/include/google/protobuf/map.h:1214 CHECK failed: it != end(): key not found: 0F0524 21:58:16.073918 1302297 global.cpp:309] /mnt/disk1/chenkaijie/doris/thirdparty/installed/include/google/protobuf/map.h:1214 CHECK failed: it != end(): key not found: 0F0524 21:58:16.137092 1302295 global.cpp:309] /mnt/disk1/chenkaijie/doris/thirdparty/installed/include/google/protobuf/map.h:1214 CHECK failed: it != end(): key not found: 0 F0524 21:58:15.057839 1302291 global.cpp:309] /mnt/disk1/chenkaijie/doris/thirdparty/installed/include/google/protobuf/map.h:1214 CHECK failed: it != end(): key not found: 1F0524 21:58:15.682101 1302292 global.cpp:309] /mnt/disk1/chenkaijie/doris/thirdparty/installed/include/google/protobuf/map.h:1214 CHECK failed: it != end(): key not found: 0F0524 21:58:16.073918 1302297 global.cpp:309] /mnt/disk1/chenkaijie/doris/thirdparty/installed/include/google/protobuf/map.h:1214 CHECK failed: it != end(): key not found: 0F0524 21:58:16.137092 1302295 global.cpp:309] /mnt/disk1/chenkaijie/doris/thirdparty/installed/include/google/protobuf/map.h:1214 CHECK failed: it != end(): key not found: 0F0524 21:58:16.141532 1302296 global.cpp:309] /mnt/disk1/chenkaijie/doris/thirdparty/installed/include/google/protobuf/map.h:1214 CHECK failed: it != end(): key not found: 0 core: #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. [Current thread is 1 (Thread 0x7f4216926700 (LWP 1172644))] (gdb) bt #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x7f43c4bea859 in __GI_abort () at abort.c:79 #2 0x55a0f047ccc9 in ?? () #3 0x55a0f04722dd in google::LogMessage::Fail() () #4 0x55a0f0474819 in google::LogMessage::SendToLog() () #5 0x55a0f0471e46 in google::LogMessage::Flush() () #6 0x55a0f0474e89 in google::LogMessageFatal::~LogMessageFatal() () #7 0x55a0f0cb0226 in ?? () #8 0x55a0f0e68545 in google::protobuf::internal::LogMessage::Finish() () #9 0x55a0e9ea10ef in google::protobuf::Map::at ( key=@0x7f4184f3bcc0: 1, this=0x7f42b26ef160) at /mnt/disk1/chenkaijie/doris/thirdparty/installed/include/google/protobuf/map.h:1214 #10 operator() (__closure=0x7f42b26ef000) at /mnt/disk1/chenkaijie/doris/be/src/service/internal_service.cpp:1315 #11 0x55a0e92f7250 in std::function::operator()() const (this=0x7f421691f728) at /mnt/disk1/chenkaijie/env/ldb_toolchain/include/c++/11/bits/std_function.h:560 #12 doris::PriorityThreadPool::work_thread (this=, thread_id=) at /mnt/disk1/chenkaijie/doris/be/src/util/priority_thread_pool.hpp:146 #13 0x55a0f33db8d0 in std::execute_native_thread_routine (__p=0x7f43981ec3e0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #14 0x7f43c4f11609 in start_thread (arg=) at pthread_create.c:477 #15 0x7f43c4ce7163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 ### What You Expected? check key is not found ### How to Reproduce? _No resp
[GitHub] [doris] Gabriel39 commented on pull request #19983: [test] test decimalv3
Gabriel39 commented on PR #19983: URL: https://github.com/apache/doris/pull/19983#issuecomment-1562387912 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] AshinGau commented on a diff in pull request #19950: [Enhencement](JDBC Catalog) refactor jdbc catalog insert logic
AshinGau commented on code in PR #19950:
URL: https://github.com/apache/doris/pull/19950#discussion_r1205084188
##
fe/java-udf/src/main/java/org/apache/doris/udf/JdbcExecutor.java:
##
@@ -154,22 +158,137 @@ public int write(String sql) throws UdfRuntimeException {
}
}
-public int write(Map params) {
+public int write(Map params) throws UdfRuntimeException {
String[] requiredFields = params.get("required_fields").split(",");
String[] types = params.get("columns_types").split("#");
long metaAddress = Long.parseLong(params.get("meta_address"));
// Get sql string from configuration map
-// String sql = params.get("write_sql");
ColumnType[] columnTypes = new ColumnType[types.length];
for (int i = 0; i < types.length; i++) {
columnTypes[i] = ColumnType.parseType(requiredFields[i], types[i]);
}
VectorTable batchTable = new VectorTable(columnTypes, requiredFields,
metaAddress);
// todo: insert the batch table by PreparedStatement
// Can't release or close batchTable, it's released by c++
+try {
+insert(batchTable);
+} catch (SQLException e) {
+throw new UdfRuntimeException("JDBC executor sql has error: ", e);
+}
return batchTable.getNumRows();
}
+private int insert(VectorTable data) throws SQLException {
+for (int i = 0; i < data.getNumRows(); ++i) {
+for (int j = 0; j < data.getColumns().length; ++j) {
+insertColumn(i, j, data.getColumns()[j]);
+}
+preparedStatement.addBatch();
+}
+preparedStatement.executeBatch();
+preparedStatement.clearBatch();
+return data.getNumRows();
+}
+
+private void insertColumn(int rowIdx, int colIdx, VectorColumn column)
throws SQLException {
+int parameterIndex = colIdx + 1;
+ColumnType.Type dorisType = column.getColumnTyp();
+if (column.isNullAt(rowIdx)) {
+insertNullColumn(parameterIndex, dorisType);
+return;
+}
+switch (dorisType) {
+case BOOLEAN:
+preparedStatement.setBoolean(parameterIndex,
column.getBoolean(rowIdx));
+break;
+case TINYINT:
+preparedStatement.setByte(parameterIndex, (byte)
column.getInt(rowIdx));
+break;
+case SMALLINT:
+preparedStatement.setShort(parameterIndex, (short)
column.getInt(rowIdx));
+break;
+case INT:
+preparedStatement.setInt(parameterIndex,
column.getInt(rowIdx));
+break;
+case BIGINT:
+preparedStatement.setLong(parameterIndex,
column.getLong(rowIdx));
+break;
+case FLOAT:
+preparedStatement.setFloat(parameterIndex,
column.getFloat(rowIdx));
+break;
+case DOUBLE:
+preparedStatement.setDouble(parameterIndex,
column.getDouble(rowIdx));
+break;
+case LARGEINT:
+case DECIMALV2:
+case DECIMAL32:
+case DECIMAL64:
+case DECIMAL128:
+preparedStatement.setBigDecimal(parameterIndex,
column.getDecimal(rowIdx));
Review Comment:
`LARGEINT` is the `java.math.BigInteger` in java, so we should add
`getBigInt()` in `org.apache.doris.jni.vec.ColumnValue`.
##
fe/java-udf/src/main/java/org/apache/doris/udf/JdbcExecutor.java:
##
@@ -154,22 +158,137 @@ public int write(String sql) throws UdfRuntimeException {
}
}
-public int write(Map params) {
+public int write(Map params) throws UdfRuntimeException {
String[] requiredFields = params.get("required_fields").split(",");
String[] types = params.get("columns_types").split("#");
long metaAddress = Long.parseLong(params.get("meta_address"));
// Get sql string from configuration map
-// String sql = params.get("write_sql");
ColumnType[] columnTypes = new ColumnType[types.length];
for (int i = 0; i < types.length; i++) {
columnTypes[i] = ColumnType.parseType(requiredFields[i], types[i]);
}
VectorTable batchTable = new VectorTable(columnTypes, requiredFields,
metaAddress);
// todo: insert the batch table by PreparedStatement
// Can't release or close batchTable, it's released by c++
+try {
+insert(batchTable);
+} catch (SQLException e) {
+throw new UdfRuntimeException("JDBC executor sql has error: ", e);
+}
return batchTable.getNumRows();
}
+private int insert(VectorTable data) throws SQLException {
+for (int i = 0; i < data.getNumRows(); ++i) {
+for (int j = 0; j < data.getColumns().length; ++j) {
+ins
[GitHub] [doris] hello-stephen commented on pull request #20037: [Improve](performance) introduce ResourePool to cache SegmentIterator…
hello-stephen commented on PR #20037: URL: https://github.com/apache/doris/pull/20037#issuecomment-1562389250 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.44 seconds stream load tsv: 445 seconds loaded 74807831229 Bytes, about 160 MB/s stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s insert into select: 25.5 seconds inserted 1000 Rows, about 392K ops/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230525070643_clickbench_pr_150242.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] airborne12 opened a new pull request, #20047: [Fix](single replica load) fix indices_size key not found core
airborne12 opened a new pull request, #20047: URL: https://github.com/apache/doris/pull/20047 # Proposed changes Issue Number: close #20046 ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20045: [fix](memory) Fix meta_tool start failed
github-actions[bot] commented on PR #20045: URL: https://github.com/apache/doris/pull/20045#issuecomment-1562390152 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] airborne12 commented on pull request #20047: [Fix](single replica load) fix indices_size key not found core
airborne12 commented on PR #20047: URL: https://github.com/apache/doris/pull/20047#issuecomment-1562390841 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20041: [improvement](exception-safe) sort node is completely exception safe
hello-stephen commented on PR #20041: URL: https://github.com/apache/doris/pull/20041#issuecomment-1562390507 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 35.06 seconds stream load tsv: 441 seconds loaded 74807831229 Bytes, about 161 MB/s stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s insert into select: 25.6 seconds inserted 1000 Rows, about 390K ops/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230525070745_clickbench_pr_150284.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 commented on a diff in pull request #20044: [Function] Support date function: microsecond()
Gabriel39 commented on code in PR #20044:
URL: https://github.com/apache/doris/pull/20044#discussion_r1205085289
##
docs/en/docs/sql-manual/sql-functions/date-time-functions/microsecond.md:
##
@@ -0,0 +1,48 @@
+---
+{
+"title": "minute",
Review Comment:
```suggestion
"title": "microsecond",
```
##
docs/zh-CN/docs/sql-manual/sql-functions/date-time-functions/microsecond.md:
##
@@ -0,0 +1,49 @@
+---
+{
+"title": "hour",
Review Comment:
```suggestion
"title": "microsecond",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20047: [Fix](single replica load) fix indices_size key not found core
github-actions[bot] commented on PR #20047: URL: https://github.com/apache/doris/pull/20047#issuecomment-1562393704 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20047: [Fix](single replica load) fix indices_size key not found core
github-actions[bot] commented on PR #20047: URL: https://github.com/apache/doris/pull/20047#issuecomment-1562393762 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20047: [Fix](single replica load) fix indices_size key not found core
github-actions[bot] commented on PR #20047: URL: https://github.com/apache/doris/pull/20047#issuecomment-1562400449 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangbo opened a new pull request, #20048: [feature](executor)Support query queue
wangbo opened a new pull request, #20048: URL: https://github.com/apache/doris/pull/20048 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20005: [Bug](runtime filter) Fix min/max filter for decimalv3
github-actions[bot] commented on PR #20005: URL: https://github.com/apache/doris/pull/20005#issuecomment-1562406385 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20030: [test](pipline)Adjust mem limit to 50% for regression
github-actions[bot] commented on PR #20030: URL: https://github.com/apache/doris/pull/20030#issuecomment-1562410946 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20030: [test](pipline)Adjust mem limit to 50% for regression
github-actions[bot] commented on PR #20030: URL: https://github.com/apache/doris/pull/20030#issuecomment-1562411008 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] LiBinfeng-01 opened a new pull request, #20049: [Fix](Nereids) Using switch to control minidump input serialize
LiBinfeng-01 opened a new pull request, #20049: URL: https://github.com/apache/doris/pull/20049 # Proposed changes ## Problem summary Before change, when doing optimize use Nereids planner, input will serialize to memory first. And when bug happen, it would be dump to minidump file when catching the exception. We found that serialization process will cause the performance when statistic message too large or when optimization time be small enough. So the user minidump using should change to ONLY YOU OPEN MINIDUMP SWITCH(set enable_minidump=true;) can you use it. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #20048: [feature](executor)Support query queue
yiguolei commented on code in PR #20048:
URL: https://github.com/apache/doris/pull/20048#discussion_r1205103243
##
fe/fe-common/src/main/java/org/apache/doris/common/Config.java:
##
@@ -1488,6 +1488,9 @@ public class Config extends ConfigBase {
@ConfField(mutable = true, masterOnly = true, expType =
ExperimentalType.EXPERIMENTAL)
public static boolean enable_resource_group = true;
+@ConfField(mutable = true, expType = ExperimentalType.EXPERIMENTAL)
+public static boolean enable_query_queue = true;
Review Comment:
add docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #20049: [Fix](Nereids) Using switch to control minidump input serialize
LiBinfeng-01 commented on PR #20049: URL: https://github.com/apache/doris/pull/20049#issuecomment-1562411941 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yangzhg merged pull request #20035: [fix](odbc) fix odbc connection may fall in dead lock because of wrong abort transaction
yangzhg merged PR #20035: URL: https://github.com/apache/doris/pull/20035 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.1-lts updated: [fix](odbc) fix odbc connection may fall in dead lock because of wrng abort transaction (#20035)
This is an automated email from the ASF dual-hosted git repository. yangzhg pushed a commit to branch branch-1.1-lts in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-1.1-lts by this push: new 8c6dd4cece [fix](odbc) fix odbc connection may fall in dead lock because of wrng abort transaction (#20035) 8c6dd4cece is described below commit 8c6dd4cece4fa8bcd1999a7d9b191ed2e33fbacc Author: Zhengguo Yang AuthorDate: Thu May 25 15:26:12 2023 +0800 [fix](odbc) fix odbc connection may fall in dead lock because of wrng abort transaction (#20035) --- be/src/exec/odbc_connector.cpp | 1 + 1 file changed, 1 insertion(+) diff --git a/be/src/exec/odbc_connector.cpp b/be/src/exec/odbc_connector.cpp index 1a12884cb0..432079b4c3 100644 --- a/be/src/exec/odbc_connector.cpp +++ b/be/src/exec/odbc_connector.cpp @@ -61,6 +61,7 @@ ODBCConnector::ODBCConnector(const ODBCConnectorParam& param) _tuple_desc(param.tuple_desc), _output_expr_ctxs(param.output_expr_ctxs), _is_open(false), + _is_in_transaction(false), _field_num(0), _env(nullptr), _dbc(nullptr), - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt opened a new pull request, #20050: [Bug](hash-join) fix hash join share hash map result not stable
BiteThet opened a new pull request, #20050: URL: https://github.com/apache/doris/pull/20050 # Proposed changes  ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #20050: [Bug](hash-join) fix hash join share hash map result not stable
BiteThet commented on PR #20050: URL: https://github.com/apache/doris/pull/20050#issuecomment-1562416207 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt merged pull request #20001: [Bug](materialized-view) forbid create mv/rollup on mow table
BiteThet merged PR #20001: URL: https://github.com/apache/doris/pull/20001 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (002c76e06f -> 618961053f)
This is an automated email from the ASF dual-hosted git repository.
panxiaolei pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 002c76e06f [vectorized](udaf) support udaf function work with window
function (#19962)
add 618961053f [Bug](materialized-view) forbid create mv/rollup on mow
table (#20001)
No new revisions were added by this update.
Summary of changes:
conf/asan_suppr.conf | 1 +
.../doris/alter/MaterializedViewHandler.java | 6 ++
.../java/org/apache/doris/alter/AlterTest.java | 2 +-
.../doris/nereids/rules/mv/SelectMvIndexTest.java | 4 ++--
.../planner/MaterializedViewFunctionTest.java | 2 +-
.../{const_invalid.groovy => mow_invalid.groovy} | 24 --
.../test_uniq_mv_schema_change.groovy | 2 +-
.../test_uniq_rollup_schema_change.groovy | 2 +-
8 files changed, 22 insertions(+), 21 deletions(-)
copy regression-test/suites/mv_p0/test_mv_useless/{const_invalid.groovy =>
mow_invalid.groovy} (66%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener opened a new pull request, #20051: [fix](Nereids): memo skipProject() shouldn't skip NotEliminated project
jackwener opened a new pull request, #20051: URL: https://github.com/apache/doris/pull/20051 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 commented on pull request #20005: [Bug](runtime filter) Fix min/max filter for decimalv3
Gabriel39 commented on PR #20005: URL: https://github.com/apache/doris/pull/20005#issuecomment-1562420100 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on a diff in pull request #20051: [fix](Nereids): memo skipProject() shouldn't skip NotEliminated project
jackwener commented on code in PR #20051:
URL: https://github.com/apache/doris/pull/20051#discussion_r120586
##
fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Memo.java:
##
@@ -111,7 +111,8 @@ public Map
getGroupExpressions() {
}
private Plan skipProject(Plan plan, Group targetGroup) {
-if (plan instanceof LogicalProject) {
+// Some top project can't be eliminated
+if (plan instanceof LogicalProject && ((LogicalProject)
plan).canEliminate()) {
Review Comment:
Can't skip project whose canEliminate() is true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20050: [Bug](hash-join) fix hash join share hash map result not stable
github-actions[bot] commented on PR #20050: URL: https://github.com/apache/doris/pull/20050#issuecomment-1562421319 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Mryange commented on pull request #19811: [feature](decimal)support cast rounding half up and div precision increment in decimalv3.
Mryange commented on PR #19811: URL: https://github.com/apache/doris/pull/19811#issuecomment-1562422515 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] mrhhsg commented on a diff in pull request #20050: [Bug](hash-join) fix hash join share hash map result not stable
mrhhsg commented on code in PR #20050:
URL: https://github.com/apache/doris/pull/20050#discussion_r1205114720
##
be/src/vec/exec/join/vhash_join_node.cpp:
##
@@ -791,11 +791,10 @@ Status
HashJoinNode::_materialize_build_side(RuntimeState* state) {
RETURN_IF_ERROR(sink(state, &block, eos));
}
-RETURN_IF_ERROR(child(1)->close(state));
} else {
-RETURN_IF_ERROR(child(1)->close(state));
RETURN_IF_ERROR(sink(state, nullptr, true));
}
+RETURN_IF_ERROR(child(1)->close(state));
Review Comment:
this change will make child(1) close after `sink`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #19983: [test] test decimalv3
hello-stephen commented on PR #19983: URL: https://github.com/apache/doris/pull/19983#issuecomment-1562424718 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.17 seconds stream load tsv: 442 seconds loaded 74807831229 Bytes, about 161 MB/s stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s insert into select: 26.0 seconds inserted 1000 Rows, about 384K ops/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230525073708_clickbench_pr_150305.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20050: [Bug](hash-join) fix hash join share hash map result not stable
github-actions[bot] commented on PR #20050: URL: https://github.com/apache/doris/pull/20050#issuecomment-1562424865 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20050: [Bug](hash-join) fix hash join share hash map result not stable
github-actions[bot] commented on PR #20050: URL: https://github.com/apache/doris/pull/20050#issuecomment-1562424913 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20005: [Bug](runtime filter) Fix min/max filter for decimalv3
github-actions[bot] commented on PR #20005: URL: https://github.com/apache/doris/pull/20005#issuecomment-1562424513 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20050: [Bug](hash-join) fix hash join share hash map result not stable
github-actions[bot] commented on PR #20050: URL: https://github.com/apache/doris/pull/20050#issuecomment-1562424916 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on pull request #20051: [fix](Nereids): memo skipProject() shouldn't skip NotEliminated project
jackwener commented on PR #20051: URL: https://github.com/apache/doris/pull/20051#issuecomment-1562426394 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju commented on a diff in pull request #19973: [improvement](bitmap) Using set to store a small number of elements to improve performance
cambyzju commented on code in PR #19973: URL: https://github.com/apache/doris/pull/19973#discussion_r1205117014 ## be/src/common/config.cpp: ## @@ -998,6 +998,8 @@ DEFINE_mInt32(s3_write_buffer_whole_size, "524288000"); //disable shrink memory by default DEFINE_Bool(enable_shrink_memory, "false"); +DEFINE_Bool(enable_set_in_bitmap_value, "true"); Review Comment: we need add more comment about enable_set_in_bitmap_value and SET_TYPE_THRESHOLD. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju commented on a diff in pull request #19973: [improvement](bitmap) Using set to store a small number of elements to improve performance
cambyzju commented on code in PR #19973: URL: https://github.com/apache/doris/pull/19973#discussion_r1205117014 ## be/src/common/config.cpp: ## @@ -998,6 +998,8 @@ DEFINE_mInt32(s3_write_buffer_whole_size, "524288000"); //disable shrink memory by default DEFINE_Bool(enable_shrink_memory, "false"); +DEFINE_Bool(enable_set_in_bitmap_value, "true"); Review Comment: 1. we need add more comment about enable_set_in_bitmap_value and SET_TYPE_THRESHOLD. 2. we should better close new feature by default. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20005: [Bug](runtime filter) Fix min/max filter for decimalv3
github-actions[bot] commented on PR #20005: URL: https://github.com/apache/doris/pull/20005#issuecomment-1562428535 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19811: [feature](decimal)support cast rounding half up and div precision increment in decimalv3.
github-actions[bot] commented on PR #19811: URL: https://github.com/apache/doris/pull/19811#issuecomment-1562430171 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] amorynan commented on pull request #19543: [Refact] (serde) refact mysql serde with data type
amorynan commented on PR #19543: URL: https://github.com/apache/doris/pull/19543#issuecomment-1562432021 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt closed pull request #20050: [Bug](hash-join) fix hash join share hash map result not stable
BiteThet closed pull request #20050: [Bug](hash-join) fix hash join share hash map result not stable URL: https://github.com/apache/doris/pull/20050 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju commented on a diff in pull request #19973: [improvement](bitmap) Using set to store a small number of elements to improve performance
cambyzju commented on code in PR #19973:
URL: https://github.com/apache/doris/pull/19973#discussion_r1205124041
##
be/src/util/bitmap_value.h:
##
@@ -1959,16 +2511,31 @@ class BitmapValue {
_is_shared = false;
}
+void _convert_to_bitmap_if_need() {
+if (_type != SET || _set.size() <= SET_TYPE_THRESHOLD) {
+return;
+}
+_prepare_bitmap_for_write();
+for (auto v : _set) {
+_bitmap->add(v);
+}
+_type = BITMAP;
+_set.clear();
+}
+
enum BitmapDataType {
EMPTY = 0,
SINGLE = 1, // single element
-BITMAP = 2 // more than one elements
+SET = 2,// elements count less or equal than 32
+BITMAP = 3 // more than one elements
};
uint64_t _sv = 0; // store the single value
when _type == SINGLE
std::shared_ptr _bitmap; // used when _type == BITMAP
+phmap::flat_hash_set _set;
BitmapDataType _type {EMPTY};
// Indicate whether the state is shared among multi BitmapValue object
bool _is_shared = true;
+static constexpr uint64_t SET_TYPE_THRESHOLD = 32;
Review Comment:
why `SET_TYPE_THRESHOLD == 32` ? Is there any performance test reports?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caoliang-web commented on pull request #16625: [Fix](multi-catalog) fix oss access issue with aws s3 sdk
caoliang-web commented on PR #16625: URL: https://github.com/apache/doris/pull/16625#issuecomment-1562435637 error while editing：be/src/io/s3_writer.cpp:57:16: error: 'SetCompliantRfc3986Encoding' is not a member of 'Aws::Http' -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju commented on a diff in pull request #19973: [improvement](bitmap) Using set to store a small number of elements to improve performance
cambyzju commented on code in PR #19973:
URL: https://github.com/apache/doris/pull/19973#discussion_r1205125340
##
be/src/util/bitmap_value.h:
##
@@ -1959,16 +2511,31 @@ class BitmapValue {
_is_shared = false;
}
+void _convert_to_bitmap_if_need() {
+if (_type != SET || _set.size() <= SET_TYPE_THRESHOLD) {
+return;
+}
+_prepare_bitmap_for_write();
+for (auto v : _set) {
+_bitmap->add(v);
+}
+_type = BITMAP;
+_set.clear();
+}
+
enum BitmapDataType {
EMPTY = 0,
SINGLE = 1, // single element
-BITMAP = 2 // more than one elements
+SET = 2,// elements count less or equal than 32
Review Comment:
could we keep `BITMAP = 2` and add `SET = 3` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20031: [fix](Nereids) local sort should not translate to unpartitioned partition
hello-stephen commented on PR #20031: URL: https://github.com/apache/doris/pull/20031#issuecomment-1562438509 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.57 seconds stream load tsv: 448 seconds loaded 74807831229 Bytes, about 159 MB/s stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s insert into select: 26.0 seconds inserted 1000 Rows, about 384K ops/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230525074815_clickbench_pr_150293.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju commented on a diff in pull request #19973: [improvement](bitmap) Using set to store a small number of elements to improve performance
cambyzju commented on code in PR #19973:
URL: https://github.com/apache/doris/pull/19973#discussion_r1205130190
##
be/src/util/bitmap_value.h:
##
@@ -1742,17 +2186,43 @@ class BitmapValue {
return _sv;
case BITMAP:
return _bitmap->maximum();
+case SET:
+return _max_in_set();
default:
return 0;
}
}
uint64_t max(bool* empty) const {
-return min_or_max(empty, [&]() { return _bitmap->maximum(); });
+return min_or_max(empty, [&]() { return maximum(); });
}
uint64_t min(bool* empty) const {
-return min_or_max(empty, [&]() { return _bitmap->minimum(); });
+return min_or_max(empty, [&]() { return minimum(); });
+}
+
+uint64_t _min_in_set() const {
+uint64_t result = std::numeric_limits::max();
+if (_type == SET) {
Review Comment:
```suggestion
DCHECK(_type == SET);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #20030: [test](pipline)Adjust mem limit to 50% for regression
dataroaring merged PR #20030: URL: https://github.com/apache/doris/pull/20030 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (618961053f -> 694b8b6cd3)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 618961053f [Bug](materialized-view) forbid create mv/rollup on mow table (#20001) add 694b8b6cd3 [test](pipline) adjust mem limit to 50% (#20030) No new revisions were added by this update. Summary of changes: regression-test/pipeline/p0/conf/be.conf | 2 +- regression-test/pipeline/p1/conf/be.conf | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-spark-connector] fsilent opened a new issue, #101: [Bug] ConnectedFailedException: Connect to Doris BE{host='xxx', port=9060}failed
fsilent opened a new issue, #101: URL: https://github.com/apache/doris-spark-connector/issues/101 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version doris: master spark: 3.3.1 scala: 2.12 ### What's Wrong? doris spark connector select data failed.  ### What You Expected? select data success ### How to Reproduce? doris side: CREATE TABLE spark_connector_test_decimal (c1 int NOT NULL, c2 VARCHAR(25) NOT NULL, c3 VARCHAR(152), c4 boolean, c5 tinyint, c6 smallint, c7 bigint, c8 float, c9 double, c10 datev2, c11 datetime, c12 char, c13 largeint, c14 varchar, c15 decimalv3(15, 5) ) DUPLICATE KEY(`c1`) COMMENT "OLAP" DISTRIBUTED BY HASH(`c1`) BUCKETS 1 PROPERTIES ( "replication_num" = "1" ); insert into spark_connector_test_decimal values(1,'aaa','abc',true, 100, 3000, 10, 1234.567, 12345.678, '2022-12-01','2022-12-01 12:00:00', 'a', 20, 'g', 1000.12345); insert into spark_connector_test_decimal values(10001,'aaa','abc',false, 100, 3000, 10, 1234.567, 12345.678, '2022-12-01','2022-12-01 12:00:00', 'a', 20, 'g', 1000.12345); insert into spark_connector_test_decimal values(10002,'aaa','abc',True, 100, 3000, 10, 1234.567, 12345.678, '2022-12-01','2022-12-01 12:00:00', 'a', 20, 'g', 1000.12345); insert into spark_connector_test_decimal values(10003,'aaa','abc',False, 100, 3000, 10, 1234.567, 12345.678, '2022-12-01','2022-12-01 12:00:00', 'a', 20, 'g', 1000.12345); select * from spark_connector_test_decimal; spark side: CREATE TEMPORARY VIEW spark_doris_decimal USING doris OPTIONS( "table.identifier"="sparkconnector.spark_connector_test_decimal", "fenodes"="fe_host:8030", "user"="root", "password"="" ); select * from spark_doris_decimal; ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19543: [Refact] (serde) refact mysql serde with data type
github-actions[bot] commented on PR #19543: URL: https://github.com/apache/doris/pull/19543#issuecomment-1562444971 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #20009: [Fix](multi catalog, metadata)Init logType in ExternalCatalog while replay meta data to avoid NPE. Remove type variable in ExternaCatalog
morningman commented on PR #20009: URL: https://github.com/apache/doris/pull/20009#issuecomment-1562449748 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20009: [Fix](multi catalog, metadata)Init logType in ExternalCatalog while replay meta data to avoid NPE. Remove type variable in Extern
github-actions[bot] commented on PR #20009: URL: https://github.com/apache/doris/pull/20009#issuecomment-1562450973 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20009: [Fix](multi catalog, metadata)Init logType in ExternalCatalog while replay meta data to avoid NPE. Remove type variable in Extern
github-actions[bot] commented on PR #20009: URL: https://github.com/apache/doris/pull/20009#issuecomment-1562451091 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] weizhengte commented on pull request #20034: [regressiontest](statistics) Comment stats unstable cases
weizhengte commented on PR #20034: URL: https://github.com/apache/doris/pull/20034#issuecomment-1562452984 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19955: [improvement](multi catalog, nereids)Support collect hive table statistics by sql.
github-actions[bot] commented on PR #19955: URL: https://github.com/apache/doris/pull/19955#issuecomment-1562453340 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19955: [improvement](multi catalog, nereids)Support collect hive table statistics by sql.
github-actions[bot] commented on PR #19955: URL: https://github.com/apache/doris/pull/19955#issuecomment-1562453411 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20047: [Fix](single replica load) fix indices_size key not found core
hello-stephen commented on PR #20047: URL: https://github.com/apache/doris/pull/20047#issuecomment-1562456463 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.05 seconds stream load tsv: 445 seconds loaded 74807831229 Bytes, about 160 MB/s stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s insert into select: 25.5 seconds inserted 1000 Rows, about 392K ops/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230525075929_clickbench_pr_150312.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] sohardforaname opened a new pull request, #20052: [Feature](Nereids)support update for Nereids
sohardforaname opened a new pull request, #20052: URL: https://github.com/apache/doris/pull/20052 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20049: [Fix](Nereids) Using switch to control minidump input serialize
hello-stephen commented on PR #20049: URL: https://github.com/apache/doris/pull/20049#issuecomment-1562473239 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.42 seconds stream load tsv: 444 seconds loaded 74807831229 Bytes, about 160 MB/s stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 26.2 seconds inserted 1000 Rows, about 381K ops/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230525081042_clickbench_pr_150321.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Reminiscent opened a new pull request, #20053: [opt](Nereids) support use the string as the hint name key
Reminiscent opened a new pull request, #20053:
URL: https://github.com/apache/doris/pull/20053
## Problem summary
We can not use the string as the variable key to use in the hint.
Before this PR
```
mysql> SET enable_nereids_planner=true;
Query OK, 0 rows affected (0.01 sec)
mysql> set enable_fallback_to_original_planner=false;
Query OK, 0 rows affected (0.10 sec)
mysql> explain select /*+ SET_var("enable_nereids_planner" = "false") */ 1;
ERROR 1105 (HY000): Exception, msg: Nereids cannot parse the SQL, and
fallback disabled. caused by:
no viable alternative at input 'select /*+
SET_var("enable_nereids_planner"'(line 1, pos 27)
```
After this PR
```
mysql> SET enable_nereids_planner=true;
Query OK, 0 rows affected (0.01 sec)
mysql> set enable_fallback_to_original_planner=false;
Query OK, 0 rows affected (0.10 sec)
mysql> select /*+ SET_var("enable_nereids_planner" = "false") */ 1;
+--+
| 1|
+--+
|1 |
+--+
1 row in set (0.00 sec)
```
Describe your changes.
Support the string for the hint key in the Parser.
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [ ] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on pull request #19934: [feat](optimizer) Support materailze CTE
Kikyou1997 commented on PR #19934: URL: https://github.com/apache/doris/pull/19934#issuecomment-1562484002 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] weizhengte commented on pull request #20034: [regressiontest](statistics) Comment stats unstable cases
weizhengte commented on PR #20034: URL: https://github.com/apache/doris/pull/20034#issuecomment-1562486559 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20050: [Bug](hash-join) fix hash join share hash map result not stable
github-actions[bot] commented on PR #20050: URL: https://github.com/apache/doris/pull/20050#issuecomment-1562497742 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zenoyang commented on a diff in pull request #19973: [improvement](bitmap) Using set to store a small number of elements to improve performance
zenoyang commented on code in PR #19973:
URL: https://github.com/apache/doris/pull/19973#discussion_r1205120276
##
be/src/util/bitmap_value.h:
##
@@ -1311,6 +1382,53 @@ class BitmapValue {
case BITMAP:
_prepare_bitmap_for_write();
*_bitmap |= *rhs._bitmap;
+break;
+case SET: {
+_prepare_bitmap_for_write();
+*_bitmap = *rhs._bitmap;
+
+for (auto v : _set) {
+_bitmap->add(v);
+}
+_type = SET;
Review Comment:
Should it be `_type = BITMAP` here?
##
be/src/util/bitmap_value.h:
##
@@ -1331,37 +1450,103 @@ class BitmapValue {
case BITMAP:
bitmaps.push_back(value->_bitmap.get());
break;
+case SET:
+sets.push_back(&value->_set);
+break;
}
}
if (!bitmaps.empty()) {
_prepare_bitmap_for_write();
+*_bitmap = detail::Roaring64Map::fastunion(bitmaps.size(),
bitmaps.data());
switch (_type) {
case EMPTY:
-*_bitmap = detail::Roaring64Map::fastunion(bitmaps.size(),
bitmaps.data());
-_type = BITMAP;
break;
case SINGLE:
-*_bitmap = detail::Roaring64Map::fastunion(bitmaps.size(),
bitmaps.data());
_bitmap->add(_sv);
-_type = BITMAP;
break;
case BITMAP:
-*_bitmap |= detail::Roaring64Map::fastunion(bitmaps.size(),
bitmaps.data());
+break;
+case SET: {
+for (auto v : _set) {
+_bitmap->add(v);
+}
+_set.clear();
break;
}
+}
+_type = BITMAP;
+}
+
+if (!sets.empty()) {
+for (auto& set : sets) {
+for (auto v : *set) {
+_set.insert(v);
+}
+}
+switch (_type) {
+case EMPTY:
+_type = SET;
+break;
+case SINGLE: {
+_set.insert(_sv);
+_type = SET;
+_convert_to_bitmap_if_need();
+break;
+}
+case BITMAP:
+_prepare_bitmap_for_write();
+for (auto v : _set) {
+_bitmap->add(v);
+}
+_type = BITMAP;
+break;
+case SET: {
+_convert_to_bitmap_if_need();
+break;
+}
+}
}
if (_type == EMPTY && single_values.size() == 1) {
_sv = single_values[0];
_type = SINGLE;
} else if (!single_values.empty()) {
-_prepare_bitmap_for_write();
-_bitmap->addMany(single_values.size(), single_values.data());
-if (_type == SINGLE) {
-_bitmap->add(_sv);
+switch (_type) {
+case EMPTY:
+case SINGLE:
+if (config::enable_set_in_bitmap_value) {
+_set.insert(single_values.cbegin(), single_values.cend());
+if (_type == SINGLE) {
+_set.insert(_sv);
+}
+if (_set.size() == 1) {
+_type = SINGLE;
+_sv = *_set.begin();
+_set.clear();
+} else {
+_type = SET;
+_convert_to_bitmap_if_need();
+}
+} else {
+_prepare_bitmap_for_write();
+_bitmap->addMany(single_values.size(),
single_values.data());
+if (_type == SINGLE) {
+_bitmap->add(_sv);
+}
+_type = SET;
Review Comment:
Should it be `_type = BITMAP` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz opened a new pull request, #20054: [docs](memory) debug-tools memory part description Jemalloc
xinyiZzz opened a new pull request, #20054: URL: https://github.com/apache/doris/pull/20054 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20005: [Bug](runtime filter) Fix min/max filter for decimalv3
hello-stephen commented on PR #20005: URL: https://github.com/apache/doris/pull/20005#issuecomment-1562500970 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.15 seconds stream load tsv: 445 seconds loaded 74807831229 Bytes, about 160 MB/s stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s stream load parquet: 29 seconds loaded 861443392 Bytes, about 28 MB/s insert into select: 25.6 seconds inserted 1000 Rows, about 390K ops/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230525083058_clickbench_pr_150333.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20050: [Chore](hash-join) remove useless conditions and add some case
github-actions[bot] commented on PR #20050: URL: https://github.com/apache/doris/pull/20050#issuecomment-1562502046 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20023: [WIP]Routineload support multi table
github-actions[bot] commented on PR #20023: URL: https://github.com/apache/doris/pull/20023#issuecomment-1562509451 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20034: [regressiontest](statistics) Comment stats unstable cases
github-actions[bot] commented on PR #20034: URL: https://github.com/apache/doris/pull/20034#issuecomment-1562510022 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20034: [regressiontest](statistics) Comment stats unstable cases
github-actions[bot] commented on PR #20034: URL: https://github.com/apache/doris/pull/20034#issuecomment-1562509957 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] DongLiang-0 commented on pull request #19909: [Enhancement](hudi) Support hudi mor only java side ,be side not support
DongLiang-0 commented on PR #19909: URL: https://github.com/apache/doris/pull/19909#issuecomment-1562514340 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20054: [docs](memory) debug-tools memory part description Jemalloc
github-actions[bot] commented on PR #20054: URL: https://github.com/apache/doris/pull/20054#issuecomment-1562517241 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20054: [docs](memory) debug-tools memory part description Jemalloc
github-actions[bot] commented on PR #20054: URL: https://github.com/apache/doris/pull/20054#issuecomment-1562517297 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] qidaye commented on pull request #19868: [opt](compaction) add pick rowset to compact interval config
qidaye commented on PR #19868: URL: https://github.com/apache/doris/pull/19868#issuecomment-1562520848 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #19811: [feature](decimal)support cast rounding half up and div precision increment in decimalv3.
hello-stephen commented on PR #19811: URL: https://github.com/apache/doris/pull/19811#issuecomment-1562520881 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.65 seconds stream load tsv: 446 seconds loaded 74807831229 Bytes, about 159 MB/s stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s insert into select: 25.7 seconds inserted 1000 Rows, about 389K ops/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230525084550_clickbench_pr_150339.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #19969: [Feature](agg_state) support agg_state combinators
BiteThet commented on PR #19969: URL: https://github.com/apache/doris/pull/19969#issuecomment-1562526006 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju commented on a diff in pull request #19973: [improvement](bitmap) Using set to store a small number of elements to improve performance
cambyzju commented on code in PR #19973:
URL: https://github.com/apache/doris/pull/19973#discussion_r1205201471
##
be/src/util/bitmap_value.h:
##
@@ -1909,14 +2446,28 @@ class BitmapValue {
void _convert_to_smaller_type() {
if (_type == BITMAP) {
uint64_t c = _bitmap->cardinality();
-if (c > 1) return;
+if (config::enable_set_in_bitmap_value && c > SET_TYPE_THRESHOLD) {
+return;
+} else if (c > 1) {
+return;
+}
if (c == 0) {
_type = EMPTY;
-} else {
+} else if (c == 1) {
_type = SINGLE;
_sv = _bitmap->minimum();
+} else {
+_type = SET;
+for (auto v : *_bitmap) {
+_set.insert(v);
+}
}
_bitmap.reset();
+} else if (_type == SET) {
+if (_set.size() == 1) {
+_type = SINGLE;
+_sv = *_set.begin();
+}
Review Comment:
else if set.size == 0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt merged pull request #20040: [fix](revert) data stream sender stop sending data to receiver if it returns eos early (#19847)"
BiteThet merged PR #20040: URL: https://github.com/apache/doris/pull/20040 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (694b8b6cd3 -> 3598518e59)
This is an automated email from the ASF dual-hosted git repository. panxiaolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 694b8b6cd3 [test](pipline) adjust mem limit to 50% (#20030) add 3598518e59 [fix](revert) data stream sender stop sending data to receiver if it returns eos early (#19847)" (#20040) No new revisions were added by this update. Summary of changes: be/src/pipeline/exec/exchange_sink_buffer.cpp | 24 +- be/src/pipeline/exec/exchange_sink_buffer.h | 3 - be/src/pipeline/exec/operator.h | 7 +- be/src/vec/runtime/vdata_stream_mgr.cpp | 6 +- be/src/vec/sink/vdata_stream_sender.cpp | 110 +++--- be/src/vec/sink/vdata_stream_sender.h | 47 +++ be/src/vec/sink/vresult_file_sink.cpp | 5 +- 7 files changed, 44 insertions(+), 158 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19868: [opt](compaction) add pick rowset to compact interval config
github-actions[bot] commented on PR #19868: URL: https://github.com/apache/doris/pull/19868#issuecomment-1562527448 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19969: [Feature](agg_state) support agg_state combinators
github-actions[bot] commented on PR #19969: URL: https://github.com/apache/doris/pull/19969#issuecomment-1562530378 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on pull request #19063: [Enhancement](alter inverted index) Improve alter inverted index performance with light weight add or drop inverted index
Tanya-W commented on PR #19063: URL: https://github.com/apache/doris/pull/19063#issuecomment-1562531317 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20051: [fix](Nereids): memo skipProject() shouldn't skip NotEliminated project
hello-stephen commented on PR #20051: URL: https://github.com/apache/doris/pull/20051#issuecomment-1562531808 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.59 seconds stream load tsv: 443 seconds loaded 74807831229 Bytes, about 161 MB/s stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s insert into select: 25.5 seconds inserted 1000 Rows, about 392K ops/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230525085328_clickbench_pr_150346.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju commented on a diff in pull request #19973: [improvement](bitmap) Using set to store a small number of elements to improve performance
cambyzju commented on code in PR #19973:
URL: https://github.com/apache/doris/pull/19973#discussion_r1205206078
##
be/src/util/bitmap_value.h:
##
@@ -1828,6 +2309,27 @@ class BitmapValue {
}
return count;
}
+case SET: {
+int64_t count = 0;
+std::vector values;
Review Comment:
reserve vector for performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on pull request #19063: [Enhancement](alter inverted index) Improve alter inverted index performance with light weight add or drop inverted index
Tanya-W commented on PR #19063: URL: https://github.com/apache/doris/pull/19063#issuecomment-1562534537 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20006: [wip][feature](load) support single-stream-multi-table
github-actions[bot] commented on PR #20006: URL: https://github.com/apache/doris/pull/20006#issuecomment-1562534548 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] qidaye commented on pull request #19868: [opt](compaction) add pick rowset to compact interval config
qidaye commented on PR #19868: URL: https://github.com/apache/doris/pull/19868#issuecomment-1562535407 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on a diff in pull request #20048: [feature](executor)Support query queue
xinyiZzz commented on code in PR #20048:
URL: https://github.com/apache/doris/pull/20048#discussion_r1205209386
##
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java:
##
@@ -885,6 +891,18 @@ public void analyze(TQueryOptions tQueryOptions) throws
UserException {
if (parsedStmt instanceof QueryStmt) {
queryStmt = (QueryStmt) parsedStmt;
queryStmt.getTables(analyzer, false, tableMap,
parentViewNameSet);
+
+// queue query here
+if (Config.enable_resource_group && Config.enable_query_queue)
{
+this.queryQueue = analyzer.getEnv().getResourceGroupMgr()
+
.getResourceGroupQueryQueue(context.sessionVariable.resourceGroup);
+try {
+queryQueue.offer();
Review Comment:
Stuck here, will the user be reminded, such as "query is in queue"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju commented on a diff in pull request #19973: [improvement](bitmap) Using set to store a small number of elements to improve performance
cambyzju commented on code in PR #19973:
URL: https://github.com/apache/doris/pull/19973#discussion_r1205209738
##
be/src/util/bitmap_value.h:
##
@@ -1828,6 +2309,27 @@ class BitmapValue {
}
return count;
}
+case SET: {
+int64_t count = 0;
+std::vector values;
+for (auto v : _set) {
+values.emplace_back(v);
Review Comment:
only push back values which `>= range_start`, to reduce sort elements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zclllyybb commented on pull request #19984: [fix](expr) Make VExprContext exit gracefully
zclllyybb commented on PR #19984: URL: https://github.com/apache/doris/pull/19984#issuecomment-1562535597 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] eldenmoon opened a new pull request, #20055: [Bug](point query) check point query before check two phase read
eldenmoon opened a new pull request, #20055: URL: https://github.com/apache/doris/pull/20055 1. checkEnableTwoPhaseRead rely on thr short circuit flag 2. add more metric to display lookup profile # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on a diff in pull request #20048: [feature](executor)Support query queue
xinyiZzz commented on code in PR #20048:
URL: https://github.com/apache/doris/pull/20048#discussion_r1205211563
##
fe/fe-core/src/main/java/org/apache/doris/resource/resourcegroup/QueryQueue.java:
##
@@ -0,0 +1,103 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.resource.resourcegroup;
+
+import com.google.common.base.Preconditions;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.ReentrantLock;
+
+// note(wb) refer java BlockingQueue, but support altering capacity
+// todo(wb) add wait time to profile
+public class QueryQueue {
+
+private static final Logger LOG = LogManager.getLogger(QueryQueue.class);
+// note(wb) use fair strategy for FIFO
+// but maybe display lower overall throughput
+// need performance test later
+private final ReentrantLock queueLock = new ReentrantLock(true);
+private final Condition queueLockCond = queueLock.newCondition();
+// resource group property
+private int maxRunningQueryNum;
+private int maxWaitingQueryNum;
+private int queryWaitTimeout; // ms
+// running property
+private int currentRunningQueryNum;
+private int currentWaitingQueryNum;
+
+public QueryQueue(int maxRunningQueryNum, int maxWaitingQueryNum, int
queryWaitTimeout) {
+this.maxRunningQueryNum = maxRunningQueryNum;
+this.maxWaitingQueryNum = maxWaitingQueryNum;
+this.queryWaitTimeout = queryWaitTimeout;
+}
+
+public void offer() throws InterruptedException {
+// to prevent hang
+// the lock shouldn't be hold for too long
+// we should catch the case when it happens
+queueLock.tryLock(5, TimeUnit.SECONDS);
+try {
+// currentRunningQueryNum may bigger than maxRunningQueryNum
+// because maxRunningQueryNum can be altered
+if (currentRunningQueryNum >= maxRunningQueryNum) {
+if (currentWaitingQueryNum >= maxWaitingQueryNum) {
+throw new QueryQueueException(
+"query waiting queue is full, queue length=" +
maxWaitingQueryNum);
+}
+
+currentWaitingQueryNum++;
+boolean ret = queueLockCond.await(queryWaitTimeout,
TimeUnit.MILLISECONDS);
+currentWaitingQueryNum--;
+if (!ret) {
+throw new QueryQueueException("query wait timeout " +
queryWaitTimeout + " ms");
+}
+}
+currentRunningQueryNum++;
+} finally {
+queueLock.unlock();
+}
+}
+
+public void poll() throws InterruptedException {
+queueLock.tryLock(5, TimeUnit.SECONDS);
+try {
+currentRunningQueryNum--;
Review Comment:
Whether to print the log fe.log regularly, print `currentRunningQueryNum`
and `currentWaitingQueryNum`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19063: [Enhancement](alter inverted index) Improve alter inverted index performance with light weight add or drop inverted index
github-actions[bot] commented on PR #19063: URL: https://github.com/apache/doris/pull/19063#issuecomment-1562539395 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
