[GitHub] [doris] luozenglin opened a new issue, #12711: [Bug] intersect query failed
luozenglin opened a new issue, #12711: URL: https://github.com/apache/doris/issues/12711 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? ``` mysql> set enable_vectorized_engine=false; Query OK, 0 rows affected (0.04 sec) mysql> SELECT * FROM (SELECT k2 FROM test_query_qa.test INTERSECT SELECT k2 FROM test_query_qa.baseall) a ORDER BY k2; ERROR 1105 (HY000): errCode = 2, detailMessage = failed to get tuple idx with tuple id: 0 ``` ### What You Expected? success ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] luozenglin opened a new pull request, #12712: [fix](intersect) fix intersect query failed in row storage code
luozenglin opened a new pull request, #12712: URL: https://github.com/apache/doris/pull/12712 # Proposed changes Issue Number: close #12711 ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #12286: [enhancement](test) add ssb_sf100 to p2 cases
morningman merged PR #12286: URL: https://github.com/apache/doris/pull/12286 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12290: [enhancement](tracing) Support forward to master tracing
github-actions[bot] commented on PR #12290: URL: https://github.com/apache/doris/pull/12290#issuecomment-1250230470 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12290: [enhancement](tracing) Support forward to master tracing
github-actions[bot] commented on PR #12290: URL: https://github.com/apache/doris/pull/12290#issuecomment-1250230480 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #12243: [enhancement](test) add some p0 cases
morningman merged PR #12243: URL: https://github.com/apache/doris/pull/12243 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhancement](regression-test) add some p0 cases (#12243)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new e9f105aa1e [enhancement](regression-test) add some p0 cases (#12243) e9f105aa1e is described below commit e9f105aa1e69f8cae2d5c82bcd7013adf75ec21f Author: Yongqiang YANG <98214048+dataroar...@users.noreply.github.com> AuthorDate: Sun Sep 18 17:36:08 2022 +0800 [enhancement](regression-test) add some p0 cases (#12243) --- regression-test/data/insert_p0/datetype.csv| 16 ++ regression-test/data/insert_p0/insert.out | 204 + .../query_p0/limit/sql/predicateNotProjected.out | 6 + .../limit/sql/useRestrictiveLimitFromSubq.out | 4 + .../data/query_p0/limit/sql/withGroupBy.out| 5 + .../limit/sql/withGroupByAndOrderByInSubq.out | 5 + .../data/query_p0/limit/sql/withGroupByInSubq.out | 4 + .../data/query_p0/limit/sql/withJoinInSubq.out | 4 + .../data/query_p0/limit/sql/withLimitZero.out | 4 + .../data/query_p0/limit/sql/withSimpleLimit.out| 8 + .../data/query_p0/limit/sql/withSubq.out | 4 + .../select_no_from/sql/projectConstants.out| 4 + .../select_no_from/sql/projectFunctions.out| 4 + .../query_p0/select_no_from/sql/withUnionAll.out | 7 + .../query_p0/set_operations/sql/except_1_order.out | 6 + .../query_p0/set_operations/sql/except_2_order.out | 6 + .../query_p0/set_operations/sql/except_3_order.out | 8 + .../set_operations/sql/intersect_1_order.out | 9 + .../set_operations/sql/intersect_2_order.out | 10 + regression-test/suites/insert_p0/ddl/datatype.sql | 15 ++ .../suites/insert_p0/ddl/mutable_datatype.sql | 15 ++ regression-test/suites/insert_p0/insert.groovy | 85 + .../suites/query_p0/aggregate/aggregate.groovy | 1 + .../suites/query_p0/limit/ddl/lineitem.sql | 18 ++ .../suites/query_p0/limit/ddl/nation.sql | 12 ++ regression-test/suites/query_p0/limit/ddl/part.sql | 11 ++ .../suites/query_p0/limit/ddl/partsupp.sql | 7 + .../suites/query_p0/limit/ddl/region.sql | 10 + .../suites/query_p0/limit/ddl/supplier.sql | 9 + .../suites/query_p0/limit/ddl/workers.sql | 10 + regression-test/suites/query_p0/limit/load.groovy | 63 +++ .../query_p0/limit/sql/predicateNotProjected.sql | 3 + .../limit/sql/useRestrictiveLimitFromSubq.sql | 2 + .../suites/query_p0/limit/sql/withGroupBy.sql | 4 + .../limit/sql/withGroupByAndOrderByInSubq.sql | 4 + .../query_p0/limit/sql/withGroupByInSubq.sql | 4 + .../suites/query_p0/limit/sql/withJoinInSubq.sql | 1 + .../suites/query_p0/limit/sql/withLimitZero.sql| 3 + .../suites/query_p0/limit/sql/withSimpleLimit.sql | 2 + .../suites/query_p0/limit/sql/withSubq.sql | 2 + .../select_no_from/sql/projectAggFuncs.sql | 4 + .../select_no_from/sql/projectConstants.sql| 2 + .../select_no_from/sql/projectFunctions.sql| 2 + .../select_no_from/sql/withGroupOrderHaving.sql| 4 + .../query_p0/select_no_from/sql/withUnionAll.sql | 8 + .../select_no_from/sql/withWhereClause.sql | 4 + .../query_p0/select_no_from/sql/withWhereFalse.sql | 4 + 47 files changed, 627 insertions(+) diff --git a/regression-test/data/insert_p0/datetype.csv b/regression-test/data/insert_p0/datetype.csv new file mode 100644 index 00..a57e5ca7d1 --- /dev/null +++ b/regression-test/data/insert_p0/datetype.csv @@ -0,0 +1,16 @@ +12|12.25|String1|1999-01-08|1999-01-08 02:05:06|TRUE|123.22|12345678901234567890.0123456789 +25|55.52|test|1952-01-05|1989-01-08 04:05:06|FALSE|321.21|-12345678901234567890.0123456789 +964|0.245|Again|1936-02-08|2005-01-09 04:05:06|FALSE|333.82|98765432109876543210.9876543210 +100|12.25|testing|1949-07-08|2002-01-07 01:05:06|TRUE|-393.22|-98765432109876543210.9876543210 +100|99.8777|AGAIN|1987-04-09|2010-01-02 04:03:06|TRUE|000.00|.00 +5252|12.25|sample|1987-04-09|2010-01-02 04:03:06|TRUE|123.00|0001.00 +100|9.8777|STRING1|1923-04-08|2010-01-02 05:09:06|TRUE|010.01|0002.00 +8996|98.8777|again|1987-04-09|2010-01-02 04:03:06|FALSE|-000.01|.99 +100|12.8788|string1|1922-04-02|2010-01-02 02:05:06|TRUE|999.99|-.99 +5748|67.87|sample|1987-04-06|2010-01-02 04:03:06|TRUE|-999.99|.01 +5748|67.87|Sample|1987-04-06|2010-01-02 04:03:06|TRUE|181.18|-.01 +5748|67.87|sample|1987-04-06|2010-01-02 04:03:06|TRUE|181.18|12345678901234567890.0123456789 +5748|67.87|sample|1987-04-06|2010-01-02 04:03:06|TRUE|181.18|12345678901234567890.0123456789 +5000|67.87|testing|\N|2010-01-02 04:03:0
[GitHub] [doris] morningman merged pull request #12290: [enhancement](tracing) Support forward to master tracing
morningman merged PR #12290: URL: https://github.com/apache/doris/pull/12290 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhancement](tracing) Support forward to master tracing (#12290)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 4f98146e83 [enhancement](tracing) Support forward to master tracing
(#12290)
4f98146e83 is described below
commit 4f98146e83e55f554604c3a99efc27f816c7130b
Author: abmdocrt
AuthorDate: Sun Sep 18 17:39:04 2022 +0800
[enhancement](tracing) Support forward to master tracing (#12290)
---
.../java/org/apache/doris/qe/ConnectProcessor.java | 51 --
.../java/org/apache/doris/qe/MasterOpExecutor.java | 26 ++-
.../java/org/apache/doris/qe/StmtExecutor.java | 46 ++-
gensrc/thrift/FrontendService.thrift | 1 +
4 files changed, 95 insertions(+), 29 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
index 7920ae6dae..3db4b9e036 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
@@ -38,6 +38,7 @@ import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.ErrorReport;
import org.apache.doris.common.Pair;
import org.apache.doris.common.UserException;
+import org.apache.doris.common.telemetry.Telemetry;
import org.apache.doris.common.util.DebugUtil;
import org.apache.doris.common.util.SqlParserUtils;
import org.apache.doris.datasource.CatalogIf;
@@ -62,8 +63,10 @@ import com.google.common.base.Strings;
import com.google.common.collect.Lists;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.SpanContext;
+import io.opentelemetry.api.trace.SpanKind;
import io.opentelemetry.context.Context;
import io.opentelemetry.context.Scope;
+import io.opentelemetry.context.propagation.TextMapGetter;
import org.apache.commons.codec.digest.DigestUtils;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -73,7 +76,9 @@ import java.io.StringReader;
import java.nio.ByteBuffer;
import java.nio.channels.AsynchronousCloseException;
import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
import java.util.UUID;
/**
@@ -81,10 +86,23 @@ import java.util.UUID;
*/
public class ConnectProcessor {
private static final Logger LOG =
LogManager.getLogger(ConnectProcessor.class);
+private static final TextMapGetter> getter =
+new TextMapGetter>() {
+@Override
+public Iterable keys(Map carrier) {
+return carrier.keySet();
+}
+@Override
+public String get(Map carrier, String key) {
+if (carrier.containsKey(key)) {
+return carrier.get(key);
+}
+return "";
+}
+};
private final ConnectContext ctx;
private ByteBuffer packetBuf;
-
private StmtExecutor executor = null;
public ConnectProcessor(ConnectContext context) {
@@ -473,8 +491,8 @@ public class ConnectProcessor {
// explain query stmt do not have profile
if (executor != null && !executor.getParsedStmt().isExplain()
&& (executor.getParsedStmt() instanceof QueryStmt // currently
only QueryStmt and insert need profile
-|| executor.getParsedStmt() instanceof LogicalPlanAdapter
-|| executor.getParsedStmt() instanceof InsertStmt)) {
+|| executor.getParsedStmt() instanceof LogicalPlanAdapter
+|| executor.getParsedStmt() instanceof InsertStmt)) {
executor.writeProfile(true);
}
}
@@ -543,6 +561,21 @@ public class ConnectProcessor {
}
}
+Map traceCarrier = new HashMap<>();
+if (request.isSetTraceCarrier()) {
+traceCarrier = request.getTraceCarrier();
+}
+Context extractedContext =
Telemetry.getOpenTelemetry().getPropagators().getTextMapPropagator()
+.extract(Context.current(), traceCarrier, getter);
+// What we want is for the Traceid to remain unchanged during
propagation.
+// ctx.initTracer() will be called only if the Context is valid,
+// so that the Traceid generated by SDKTracer is the same as the
follower. Otherwise,
+// if the Context is invalid and ctx.initTracer() is called,
+// SDKTracer will generate a different Traceid.
+if (Span.fromContext(extractedContext).getSpanContext().isValid()) {
+ctx.initTracer("master trace");
+}
+
ctx.setThreadLocalInfo();
StmtExecutor executor = null;
try {
@@ -557,7 +590,17 @@ public class ConnectProcessor {
[GitHub] [doris] morningman merged pull request #11856: [fix](tracing) Fix opentelemetry log output to be.out
morningman merged PR #11856: URL: https://github.com/apache/doris/pull/11856 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](tracing) Fix opentelemetry log output to be.out (#11856)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new cb06e67fba [fix](tracing) Fix opentelemetry log output to be.out
(#11856)
cb06e67fba is described below
commit cb06e67fbaceb63884f50d9d0f0661f12a550b2a
Author: luozenglin <37725793+luozeng...@users.noreply.github.com>
AuthorDate: Sun Sep 18 17:40:23 2022 +0800
[fix](tracing) Fix opentelemetry log output to be.out (#11856)
---
be/src/util/telemetry/telemetry.cpp | 18 ++
thirdparty/patches/opentelemetry-cpp-1.4.0.patch | 9 +
2 files changed, 23 insertions(+), 4 deletions(-)
diff --git a/be/src/util/telemetry/telemetry.cpp
b/be/src/util/telemetry/telemetry.cpp
index b6301862a4..44904355f4 100644
--- a/be/src/util/telemetry/telemetry.cpp
+++ b/be/src/util/telemetry/telemetry.cpp
@@ -26,6 +26,8 @@
#include "opentelemetry/exporters/otlp/otlp_http_exporter.h"
#include "opentelemetry/exporters/zipkin/zipkin_exporter.h"
#include "opentelemetry/nostd/shared_ptr.h"
+#include "opentelemetry/sdk/common/attribute_utils.h"
+#include "opentelemetry/sdk/common/global_log_handler.h"
#include "opentelemetry/sdk/trace/batch_span_processor.h"
#include "opentelemetry/sdk/trace/tracer_provider.h"
#include "opentelemetry/trace/propagation/http_trace_context.h"
@@ -38,6 +40,18 @@ namespace zipkin = opentelemetry::exporter::zipkin;
namespace resource = opentelemetry::sdk::resource;
namespace propagation = opentelemetry::context::propagation;
namespace otlp = opentelemetry::exporter::otlp;
+namespace internal_log = opentelemetry::sdk::common::internal_log;
+
+class OpenTelemetryLogHandler : public internal_log::LogHandler {
+public:
+void Handle(internal_log::LogLevel level, const char* file, int line,
const char* msg,
+const opentelemetry::sdk::common::AttributeMap& attributes)
noexcept override {
+if ((level == internal_log::LogLevel::Error || level ==
internal_log::LogLevel::Warning) &&
+file != nullptr && msg != nullptr) {
+LOG(WARNING) << fmt::format("OpenTelemetry File: {}:{} {}", file,
line, msg);
+}
+}
+};
void doris::telemetry::init_tracer() {
if (!doris::config::enable_tracing) {
@@ -82,4 +96,8 @@ void doris::telemetry::init_tracer() {
propagation::GlobalTextMapPropagator::SetGlobalPropagator(
nostd::shared_ptr(
new
opentelemetry::trace::propagation::HttpTraceContext()));
+
+// Output OpenTelemetry logs by glog
+internal_log::GlobalLogHandler::SetLogHandler(
+nostd::shared_ptr(new
OpenTelemetryLogHandler()));
}
diff --git a/thirdparty/patches/opentelemetry-cpp-1.4.0.patch
b/thirdparty/patches/opentelemetry-cpp-1.4.0.patch
index 6317c60bf3..47b16b6928 100644
--- a/thirdparty/patches/opentelemetry-cpp-1.4.0.patch
+++ b/thirdparty/patches/opentelemetry-cpp-1.4.0.patch
@@ -301,14 +301,15 @@ index ecd6dab..1aaa220 100644
} // namespace sdk
diff --git a/sdk/src/common/threadlocal.cc b/sdk/src/common/threadlocal.cc
new file mode 100644
-index 000..1f8b6b1
+index 000..ec2038b
--- /dev/null
+++ b/sdk/src/common/threadlocal.cc
-@@ -0,0 +1,82 @@
+@@ -0,0 +1,83 @@
+// Copyright The OpenTelemetry Authors
+// SPDX-License-Identifier: Apache-2.0
+
+#include
++#include "opentelemetry/sdk/common/global_log_handler.h"
+
+#include
+#include
@@ -360,7 +361,7 @@ index 000..1f8b6b1
+std::stringstream ss;
+ss << "[thread local] pthread_key_create() failed, cannot add destructor
to thread: "
+ << "error " << ret;
-+fprintf(stderr, "%s\n", ss.str().c_str());
++OTEL_INTERNAL_LOG_ERROR(ss.str());
+ }
+}
+
@@ -382,7 +383,7 @@ index 000..1f8b6b1
+std::stringstream ss;
+ss << "[thread local] pthread_setspecific() failed, cannot update
destructor list: "
+ << "error " << ret;
-+fprintf(stderr, "%s\n", ss.str().c_str());
++OTEL_INTERNAL_LOG_ERROR(ss.str());
+ }
+}
+} // namespace internal_threadlocal
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11836: [Enhancement](meta) add err type/code/msg in audit event
github-actions[bot] commented on PR #11836: URL: https://github.com/apache/doris/pull/11836#issuecomment-1250231601 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11836: [Enhancement](meta) add err type/code/msg in audit event
github-actions[bot] commented on PR #11836: URL: https://github.com/apache/doris/pull/11836#issuecomment-1250231613 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #11793: [fix](broker-load) Correction of kerberos authentication time determination rule
morningman merged PR #11793: URL: https://github.com/apache/doris/pull/11793 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](broker-load) Correction of kerberos authentication time determination rule (#11793)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 00dda79735 [fix](broker-load) Correction of kerberos authentication
time determination rule (#11793)
00dda79735 is described below
commit 00dda79735bff74304987007e547ff7585296811
Author: HB
AuthorDate: Sun Sep 18 17:46:13 2022 +0800
[fix](broker-load) Correction of kerberos authentication time determination
rule (#11793)
Every time a new broker load comes in, Doris will update the start time of
Kerberos authentication,
but this logic is wrong.
Because the authentication duration of Kerberos is calculated from the
moment when the ticket is obtained.
This PR change the logic:
1. If it is kerberos, check fs expiration by create time.
2.Otherwise, check fs expiration by access time
---
.../conf/apache_hdfs_broker.conf | 2 +-
.../org/apache/doris/broker/hdfs/BrokerConfig.java | 12 +--
.../apache/doris/broker/hdfs/BrokerFileSystem.java | 34 ---
.../doris/broker/hdfs/ClientContextManager.java| 78 +++
.../doris/broker/hdfs/FileSystemManager.java | 108 -
5 files changed, 127 insertions(+), 107 deletions(-)
diff --git a/fs_brokers/apache_hdfs_broker/conf/apache_hdfs_broker.conf
b/fs_brokers/apache_hdfs_broker/conf/apache_hdfs_broker.conf
index 8466de02cd..e023067cf0 100644
--- a/fs_brokers/apache_hdfs_broker/conf/apache_hdfs_broker.conf
+++ b/fs_brokers/apache_hdfs_broker/conf/apache_hdfs_broker.conf
@@ -27,7 +27,7 @@
broker_ipc_port = 8000
# client session will be deleted if not receive ping after this time
-client_expire_seconds = 1800
+client_expire_seconds = 3600
# Advanced configurations
# sys_log_dir = ${BROKER_HOME}/log
diff --git
a/fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/BrokerConfig.java
b/fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/BrokerConfig.java
index 6381f16db2..aa91b9d393 100644
---
a/fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/BrokerConfig.java
+++
b/fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/BrokerConfig.java
@@ -21,19 +21,19 @@ import org.apache.doris.common.ConfigBase;
public class BrokerConfig extends ConfigBase {
-
+
@ConfField
public static int hdfs_read_buffer_size_kb = 1024;
-
+
@ConfField
public static int hdfs_write_buffer_size_kb = 1024;
-
+
@ConfField
-public static int client_expire_seconds = 1800;
-
+public static int client_expire_seconds = 3600;
+
@ConfField
public static int broker_ipc_port = 8000;
-
+
@ConfField
public static String sys_log_dir = System.getenv("BROKER_HOME") + "/log";
@ConfField
diff --git
a/fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/BrokerFileSystem.java
b/fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/BrokerFileSystem.java
index 7b9f6cc679..c8a217d205 100644
---
a/fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/BrokerFileSystem.java
+++
b/fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/BrokerFileSystem.java
@@ -27,26 +27,29 @@ public class BrokerFileSystem {
private static Logger logger = Logger
.getLogger(BrokerFileSystem.class.getName());
-
+
private ReentrantLock lock;
private FileSystemIdentity identity;
private FileSystem dfsFileSystem;
private long lastAccessTimestamp;
+private long createTimestamp;
private UUID fileSystemId;
-
+
public BrokerFileSystem(FileSystemIdentity identity) {
this.identity = identity;
this.lock = new ReentrantLock();
this.dfsFileSystem = null;
this.lastAccessTimestamp = System.currentTimeMillis();
+this.createTimestamp = System.currentTimeMillis();
this.fileSystemId = UUID.randomUUID();
}
-
+
public synchronized void setFileSystem(FileSystem fileSystem) {
this.dfsFileSystem = fileSystem;
this.lastAccessTimestamp = System.currentTimeMillis();
+this.createTimestamp = System.currentTimeMillis();
}
-
+
public void closeFileSystem() {
lock.lock();
try {
@@ -63,31 +66,32 @@ public class BrokerFileSystem {
lock.unlock();
}
}
-
+
public FileSystem getDFSFileSystem() {
this.lastAccessTimestamp = System.currentTimeMillis();
return dfsFileSystem;
}
-
+
public void updateLastUpdateAccessTime() {
this.lastAccessTimestamp = System.currentTimeMillis();
}
-
+
public FileSystemIdentity getIdentity() {
return identity;
}
-
+
public ReentrantLock getLock() {
[GitHub] [doris] morningman closed pull request #11535: [improvement](olapscan) Simplify data block queue of VOlapScanNode
morningman closed pull request #11535: [improvement](olapscan) Simplify data block queue of VOlapScanNode URL: https://github.com/apache/doris/pull/11535 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman closed pull request #11060: [improvement] Simplify scan node logic: removing _materialized_blocks…
morningman closed pull request #11060: [improvement] Simplify scan node logic: removing _materialized_blocks… URL: https://github.com/apache/doris/pull/11060 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman closed pull request #10440: upgrade brpc to 1.1.0 and applying a patch
morningman closed pull request #10440: upgrade brpc to 1.1.0 and applying a patch URL: https://github.com/apache/doris/pull/10440 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman closed pull request #10437: [fix](vectorized) Support outer join for vectorized exec engine
morningman closed pull request #10437: [fix](vectorized) Support outer join for vectorized exec engine URL: https://github.com/apache/doris/pull/10437 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12710: [refactor](SimpleScheduler) refactor code for getAvailable backend in SimpleScheduler
github-actions[bot] commented on PR #12710: URL: https://github.com/apache/doris/pull/12710#issuecomment-1250232908 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12710: [refactor](SimpleScheduler) refactor code for getAvailable backend in SimpleScheduler
github-actions[bot] commented on PR #12710: URL: https://github.com/apache/doris/pull/12710#issuecomment-1250232919 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] luozenglin opened a new pull request, #12713: [docs](function) add a series of date function documents
luozenglin opened a new pull request, #12713: URL: https://github.com/apache/doris/pull/12713 add docs for `hours_add`, `hours_sub`, `minutes_add`, `minutes_sub`, `seconds_add`, `seconds_sub`, `years_sub`, `years_add`, `months_add`, `months_sub`, `days_add`, `days_add`, `weeks_add`, `weeks_sub` functions. # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] weizhengte opened a new pull request, #12714: [feature-wip](statistics) Template for building internal query SQL statements.
weizhengte opened a new pull request, #12714:
URL: https://github.com/apache/doris/pull/12714
# Proposed changes
Template for building internal query SQL statements，it mainly used for
statistics module. After the template is defined, the executable statement will
be built after the given parameters.
e.g.:
template and parameters:
- template: `SELECT ${col} FROM ${table} WHERE id = ${id};`,
- parameters: `{col=colName, table=tableName, id=1}`
- result sql: `SELECT colName FROM tableName WHERE id = 1;`
Issue Number: close #xxx
## Problem summary
Describe your changes.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #12715: [improvement](new-scan) graceful quit scanner scheduler
morningman opened a new pull request, #12715: URL: https://github.com/apache/doris/pull/12715 # Proposed changes Issue Number: close #xxx ## Problem summary Now, BE process can quit by using `kill -15` ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman closed pull request #12649: [fix](broker-load) fix npe when executing broker load
morningman closed pull request #12649: [fix](broker-load) fix npe when executing broker load URL: https://github.com/apache/doris/pull/12649 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #12710: [refactor](SimpleScheduler) refactor code for getting available backend in SimpleScheduler
morningman commented on PR #12710: URL: https://github.com/apache/doris/pull/12710#issuecomment-1250313753 Is there any behavior change? Or just a refactor? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] luozenglin commented on pull request #12713: [docs](function) add a series of date function documents
luozenglin commented on PR #12713: URL: https://github.com/apache/doris/pull/12713#issuecomment-1250322400 > I think if you add new function documents, you need to add the path of these documents according to the title of the file in `sidebars.json`, so that they can be displayed on the official website. done. Thank you for reminding me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen opened a new pull request, #12716: [Enhancement](load) Refine the load channel flush policy on mem limit (WIP)
zhannngchen opened a new pull request, #12716: URL: https://github.com/apache/doris/pull/12716 # Proposed changes Issue Number: close #xxx ## Problem summary Testing, will complete the description later. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12713: [docs](function) add a series of date function documents
github-actions[bot] commented on PR #12713: URL: https://github.com/apache/doris/pull/12713#issuecomment-1250335727 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] closed pull request #8473: [Audit][Transaction] Support audit for stream load 2PC
github-actions[bot] closed pull request #8473: [Audit][Transaction] Support audit for stream load 2PC URL: https://github.com/apache/doris/pull/8473 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary commented on pull request #12648: [fix](LOAD statement): fix bug for `toSql` func of LoadStmt.
stalary commented on PR #12648: URL: https://github.com/apache/doris/pull/12648#issuecomment-1250421266 Please check ut. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12644: [fix](colocation) fix decommission failure with 2 BEs and colocation table
yiguolei merged PR #12644: URL: https://github.com/apache/doris/pull/12644 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](colocation) fix decommission failure with 2 BEs and colocation table (#12644)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a4ed023bad [fix](colocation) fix decommission failure with 2 BEs and
colocation table (#12644)
a4ed023bad is described below
commit a4ed023bade9d011d9fdf86457deb672e5318d56
Author: Mingyu Chen
AuthorDate: Mon Sep 19 08:34:50 2022 +0800
[fix](colocation) fix decommission failure with 2 BEs and colocation table
(#12644)
This PR fix:
2 Backends.
Create tables with colocation group, 1 replica.
Decommission one of Backends.
The tablet on decommissioned Backend is not reduced.
This is a bug of ColocateTableCheckerAndBalancer.
---
fe/fe-core/src/main/java/org/apache/doris/alter/SystemHandler.java | 2 +-
.../java/org/apache/doris/clone/ColocateTableCheckerAndBalancer.java | 3 ++-
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/SystemHandler.java
b/fe/fe-core/src/main/java/org/apache/doris/alter/SystemHandler.java
index 9d2b49531a..a6fe838fa8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/SystemHandler.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/SystemHandler.java
@@ -96,7 +96,7 @@ public class SystemHandler extends AlterHandler {
}
LOG.info("backend {} lefts {} replicas to decommission: {}", beId,
backendTabletIds.size(),
-backendTabletIds.size() <= 20 ? backendTabletIds : "too
many");
+backendTabletIds.subList(0, Math.min(10,
backendTabletIds.size(;
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/clone/ColocateTableCheckerAndBalancer.java
b/fe/fe-core/src/main/java/org/apache/doris/clone/ColocateTableCheckerAndBalancer.java
index 282d361904..c7848b3937 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/clone/ColocateTableCheckerAndBalancer.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/clone/ColocateTableCheckerAndBalancer.java
@@ -385,6 +385,7 @@ public class ColocateTableCheckerAndBalancer extends
MasterDaemon {
if (!seqIndexes.isEmpty()) {
srcBeId = beId;
hasUnavailableBe = true;
+LOG.info("find unavailable backend {} in colocate group:
{}", beId, groupId);
break;
}
}
@@ -394,7 +395,7 @@ public class ColocateTableCheckerAndBalancer extends
MasterDaemon {
unavailableBeIds, statistic,
flatBackendsPerBucketSeq);
// if there is only one available backend and no unavailable
bucketId to relocate, end the outer loop
-if (backendWithReplicaNum.size() <= 1) {
+if (backendWithReplicaNum.size() <= 1 && !hasUnavailableBe) {
break;
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12637: [fix](inpredicate)always use PredicateColumn for CHAR, VARCHAR and STRING type
yiguolei merged PR #12637: URL: https://github.com/apache/doris/pull/12637 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](inpredicate)always use PredicateColumn for CHAR, VARCHAR and STRING type (#12637)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 3b7a04ee8b [fix](inpredicate)always use PredicateColumn
for CHAR, VARCHAR and STRING type (#12637)
3b7a04ee8b is described below
commit 3b7a04ee8b61033618aaafd6262f5d70e2029187
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Mon Sep 19 08:37:06 2022 +0800
[fix](inpredicate)always use PredicateColumn for CHAR, VARCHAR
and STRING type (#12637)
The predicate column type for char, varchar and string is
PredicateColumnType, so _base_evaluate method should convert the
input column to PredicateColumnType always.
---
be/src/olap/comparison_predicate.h | 7 ++-
be/src/olap/in_list_predicate.h| 7 ++-
2 files changed, 12 insertions(+), 2 deletions(-)
diff --git a/be/src/olap/comparison_predicate.h
b/be/src/olap/comparison_predicate.h
index d05c847e2c..6fcb45bd3c 100644
--- a/be/src/olap/comparison_predicate.h
+++ b/be/src/olap/comparison_predicate.h
@@ -552,7 +552,8 @@ private:
}
} else {
auto* data_array =
-
vectorized::check_and_get_column>(column)
+
vectorized::check_and_get_column>(
+column)
->get_data()
.data();
@@ -561,6 +562,10 @@ private:
}
T _value;
+static constexpr PrimitiveType EvalType = (Type == TYPE_CHAR ? TYPE_STRING
: Type);
};
+template
+constexpr PrimitiveType ComparisonPredicateBase::EvalType;
+
} //namespace doris
diff --git a/be/src/olap/in_list_predicate.h b/be/src/olap/in_list_predicate.h
index 87acd34683..b0b9fc5330 100644
--- a/be/src/olap/in_list_predicate.h
+++ b/be/src/olap/in_list_predicate.h
@@ -375,7 +375,8 @@ private:
}
} else {
auto* nested_col_ptr =
-
vectorized::check_and_get_column>(column);
+
vectorized::check_and_get_column>(
+column);
auto& data_array = nested_col_ptr->get_data();
for (uint16_t i = 0; i < size; i++) {
@@ -410,6 +411,10 @@ private:
mutable std::vector _value_in_dict_flags;
T _min_value;
T _max_value;
+static constexpr PrimitiveType EvalType = (Type == TYPE_CHAR ? TYPE_STRING
: Type);
};
+template
+constexpr PrimitiveType InListPredicateBase::EvalType;
+
} //namespace doris
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12692: [enhancement](test) add yandex_metrica cases to p2
yiguolei merged PR #12692: URL: https://github.com/apache/doris/pull/12692 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhancement](test) add yandex_metrica cases to p2 (#12692)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new fc8f4c787d [enhancement](test) add yandex_metrica cases to p2 (#12692) fc8f4c787d is described below commit fc8f4c787d87a89a5de71f1c90223f24bff39060 Author: Yongqiang YANG <98214048+dataroar...@users.noreply.github.com> AuthorDate: Mon Sep 19 08:37:48 2022 +0800 [enhancement](test) add yandex_metrica cases to p2 (#12692) --- .../sql/duplicateAggregationKeys.out | 225 + .../data/yandex_metrica_p2/sql/like1.out | 4 + .../data/yandex_metrica_p2/sql/like2.out | 4 + .../data/yandex_metrica_p2/sql/like3.out | 4 + .../data/yandex_metrica_p2/sql/like4.out | 4 + .../data/yandex_metrica_p2/sql/loyalty.out | 15 ++ .../data/yandex_metrica_p2/sql/maxStringIf.out | 23 +++ .../data/yandex_metrica_p2/sql/minMax.out | 23 +++ .../sql/monotonicEvaluationSegfault.out| 4 + .../data/yandex_metrica_p2/sql/subqueryInWhere.out | 4 + .../data/yandex_metrica_p2/sql/where01.out | 3 + .../data/yandex_metrica_p2/sql/where02.out | 3 + regression-test/data/yandex_metrica_p2/view.out| 28 +++ .../suites/yandex_metrica_p2/ddl/hits.sql | 142 + .../suites/yandex_metrica_p2/ddl/visits.sql| 190 + .../suites/yandex_metrica_p2/load.groovy | 74 +++ .../sql/duplicateAggregationKeys.sql | 1 + .../suites/yandex_metrica_p2/sql/like1.sql | 1 + .../suites/yandex_metrica_p2/sql/like2.sql | 1 + .../suites/yandex_metrica_p2/sql/like3.sql | 1 + .../suites/yandex_metrica_p2/sql/like4.sql | 1 + .../suites/yandex_metrica_p2/sql/loyalty.sql | 13 ++ .../suites/yandex_metrica_p2/sql/maxStringIf.sql | 1 + .../suites/yandex_metrica_p2/sql/minMax.sql| 1 + .../sql/monotonicEvaluationSegfault.sql| 1 + .../yandex_metrica_p2/sql/subqueryInWhere.sql | 1 + .../suites/yandex_metrica_p2/sql/where01.sql | 1 + .../suites/yandex_metrica_p2/sql/where02.sql | 1 + .../suites/yandex_metrica_p2/view.groovy | 45 + 29 files changed, 819 insertions(+) diff --git a/regression-test/data/yandex_metrica_p2/sql/duplicateAggregationKeys.out b/regression-test/data/yandex_metrica_p2/sql/duplicateAggregationKeys.out new file mode 100644 index 00..c85371a435 --- /dev/null +++ b/regression-test/data/yandex_metrica_p2/sql/duplicateAggregationKeys.out @@ -0,0 +1,225 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !duplicateAggregationKeys -- +http://auto.ru/chatay-barana.ru/traction.html#maybettaya 2014-03-17 http://auto.ru/chatay-barana.ru/traction.html#maybettaya +http://auto.ru/chatay-barana.ru/traction.html#maybettaya 2014-03-18 http://auto.ru/chatay-barana.ru/traction.html#maybettaya +http://auto.ru/chatay-barana.ru/traction.html#maybettaya 2014-03-19 http://auto.ru/chatay-barana.ru/traction.html#maybettaya +http://auto.ru/chatay-barana.ru/traction.html#maybettaya 2014-03-20 http://auto.ru/chatay-barana.ru/traction.html#maybettaya +http://auto.ru/chatay-barana.ru/traction.html#maybettaya 2014-03-21 http://auto.ru/chatay-barana.ru/traction.html#maybettaya +http://auto.ru/chatay-barana.ru/traction.html#maybettaya 2014-03-22 http://auto.ru/chatay-barana.ru/traction.html#maybettaya +http://auto.ru/chatay-barana.ru/traction.html#maybettaya 2014-03-23 http://auto.ru/chatay-barana.ru/traction.html#maybettaya +http://auto.ru/chatay-baranta_bound-in-tanks.ru/forumyazan 2014-03-17 http://auto.ru/chatay-baranta_bound-in-tanks.ru/forumyazan +http://auto.ru/chatay-baranta_bound-in-tanks.ru/forumyazan 2014-03-18 http://auto.ru/chatay-baranta_bound-in-tanks.ru/forumyazan +http://auto.ru/chatay-baranta_bound-in-tanks.ru/forumyazan 2014-03-19 http://auto.ru/chatay-baranta_bound-in-tanks.ru/forumyazan +http://auto.ru/chatay-baranta_bound-in-tanks.ru/forumyazan 2014-03-20 http://auto.ru/chatay-baranta_bound-in-tanks.ru/forumyazan +http://auto.ru/chatay-baranta_bound-in-thankYou=ru/tver/zhanny 2014-03-17 http://auto.ru/chatay-baranta_bound-in-thankYou=ru/tver/zhanny +http://auto.ru/chatay-baranta_bound-in-thankYou=ru/tver/zhanny 2014-03-18 http://auto.ru/chatay-baranta_bound-in-thankYou=ru/tver/zhanny +http://auto.ru/chatay-baranta_bound-in-thankYou=ru/tver/zhanny 2014-03-19 http://auto.ru/chatay-baranta_bound-in-thankYou=ru/tver/zhanny +http://auto.ru/chatay-baranta_bound-in-thankYou=ru/tver/zhanny 2014-03-20 http://auto.ru/chatay-baranta_bound-in-thankYou=ru/tver/zhanny +http://auto.ru/c
[GitHub] [doris] yiguolei merged pull request #12693: [enhancement](test) add opensky cases to p2
yiguolei merged PR #12693: URL: https://github.com/apache/doris/pull/12693 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhancement](test) add opensky cases to p2 (#12693)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 625ac83f72 [enhancement](test) add opensky cases to p2 (#12693) 625ac83f72 is described below commit 625ac83f72c52b6baabaaa340007341c3bde3df2 Author: Yongqiang YANG <98214048+dataroar...@users.noreply.github.com> AuthorDate: Mon Sep 19 08:38:17 2022 +0800 [enhancement](test) add opensky cases to p2 (#12693) --- .../data/opensky_p2/sql/avgDistance.out| 4 + regression-test/data/opensky_p2/sql/count.out | 4 + .../data/opensky_p2/sql/flightsCount.out | 134 + .../data/opensky_p2/sql/mostBusyOrigin.out | 103 .../data/opensky_p2/sql/totalDistance.out | 4 + regression-test/suites/opensky_p2/ddl/opensky.sql | 18 +++ regression-test/suites/opensky_p2/load.groovy | 68 +++ .../suites/opensky_p2/sql/avgDistance.sql | 1 + regression-test/suites/opensky_p2/sql/count.sql| 1 + .../suites/opensky_p2/sql/flightsCount.sql | 7 ++ .../suites/opensky_p2/sql/mostBusyOrigin.sql | 11 ++ .../suites/opensky_p2/sql/totalDistance.sql| 1 + 12 files changed, 356 insertions(+) diff --git a/regression-test/data/opensky_p2/sql/avgDistance.out b/regression-test/data/opensky_p2/sql/avgDistance.out new file mode 100644 index 00..35c0077a6f --- /dev/null +++ b/regression-test/data/opensky_p2/sql/avgDistance.out @@ -0,0 +1,4 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !avgDistance -- +1040768 + diff --git a/regression-test/data/opensky_p2/sql/count.out b/regression-test/data/opensky_p2/sql/count.out new file mode 100644 index 00..7db64bddff --- /dev/null +++ b/regression-test/data/opensky_p2/sql/count.out @@ -0,0 +1,4 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !count -- +66010789 + diff --git a/regression-test/data/opensky_p2/sql/flightsCount.out b/regression-test/data/opensky_p2/sql/flightsCount.out new file mode 100644 index 00..33eea84a68 --- /dev/null +++ b/regression-test/data/opensky_p2/sql/flightsCount.out @@ -0,0 +1,134 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !flightsCount -- +201852 4347 +201901 6290 +201902 5779 +201903 5667 +201904 5714 +201905 5678 +201906 5740 +201907 5733 +201908 5857 +201909 5951 +201910 5896 +201911 5834 +201912 5881 +201913 6058 +201914 6107 +201915 6177 +201916 6404 +201917 6759 +201918 6619 +201919 6656 +201920 6553 +201921 6744 +201922 6854 +201923 7073 +201924 7021 +201925 6850 +201926 7240 +201927 7263 +201928 7197 +201929 7085 +201930 7231 +201931 7393 +201932 7430 +201933 7495 +201934 7567 +201935 7262 +201936 7311 +201937 6621 +201938 6021 +201939 6934 +201940 6528 +201941 6696 +201942 6721 +201943 6305 +201944 6303 +201945 6179 +201946 6012 +201947 6019 +201948 5770 +201949 4959 +201950 5673 +201951 5967 +201952 5921 +202001 6098 +202002 5745 +202003 5269 +202004 5493 +202005 5812 +202006 5645 +202007 5935 +202008 6006 +202009 6112 +202010 5840 +202011 4918 +202012 3841 +202013 2055 +202014 889 +202015 734 +202016 688 +202017 944 +202018 829 +202019 1028 +202020 1120 +202021 1239 +202022 1733 +202023 1897 +202024 2248 +202025 2755 +202026 3308 +202027 3531 +202028 3610 +202029 3679 +202030 3893 +202031 4320 +202032 4549 +202033 4745 +202034 4907 +202035 4762 +202036 4622 +202037 4612 +202038 4477 +202039 4439 +202040 4248 +202041 4240 +202042 4034 +202043 3614 +202044 3593 +202045 3434 +202046 3351 +202047 3247 +202048 3172 +202049 3282 +202050 3339 +202051 3637 +202052 4023 +202101 3839 +202102 3597 +202103 3144 +202104 3211 +202105 3178 +202106 3294 +202107 3465 +202108 3518 +202109 3775 +202110 3624 +202111 3237 +202112 3759 +202113 4197 +202114 4294 +202115 4253 +202116 4346 +202117 4790 +202118 4966 +202119 5143 +202120 5046 +202121 5094 +202122 4635 +202123 5650 +202124 5590 +202125 6016 +202126 3461 + diff --git a/regression-test/data/opensky_p2/sql/mostBusyOrigin.out b/regression-test/data/opensky_p2/sql/mostBusyOrigin.out new file mode 100644 index 00..a1cb7bacd5 --- /dev/null +++ b/regression-test/data/opensky_p2/sql/mostBusyOrigin.out @@ -0,0 +1,103 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !mostBusyOrigin -- +KORD 745006 1545579 +KDFW 696702 1358356 +KATL 667286 1169451 +KDEN 582709 1287105 +KLAX 581949 2628301 +KLAS 447789 1336521 +KPHX 428558 1344938 +KSEA 412592 1757171 +KCLT 404612 880154 +VIDP 363074 1446759 +EDDF 362643 2263164 +KSFO 361869 2445545 +KJFK 349232 2996621 +KMSP 346010 1286682 +LFPG 344748 2205349 +EGLL 341370 3215745 +EHAM 340272 211
[GitHub] [doris] yiguolei merged pull request #12715: [improvement](new-scan) graceful quit scanner scheduler
yiguolei merged PR #12715: URL: https://github.com/apache/doris/pull/12715 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improvement](new-scan) graceful quit scanner scheduler (#12715)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new bc38b2fdfb [improvement](new-scan) graceful quit scanner scheduler
(#12715)
bc38b2fdfb is described below
commit bc38b2fdfb2435f92f732a94f457c1881da3ba77
Author: Mingyu Chen
AuthorDate: Mon Sep 19 08:39:08 2022 +0800
[improvement](new-scan) graceful quit scanner scheduler (#12715)
---
be/src/vec/exec/scan/scanner_scheduler.cpp | 29 -
be/src/vec/exec/scan/scanner_scheduler.h | 14 +++---
2 files changed, 27 insertions(+), 16 deletions(-)
diff --git a/be/src/vec/exec/scan/scanner_scheduler.cpp
b/be/src/vec/exec/scan/scanner_scheduler.cpp
index 7108143967..9787ecae5d 100644
--- a/be/src/vec/exec/scan/scanner_scheduler.cpp
+++ b/be/src/vec/exec/scan/scanner_scheduler.cpp
@@ -32,11 +32,28 @@ namespace doris::vectorized {
ScannerScheduler::ScannerScheduler() {}
ScannerScheduler::~ScannerScheduler() {
+if (!_is_init) {
+return;
+}
+
+for (int i = 0; i < QUEUE_NUM; i++) {
+_pending_queues[i]->shutdown();
+}
+
_is_closed = true;
+
_scheduler_pool->shutdown();
_local_scan_thread_pool->shutdown();
_remote_scan_thread_pool->shutdown();
-// TODO: safely delete all objects and graceful exit
+
+_scheduler_pool->wait();
+_local_scan_thread_pool->join();
+_remote_scan_thread_pool->join();
+
+for (int i = 0; i < QUEUE_NUM; i++) {
+delete _pending_queues[i];
+}
+delete[] _pending_queues;
}
Status ScannerScheduler::init(ExecEnv* env) {
@@ -53,14 +70,16 @@ Status ScannerScheduler::init(ExecEnv* env) {
}
// 2. local scan thread pool
-_local_scan_thread_pool = new PriorityWorkStealingThreadPool(
+_local_scan_thread_pool.reset(new PriorityWorkStealingThreadPool(
config::doris_scanner_thread_pool_thread_num,
env->store_paths().size(),
-config::doris_scanner_thread_pool_queue_size);
+config::doris_scanner_thread_pool_queue_size));
// 3. remote scan thread pool
-_remote_scan_thread_pool = new
PriorityThreadPool(config::doris_scanner_thread_pool_thread_num,
-
config::doris_scanner_thread_pool_queue_size);
+_remote_scan_thread_pool.reset(
+new
PriorityThreadPool(config::doris_scanner_thread_pool_thread_num,
+
config::doris_scanner_thread_pool_queue_size));
+_is_init = true;
return Status::OK();
}
diff --git a/be/src/vec/exec/scan/scanner_scheduler.h
b/be/src/vec/exec/scan/scanner_scheduler.h
index a72fd5021e..f8c1a8f3df 100644
--- a/be/src/vec/exec/scan/scanner_scheduler.h
+++ b/be/src/vec/exec/scan/scanner_scheduler.h
@@ -40,13 +40,6 @@ namespace doris::vectorized {
// Each Scanner will act as a producer, read a group of blocks and put
them into
// the corresponding block queue.
// The corresponding ScanNode will act as a consumer to consume blocks
from the block queue.
-
-using ContextMap = phmap::parallel_flat_hash_map<
-std::string, std::shared_ptr,
phmap::priv::hash_default_hash,
-phmap::priv::hash_default_eq,
-std::allocator>>, 12,
-std::mutex>;
-
class Env;
class ScannerScheduler {
public:
@@ -82,13 +75,12 @@ private:
// execution thread pool
// _local_scan_thread_pool is for local scan task(typically, olap scanner)
// _remote_scan_thread_pool is for remote scan task(cold data on s3, hdfs,
etc.)
-PriorityThreadPool* _local_scan_thread_pool;
-PriorityThreadPool* _remote_scan_thread_pool;
+std::unique_ptr _local_scan_thread_pool;
+std::unique_ptr _remote_scan_thread_pool;
// true is the scheduler is closed.
std::atomic_bool _is_closed = {false};
-
-ContextMap _context_map;
+bool _is_init = false;
};
} // namespace doris::vectorized
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12674: [enhancement](compaction) prevent quick_compaction&auto_compaction conflict
yiguolei merged PR #12674: URL: https://github.com/apache/doris/pull/12674 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhancement](compaction) prevent quick_compaction&auto_compaction conflict (#12674)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 65cff8d40c [enhancement](compaction) prevent
quick_compaction&auto_compaction conflict (#12674)
65cff8d40c is described below
commit 65cff8d40c1f22d662b19c9a8218ee5ec0aef7df
Author: yixiutt <102007456+yixi...@users.noreply.github.com>
AuthorDate: Mon Sep 19 08:39:27 2022 +0800
[enhancement](compaction) prevent quick_compaction&auto_compaction conflict
(#12674)
Co-authored-by: yixiutt
---
be/src/olap/olap_server.cpp | 8
1 file changed, 8 insertions(+)
diff --git a/be/src/olap/olap_server.cpp b/be/src/olap/olap_server.cpp
index d35ec4d712..e86f88dcb1 100644
--- a/be/src/olap/olap_server.cpp
+++ b/be/src/olap/olap_server.cpp
@@ -683,10 +683,18 @@ Status
StorageEngine::submit_compaction_task(TabletSharedPtr tablet,
Status StorageEngine::_handle_quick_compaction(TabletSharedPtr tablet) {
CumulativeCompaction compact(tablet);
compact.quick_rowsets_compact();
+_pop_tablet_from_submitted_compaction(tablet,
CompactionType::CUMULATIVE_COMPACTION);
return Status::OK();
}
Status StorageEngine::submit_quick_compaction_task(TabletSharedPtr tablet) {
+bool already_exist =
+_push_tablet_into_submitted_compaction(tablet,
CompactionType::CUMULATIVE_COMPACTION);
+if (already_exist) {
+return Status::AlreadyExist(
+"compaction task has already been submitted, tablet_id={},
compaction_type={}.",
+tablet->tablet_id(), CompactionType::CUMULATIVE_COMPACTION);
+}
_quick_compaction_thread_pool->submit_func(
std::bind(&StorageEngine::_handle_quick_compaction, this,
tablet));
return Status::OK();
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12424: [fix](array-type) fix the invalid format load for stream load
yiguolei merged PR #12424: URL: https://github.com/apache/doris/pull/12424 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](array-type) fix the invalid format load for stream load (#12424)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new fa8ed2bccc [fix](array-type) fix the invalid format load for stream
load (#12424)
fa8ed2bccc is described below
commit fa8ed2bfd20eb9d217dfe0c8d0c452c9365f
Author: carlvinhust2012
AuthorDate: Mon Sep 19 08:52:59 2022 +0800
[fix](array-type) fix the invalid format load for stream load (#12424)
this pr is used to fix the invalid format load for stream load.
before the change , we will get the error when we load the invalid array
format.
the origin file to load :
1 [1, 2, 3]
2 [4, 5, 6]
3 \N
4 [7, \N, 8]
5 10, 11, 12
[hugo@xafj-palo]$ sh curl_cmd.sh
{
"TxnId": 11035,
"Label": "11c9f111-188e-4616-9a50-aec8b7814513",
"TwoPhaseCommit": "false",
"Status": "Fail",
"Message": "Array does not start with '[' character, found '1'",
"NumberTotalRows": 0,
"NumberLoadedRows": 0,
"NumberFilteredRows": 0,
"NumberUnselectedRows": 0,
"LoadBytes": 55,
"LoadTimeMs": 7,
"BeginTxnTimeMs": 0,
"StreamLoadPutTimeMs": 2,
"ReadDataTimeMs": 0,
"WriteDataTimeMs": 3,
"CommitAndPublishTimeMs": 0
}
3. after this change, we will get success and the error url which report
the error line.
[hugo@xafj-palo]$ sh curl_cmd.sh
{
"TxnId": 11046,
"Label": "249808ee-55f4-4c08-b671-b3d82689d614",
"TwoPhaseCommit": "false",
"Status": "Success",
"Message": "OK",
"NumberTotalRows": 5,
"NumberLoadedRows": 4,
"NumberFilteredRows": 1,
"NumberUnselectedRows": 0,
"LoadBytes": 55,
"LoadTimeMs": 39,
"BeginTxnTimeMs": 0,
"StreamLoadPutTimeMs": 2,
"ReadDataTimeMs": 0,
"WriteDataTimeMs": 19,
"CommitAndPublishTimeMs": 16,
"ErrorURL":
"http://10.81.85.89:8502/api/_load_error_log?file=__shard_3/error_log_insert_stmt_8d4130f0c18aeb0a-ad7ffd4233c41893_8d4130f0c18aeb0a_ad7ffd4233c41893";
}
the sql select result:
MySQL [example_db]> select * from array_test06;
+--+--+
| k1 | k2 |
+--+--+
| 1 | [1, 2, 3] |
| 2 | [4, 5, 6] |
| 3 | NULL |
| 4 | [7, NULL, 8] |
+--+--+
4 rows in set (0.019 sec)
the url page show us:
"Reason: Invalid format for array column(k2). src line [10, 11, 12]; "
Issue Number: #7570
---
be/src/exec/base_scanner.cpp | 34 ++
be/src/exec/base_scanner.h | 4 +++
be/src/exec/broker_scanner.cpp | 8 ++---
be/src/runtime/types.h | 2 ++
be/src/vec/exec/vbroker_scanner.cpp| 8 ++---
.../data/load_p0/broker_load/simple_array.data | 4 ++-
.../load_p0/broker_load/test_array_load.groovy | 3 +-
7 files changed, 53 insertions(+), 10 deletions(-)
diff --git a/be/src/exec/base_scanner.cpp b/be/src/exec/base_scanner.cpp
index 69edbc4b05..864d86b72e 100644
--- a/be/src/exec/base_scanner.cpp
+++ b/be/src/exec/base_scanner.cpp
@@ -487,4 +487,38 @@ void BaseScanner::_fill_columns_from_path() {
}
}
+bool BaseScanner::is_null(const Slice& slice) {
+return slice.size == 2 && slice.data[0] == '\\' && slice.data[1] == 'N';
+}
+
+bool BaseScanner::is_array(const Slice& slice) {
+return slice.size > 1 && slice.data[0] == '[' && slice.data[slice.size -
1] == ']';
+}
+
+bool BaseScanner::check_array_format(std::vector& split_values) {
+// if not the array format, filter this line and return error url
+auto dest_slot_descs = _dest_tuple_desc->slots();
+for (int j = 0; j < split_values.size() && j < dest_slot_descs.size();
++j) {
+auto dest_slot_desc = dest_slot_descs[j];
+if (!dest_slot_desc->is_materialized()) {
+continue;
+}
+const Slice& value = split_values[j];
+if (dest_slot_desc->type().is_array_type() && !is_null(value) &&
!is_array(value)) {
+RETURN_IF_ERROR(_state->append_error_msg_to_file(
+[&]() -> std::string { return std::string(value.data,
value.size); },
+[&]() -> std::string {
+fmt::memory_buffer err_msg;
+fmt::format_to(err_msg, "Invalid format for array
column({})",
+ dest_slot_desc->col_name());
+return fmt::to_string(err_msg);
+},
+&_scanner_eof));
+_counter->num_rows_filtered++;
+return false;
+}
+}
+return true;
+}
+
} // namespace doris
diff --git a/be/src/exec/base_scanner.h b/be/src/exec/base_scanner.h
index 80157c939c..6711836e53 100644
--- a/be/src/exec/base_scanner.h
+++ b/be/src/exec/base_scanner.h
@@ -92,6 +92,10 @
[GitHub] [doris] yiguolei merged pull request #12344: [regression](left join)Add left join, the left table is empty, the query result is not empty…
yiguolei merged PR #12344: URL: https://github.com/apache/doris/pull/12344 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [regression](left join)Add left join, the left table is empty, the query result is not empty case (#12344)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 6d3ae1e69c [regression](left join)Add left join, the left table is
empty, the query result is not empty case (#12344)
6d3ae1e69c is described below
commit 6d3ae1e69c1afd42d89cc9de9a2d9b85b21af8d0
Author: caoliang-web <71004656+caoliang-...@users.noreply.github.com>
AuthorDate: Mon Sep 19 08:53:50 2022 +0800
[regression](left join)Add left join, the left table is empty, the query
result is not empty case (#12344)
Add left join, the left table is empty, the query result is not empty case
---
.../data/query/join/test_left_join_null.out| 4 ++
.../suites/query/join/test_left_join_null.groovy | 63 ++
2 files changed, 67 insertions(+)
diff --git a/regression-test/data/query/join/test_left_join_null.out
b/regression-test/data/query/join/test_left_join_null.out
new file mode 100644
index 00..ec39c781df
--- /dev/null
+++ b/regression-test/data/query/join/test_left_join_null.out
@@ -0,0 +1,4 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !select --
+1001 test\N
+
diff --git a/regression-test/suites/query/join/test_left_join_null.groovy
b/regression-test/suites/query/join/test_left_join_null.groovy
new file mode 100644
index 00..42227b6485
--- /dev/null
+++ b/regression-test/suites/query/join/test_left_join_null.groovy
@@ -0,0 +1,63 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_left_join_null", "query") {
+
+def tbName1 = "dept_emp"
+def tbName2 = "departments"
+
+sql """
+ CREATE TABLE IF NOT EXISTS ${tbName1} (
+ `emp_no` int NOT NULL,
+ `dept_no` char(4) NOT NULL,
+ `from_date` date NOT NULL,
+ `to_date` date NOT NULL
+)
+unique KEY (`emp_no`,`dept_no`)
+COMMENT "OLAP"
+DISTRIBUTED BY HASH(`emp_no`) BUCKETS 3
+PROPERTIES (
+"replication_allocation" = "tag.location.default: 1",
+"in_memory" = "false",
+"storage_format" = "V2"
+);
+ """
+
+sql """
+CREATE TABLE IF NOT EXISTS ${tbName2} (
+ `dept_no` char(4) NOT NULL,
+ `dept_name` varchar(40) NOT NULL
+)
+ UNIQUE KEY (`dept_no`)
+ COMMENT "OLAP"
+DISTRIBUTED BY HASH(`dept_no`) BUCKETS 3
+PROPERTIES (
+"replication_allocation" = "tag.location.default: 1",
+"in_memory" = "false",
+"storage_format" = "V2"
+);
+"""
+sql """insert into ${tbName2} values ('1001','test')"""
+
+qt_select """
+with test as (
+SELECT dept_no from dept_emp
+ )
+ SELECT * from departments a LEFT JOIN test b on
a.dept_no=b.dept_no
+ """
+
+}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12709: [fix](compile)compile error: open_telemetry_scop_wrapper.hpp cannot file 'UNLIKELY'
yiguolei merged PR #12709: URL: https://github.com/apache/doris/pull/12709 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](compile)compile error: open_telemetry_scop_wrapper.hpp cannot file 'UNLIKELY' (#12709)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new b608de668f [fix](compile)compile error:
open_telemetry_scop_wrapper.hpp cannot file 'UNLIKELY' (#12709)
b608de668f is described below
commit b608de668f5c2108a98428210149b61f75070230
Author: minghong
AuthorDate: Mon Sep 19 09:18:04 2022 +0800
[fix](compile)compile error: open_telemetry_scop_wrapper.hpp cannot file
'UNLIKELY' (#12709)
---
be/src/util/telemetry/open_telemetry_scop_wrapper.hpp | 1 +
1 file changed, 1 insertion(+)
diff --git a/be/src/util/telemetry/open_telemetry_scop_wrapper.hpp
b/be/src/util/telemetry/open_telemetry_scop_wrapper.hpp
index 138d1c014e..4c30333476 100644
--- a/be/src/util/telemetry/open_telemetry_scop_wrapper.hpp
+++ b/be/src/util/telemetry/open_telemetry_scop_wrapper.hpp
@@ -17,6 +17,7 @@
#pragma once
+#include "common/compiler_util.h"
#include "opentelemetry/trace/provider.h"
namespace doris {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12697: [enhancement](test) add tpch_sf100_unique p2 test
yiguolei merged PR #12697: URL: https://github.com/apache/doris/pull/12697 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (b608de668f -> 4669fa54cc)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from b608de668f [fix](compile)compile error: open_telemetry_scop_wrapper.hpp cannot file 'UNLIKELY' (#12709) add 4669fa54cc [enhancement](test) add tpch_sf100_unique p2 test (#12697) No new revisions were added by this update. Summary of changes: .../data/tpch_sf100_unique_sql_p2/sql/q01.out | 5 + .../data/tpch_sf100_unique_sql_p2/sql/q02.out | 101 + .../data/tpch_sf100_unique_sql_p2/sql/q03.out |11 + .../data/tpch_sf100_unique_sql_p2/sql/q04.out | 6 + .../data/tpch_sf100_unique_sql_p2/sql/q05.out | 6 + .../data/tpch_sf100_unique_sql_p2/sql/q06.out | 2 + .../data/tpch_sf100_unique_sql_p2/sql/q07.out | 5 + .../data/tpch_sf100_unique_sql_p2/sql/q08.out | 3 + .../data/tpch_sf100_unique_sql_p2/sql/q09.out | 176 + .../data/tpch_sf100_unique_sql_p2/sql/q10.out |21 + .../data/tpch_sf100_unique_sql_p2/sql/q11.out | 1 + .../data/tpch_sf100_unique_sql_p2/sql/q12.out | 3 + .../data/tpch_sf100_unique_sql_p2/sql/q13.out |46 + .../data/tpch_sf100_unique_sql_p2/sql/q14.out | 2 + .../data/tpch_sf100_unique_sql_p2/sql/q15.out | 2 + .../data/tpch_sf100_unique_sql_p2/sql/q16.out | 27841 +++ .../data/tpch_sf100_unique_sql_p2/sql/q17.out | 2 + .../data/tpch_sf100_unique_sql_p2/sql/q18.out | 101 + .../data/tpch_sf100_unique_sql_p2/sql/q19.out | 2 + .../data/tpch_sf100_unique_sql_p2/sql/q20.out | 17972 .../data/tpch_sf100_unique_sql_p2/sql/q21.out | 101 + .../data/tpch_sf100_unique_sql_p2/sql/q22.out | 8 + .../suites/tpch_sf100_unique_p2/ddl/customer.sql |17 + .../ddl/customer_delete.sql| 0 .../tpch_sf100_unique_p2/ddl/customer_load.sql | 6 + .../ddl/customer_load_sequence.sql | 7 + .../ddl/customer_part_delete.sql | 1 + .../tpch_sf100_unique_p2/ddl/customer_sequence.sql |18 + .../suites/tpch_sf100_unique_p2/ddl/lineitem.sql |25 + .../tpch_sf100_unique_p2/ddl/lineitem_delete.sql | 1 + .../tpch_sf100_unique_p2/ddl/lineitem_load.sql | 6 + .../ddl/lineitem_load_sequence.sql | 7 + .../ddl/lineitem_part_delete.sql | 1 + .../tpch_sf100_unique_p2/ddl/lineitem_sequence.sql |26 + .../suites/tpch_sf100_unique_p2/ddl/nation.sql |13 + .../tpch_sf100_unique_p2/ddl/nation_delete.sql | 1 + .../tpch_sf100_unique_p2/ddl/nation_load.sql | 6 + .../ddl/nation_load_sequence.sql | 7 + .../ddl/nation_part_delete.sql | 1 + .../tpch_sf100_unique_p2/ddl/nation_sequence.sql |14 + .../suites/tpch_sf100_unique_p2/ddl/orders.sql |17 + .../tpch_sf100_unique_p2/ddl/orders_delete.sql | 1 + .../tpch_sf100_unique_p2/ddl/orders_load.sql | 6 + .../ddl/orders_load_sequence.sql | 7 + .../ddl/orders_part_delete.sql | 1 + .../tpch_sf100_unique_p2/ddl/orders_sequence.sql |18 + .../suites/tpch_sf100_unique_p2/ddl/part.sql |18 + .../tpch_sf100_unique_p2/ddl/part_delete.sql | 1 + .../suites/tpch_sf100_unique_p2/ddl/part_load.sql | 6 + .../ddl/part_load_sequence.sql | 7 + .../tpch_sf100_unique_p2/ddl/part_part_delete.sql | 1 + .../tpch_sf100_unique_p2/ddl/part_sequence.sql |19 + .../suites/tpch_sf100_unique_p2/ddl/partsupp.sql |14 + .../tpch_sf100_unique_p2/ddl/partsupp_delete.sql | 1 + .../tpch_sf100_unique_p2/ddl/partsupp_load.sql | 6 + .../ddl/partsupp_load_sequence.sql | 7 + .../ddl/partsupp_part_delete.sql | 1 + .../tpch_sf100_unique_p2/ddl/partsupp_sequence.sql |15 + .../suites/tpch_sf100_unique_p2/ddl/region.sql |12 + .../tpch_sf100_unique_p2/ddl/region_delete.sql | 1 + .../tpch_sf100_unique_p2/ddl/region_load.sql | 6 + .../ddl/region_load_sequence.sql | 7 + .../ddl/region_part_delete.sql | 1 + .../tpch_sf100_unique_p2/ddl/region_sequence.sql |13 + .../suites/tpch_sf100_unique_p2/ddl/supplier.sql |15 + .../tpch_sf100_unique_p2/ddl/supplier_delete.sql | 1 + .../tpch_sf100_unique_p2/ddl/supplier_load.sql | 6 + .../ddl/supplier_load_sequence.sql | 7 + .../ddl/supplier_part_delete.sql | 1 + .../tpch_sf100_unique_p2/ddl/supplier_sequence.sql |16 + .../tpch_sf100_unique_p2/load_four_step.groovy | 117 + .../tpch_sf100_unique_p2/load_one_step.groovy |70 + .../tpch_sf100_unique_p2/load_three_step.groovy|94 + .../tpch_sf10
[doris] branch master updated (4669fa54cc -> 1fa65708d7)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 4669fa54cc [enhancement](test) add tpch_sf100_unique p2 test (#12697) add 1fa65708d7 [test](time_add or sub)add time_add and time_sub funcation case #12641 No new revisions were added by this update. Summary of changes: .../datetime_functions/test_date_function.out | 41 ++ .../datetime_functions/test_date_function.groovy | 48 ++ 2 files changed, 89 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12641: [test](time_add or sub)add time_add and time_sub funcation case
yiguolei merged PR #12641: URL: https://github.com/apache/doris/pull/12641 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhuyongji opened a new issue, #12717: [Bug] Cast函数bug
zhuyongji opened a new issue, #12717: URL: https://github.com/apache/doris/issues/12717 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version 1.0.0-rc03 ### What's Wrong? select cast(round(16698/123,2) as DECIMAL(20,2)) 预期返回135.76 实际返回135.75999 ### What You Expected? 返回135.76 ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12690: [enhancement](pred column) improve predicate column insert performance
github-actions[bot] commented on PR #12690: URL: https://github.com/apache/doris/pull/12690#issuecomment-1250466844 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12690: [enhancement](pred column) improve predicate column insert performance
github-actions[bot] commented on PR #12690: URL: https://github.com/apache/doris/pull/12690#issuecomment-1250466861 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] nextdreamblue commented on a diff in pull request #12513: [feature](http) refactor version info and add new http api for get version info
nextdreamblue commented on code in PR #12513: URL: https://github.com/apache/doris/pull/12513#discussion_r973820465 ## gensrc/script/gen_build_version.sh: ## @@ -27,7 +27,13 @@ set -eo pipefail -build_version="trunk" +build_version_prefix="doris" Review Comment: you can specify any version info you want in this script when you complie -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #12667: [Feature](serialize) move block_data_version to fe heart beat
BiteThet commented on PR #12667: URL: https://github.com/apache/doris/pull/12667#issuecomment-1250472705 > Please add some document for this config done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](vstream load) Fix bug when load json with jsonpath (#12660)
This is an automated email from the ASF dual-hosted git repository.
yangzhg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new fb9e48a34a [fix](vstream load) Fix bug when load json with jsonpath
(#12660)
fb9e48a34a is described below
commit fb9e48a34a5c0a7f52fe032836de889094ef3121
Author: yinzhijian <373141...@qq.com>
AuthorDate: Mon Sep 19 10:13:18 2022 +0800
[fix](vstream load) Fix bug when load json with jsonpath (#12660)
---
be/src/vec/exec/vjson_scanner.cpp | 70 -
be/test/vec/exec/vjson_scanner_test.cpp| 13 +--
.../stream_load/load_json_with_jsonpath.out| 11 +++
.../stream_load/test_load_with_jsonpath.json | 1 +
.../stream_load/load_json_with_jsonpath.groovy | 109 +
5 files changed, 168 insertions(+), 36 deletions(-)
diff --git a/be/src/vec/exec/vjson_scanner.cpp
b/be/src/vec/exec/vjson_scanner.cpp
index b0f8ed5ba8..95f4c8658a 100644
--- a/be/src/vec/exec/vjson_scanner.cpp
+++ b/be/src/vec/exec/vjson_scanner.cpp
@@ -434,12 +434,15 @@ Status
VJsonReader::_write_columns_by_jsonpath(rapidjson::Value& objectValue,
const
std::vector& slot_descs,
std::vector&
columns,
bool* valid) {
-int nullcount = 0;
int ctx_idx = 0;
-size_t column_num = slot_descs.size();
-for (size_t i = 0; i < column_num; i++) {
-int dest_index = ctx_idx++;
-auto* column_ptr = columns[dest_index].get();
+bool has_valid_value = false;
+size_t cur_row_count = columns[0]->size();
+for (auto slot_desc : slot_descs) {
+if (!slot_desc->is_materialized()) {
+continue;
+}
+int i = ctx_idx++;
+auto* column_ptr = columns[i].get();
rapidjson::Value* json_values = nullptr;
bool wrap_explicitly = false;
if (LIKELY(i < _parsed_jsonpaths.size())) {
@@ -450,16 +453,12 @@ Status
VJsonReader::_write_columns_by_jsonpath(rapidjson::Value& objectValue,
if (json_values == nullptr) {
// not match in jsondata.
-if (slot_descs[i]->is_nullable()) {
-auto* nullable_column =
reinterpret_cast(column_ptr);
-nullable_column->insert_default();
-nullcount++;
-} else {
+if (!slot_descs[i]->is_nullable()) {
RETURN_IF_ERROR(_append_error_msg(
objectValue,
"The column `{}` is not nullable, but it's not found
in jsondata.",
slot_descs[i]->col_name(), valid));
-break;
+return Status::OK();
}
} else {
CHECK(json_values->IsArray());
@@ -471,15 +470,29 @@ Status
VJsonReader::_write_columns_by_jsonpath(rapidjson::Value& objectValue,
}
RETURN_IF_ERROR(_write_data_to_column(json_values, slot_descs[i],
column_ptr, valid));
if (!(*valid)) {
-break;
+return Status::OK();
}
+has_valid_value = true;
}
}
-
-if (nullcount == column_num) {
+if (!has_valid_value) {
RETURN_IF_ERROR(_append_error_msg(
objectValue, "All fields is null or not matched, this is a
invalid row.", "",
valid));
+return Status::OK();
+}
+ctx_idx = 0;
+for (auto slot_desc : slot_descs) {
+if (!slot_desc->is_materialized()) {
+continue;
+}
+int dest_index = ctx_idx++;
+auto* column_ptr = columns[dest_index].get();
+if (column_ptr->size() < cur_row_count + 1) {
+DCHECK(column_ptr->size() == cur_row_count);
+column_ptr->assume_mutable()->insert_default();
+}
+DCHECK(column_ptr->size() == cur_row_count + 1);
}
return Status::OK();
}
@@ -1062,9 +1075,10 @@ Status VSIMDJsonReader::_generate_json_paths(const
std::string& jsonpath,
Status VSIMDJsonReader::_write_columns_by_jsonpath(simdjson::ondemand::value
value,
const
std::vector& slot_descs,
Block& block, bool* valid) {
-int nullcount = 0;
size_t column_num = slot_descs.size();
auto object_value = value.get_object();
+bool has_valid_value = false;
+size_t cur_row_count = block.rows();
for (size_t i = 0; i < column_num; i++) {
auto* column_ptr =
block.get_by_position(i).column->assume_mutable().get();
simdjson::simdjson_result json_value;
@@ -1074,29 +1088,35 @@ Status
VSIMDJsonReader::_write_columns_by_jsonpath(simdjson::ondemand::value val
}
if (i >= _parsed_jsonpaths.size() ||
[GitHub] [doris] yangzhg merged pull request #12660: [fix](vstream load) Fix bug when load json with jsonpath
yangzhg merged PR #12660: URL: https://github.com/apache/doris/pull/12660 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12607: [fix](agg)the output of window function's nullability should be consistent with output slot
github-actions[bot] commented on PR #12607: URL: https://github.com/apache/doris/pull/12607#issuecomment-1250478885 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12607: [fix](agg)the output of window function's nullability should be consistent with output slot
github-actions[bot] commented on PR #12607: URL: https://github.com/apache/doris/pull/12607#issuecomment-1250478901 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TaoZex commented on pull request #12532: [DOCS](function) Add docs of math function
TaoZex commented on PR #12532: URL: https://github.com/apache/doris/pull/12532#issuecomment-1250482385 @zy-kkk Could you please help me review these documents?Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: fix
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new 5a4fddbc642 fix 5a4fddbc642 is described below commit 5a4fddbc6426a59e4662e99f78a412f1d1354733 Author: jiafeng.zhang AuthorDate: Mon Sep 19 10:34:06 2022 +0800 fix --- .../Account-Management-Statements/GRANT.md | 37 .../ALTER-SYSTEM-ADD-BACKEND.md| 10 -- .../ALTER-SYSTEM-MODIFY-BACKEND.md | 12 ++- .../Alter/ALTER-TABLE-BITMAP.md| 2 +- .../Alter/ALTER-TABLE-COMMENT.md | 1 + .../Alter/ALTER-TABLE-PARTITION.md | 4 +- .../Alter/ALTER-TABLE-PROPERTY.md | 54 +-- .../Alter/ALTER-TABLE-REPLACE.md | 43 +++-- .../Alter/ALTER-TABLE-ROLLUP.md| 13 ++- .../Backup-and-Restore/BACKUP.md | 21 - .../Backup-and-Restore/RESTORE.md | 1 + .../Create/CREATE-TABLE-AS-SELECT.md | 12 ++- .../Create/CREATE-TABLE.md | 66 -- .../Data-Definition-Statements/Drop/DROP-TABLE.md | 2 +- .../ADMIN-COPY-TABLET.md | 101 + .../ADMIN-DIAGNOSE-TABLET.md | 24 +++-- .../Account-Management-Statements/GRANT.md | 21 +++-- .../ALTER-SYSTEM-ADD-BACKEND.md| 14 +-- .../ALTER-SYSTEM-MODIFY-BACKEND.md | 12 ++- .../Alter/ALTER-TABLE-BITMAP.md| 2 +- .../Alter/ALTER-TABLE-COMMENT.md | 1 + .../Alter/ALTER-TABLE-PARTITION.md | 4 +- .../Alter/ALTER-TABLE-PROPERTY.md | 73 --- .../Alter/ALTER-TABLE-REPLACE.md | 43 +++-- .../Alter/ALTER-TABLE-ROLLUP.md| 9 ++ .../Backup-and-Restore/BACKUP.md | 8 +- .../Backup-and-Restore/RESTORE.md | 1 + .../Create/CREATE-TABLE-AS-SELECT.md | 12 ++- .../Create/CREATE-TABLE.md | 65 +++-- .../Data-Definition-Statements/Drop/DROP-TABLE.md | 2 +- .../ADMIN-COPY-TABLET.md | 101 + .../ADMIN-DIAGNOSE-TABLET.md | 24 +++-- sidebars.json | 1 + 33 files changed, 607 insertions(+), 189 deletions(-) diff --git a/docs/sql-manual/sql-reference/Account-Management-Statements/GRANT.md b/docs/sql-manual/sql-reference/Account-Management-Statements/GRANT.md index c5e9e3f464c..05dff1c354a 100644 --- a/docs/sql-manual/sql-reference/Account-Management-Statements/GRANT.md +++ b/docs/sql-manual/sql-reference/Account-Management-Statements/GRANT.md @@ -35,7 +35,7 @@ GRANT The GRANT command is used to grant the specified user or role specified permissions ```sql -GRANT privilege_list ON db_name[.tbl_name] TO user_identity [ROLE role_name] +GRANT privilege_list ON priv_level TO user_identity [ROLE role_name] GRANT privilege_list ON RESOURCE resource_name TO user_identity [ROLE role_name] @@ -45,11 +45,11 @@ privilege_list is a list of privileges to be granted, separated by commas. Curre NODE_PRIV: Cluster node operation permissions, including node online and offline operations. Only the root user has this permission and cannot be granted to other users. ADMIN_PRIV: All privileges except NODE_PRIV. GRANT_PRIV: Privilege for operation privileges. Including creating and deleting users, roles, authorization and revocation, setting passwords, etc. -SELECT_PRIV: read permission on the specified library or table -LOAD_PRIV: Import privileges on the specified library or table -ALTER_PRIV: Schema change permission for the specified library or table -CREATE_PRIV: Create permission on the specified library or table -DROP_PRIV: drop privilege on the specified library or table +SELECT_PRIV: read permission on the specified database or table +LOAD_PRIV: Import privileges on the specified database or table +ALTER_PRIV: Schema change permission for the specified database or table +CREATE_PRIV: Create permission on the specified database or table +DROP_PRIV: drop privilege on the specified database or table USAGE_PRIV: access to the specified resource ALL and READ_WRITE in legacy permissions will be converted to: SELECT_PRIV,LOAD_PRIV,ALTER_PRIV,CREATE_PRIV,DROP_PRIV; @@ -58,16 +58,17 @@ privilege_list is a list of privileges to be granted, separated by commas. Curre Permission classification: 1. Node Privilege: NODE_PRIV -2. Library table permissions: SELECT_PRIV, LOAD_PRIV, ALTER_PRIV, CREATE_PRIV, DROP_PRIV +2. database table permissions: SELECT_PRIV
[GitHub] [doris] Wilson-BT opened a new issue, #12718: 【Bug】doris writer write failed on json format
Wilson-BT opened a new issue, #12718: URL: https://github.com/apache/doris/issues/12718 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version Doris: 1.1.2 DataxWriter: 0.0.1-SNAPSHOT ### What's Wrong? 1. When we use json format and batch write to doris, the order of column key on http header is different with json-data. 2. The method ```Key.getFormat``` not match【DOC】,we should make compatible between loadProps and out-format. ### What You Expected? fix bug ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12690: [enhancement](pred column) improve predicate column insert performance
yiguolei merged PR #12690: URL: https://github.com/apache/doris/pull/12690 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] weizhengte commented on a diff in pull request #9983: [feature-wip](statistics) Internal-query, execute SQL query statement internally (in FE)
weizhengte commented on code in PR #9983:
URL: https://github.com/apache/doris/pull/9983#discussion_r973668119
##
fe/fe-core/src/main/java/org/apache/doris/statistics/util/Packet.java:
##
@@ -0,0 +1,42 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.statistics.util;
+
+import java.nio.ByteBuffer;
+
+public abstract class Packet {
Review Comment:
the relevant code has been deeply refactored, and some problem have been
resolved. same as following.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhancement](pred column) improve predicate column insert performance (#12690)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 415721ef20 [enhancement](pred column) improve predicate column insert
performance (#12690)
415721ef20 is described below
commit 415721ef2021fdf428bb7f94a796f9cb02c67aa6
Author: yiguolei <676222...@qq.com>
AuthorDate: Mon Sep 19 10:53:48 2022 +0800
[enhancement](pred column) improve predicate column insert performance
(#12690)
Co-authored-by: yiguolei
---
be/src/vec/columns/predicate_column.h | 21 ++---
1 file changed, 14 insertions(+), 7 deletions(-)
diff --git a/be/src/vec/columns/predicate_column.h
b/be/src/vec/columns/predicate_column.h
index 6662545b2f..68a4095450 100644
--- a/be/src/vec/columns/predicate_column.h
+++ b/be/src/vec/columns/predicate_column.h
@@ -90,11 +90,14 @@ private:
void insert_string_to_res_column(const uint16_t* sel, size_t sel_size,
vectorized::ColumnString* res_ptr) {
+StringRef refs[sel_size];
for (size_t i = 0; i < sel_size; i++) {
uint16_t n = sel[i];
auto& sv = reinterpret_cast(data[n]);
-res_ptr->insert_data(sv.ptr, sv.len);
+refs[i].data = sv.ptr;
+refs[i].size = sv.len;
}
+res_ptr->insert_many_strings(refs, sel_size);
}
void insert_decimal_to_res_column(const uint16_t* sel, size_t sel_size,
@@ -261,6 +264,9 @@ public:
void insert_many_binary_data(char* data_array, uint32_t* len_array,
uint32_t* start_offset_array, size_t num)
override {
+if (num == 0) {
+return;
+}
if constexpr (std::is_same_v) {
if (_pool == nullptr) {
_pool.reset(new MemPool());
@@ -272,13 +278,14 @@ public:
}
char* destination = (char*)_pool->allocate(total_mem_size);
+memcpy(destination, data_array, total_mem_size);
+// Resize the underline data to allow data copy directly
+size_t org_elem_num = data.size();
+data.resize(org_elem_num + num);
for (size_t i = 0; i < num; i++) {
-uint32_t len = len_array[i];
-uint32_t start_offset = start_offset_array[i];
-memcpy(destination, data_array + start_offset, len);
-StringValue sv(destination, len);
-data.push_back_without_reserve(sv);
-destination += len;
+data[org_elem_num + i].ptr = destination;
+data[org_elem_num + i].len = len_array[i];
+destination += len_array[i];
}
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 opened a new issue, #12719: [Bug] `CURDATE` returns wrong result
Gabriel39 opened a new issue, #12719: URL: https://github.com/apache/doris/issues/12719 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? ![Uploading image.png…]() ### What You Expected?  ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 opened a new pull request, #12720: [Bug](date) Fix wrong result produced by date function
Gabriel39 opened a new pull request, #12720: URL: https://github.com/apache/doris/pull/12720 # Proposed changes Issue Number: close #12719 ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12701: [improve](Nereids): Add all slots used by onClause in project
github-actions[bot] commented on PR #12701: URL: https://github.com/apache/doris/pull/12701#issuecomment-1250508168 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #12701: [improve](Nereids): Add all slots used by onClause in project
morrySnow merged PR #12701: URL: https://github.com/apache/doris/pull/12701 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12701: [improve](Nereids): Add all slots used by onClause in project
github-actions[bot] commented on PR #12701: URL: https://github.com/apache/doris/pull/12701#issuecomment-1250508276 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improve](Nereids) Add all slots used by onClause to project when reorder and fix reorder mark (#12701)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 75d7de89a5 [improve](Nereids) Add all slots used by onClause to
project when reorder and fix reorder mark (#12701)
75d7de89a5 is described below
commit 75d7de89a56e4eca9c1831a887e38b4683208235
Author: jakevin
AuthorDate: Mon Sep 19 11:01:25 2022 +0800
[improve](Nereids) Add all slots used by onClause to project when reorder
and fix reorder mark (#12701)
1. Add all slots used by onClause in project
```
(A & B) & C like
join(hash conjuncts: C.t2 = A.t2)
|---project(A.t2)
| +---join(hash conjuncts: A.t1 = B.t1)
| +---A
| +---B
+---C
transform to (A & C) & B
join(hash conjuncts: A.t1 = B.t1)
|---project(A.t2)
| +---join(hash conjuncts: C.t2 = A.t2)
| +---A
| +---C
+---B
```
But projection just include `A.t2`, can't find `A.t1`, we should add slots
used by onClause when projection exist.
2. fix join reorder mark
Add mark `LAsscom` when apply `LAsscom`
3. remove slotReference
use `Slot` instead of `SlotReference` to avoid cast.
---
.../rules/exploration/join/InnerJoinLAsscom.java | 11 +++-
.../exploration/join/InnerJoinLAsscomProject.java | 4 +-
.../rules/exploration/join/JoinCommuteHelper.java | 7 +--
.../rules/exploration/join/JoinLAsscomHelper.java | 70 ++
.../rules/exploration/join/OuterJoinLAsscom.java | 21 +--
.../exploration/join/OuterJoinLAsscomProject.java | 4 +-
.../rules/exploration/join/ThreeJoinHelper.java| 31 +-
.../LogicalOlapScanToPhysicalOlapScan.java | 5 +-
.../org/apache/doris/nereids/util/JoinUtils.java | 43 +++--
.../java/org/apache/doris/nereids/util/Utils.java | 10
10 files changed, 103 insertions(+), 103 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLAsscom.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLAsscom.java
index 9286ba33f7..021987cb7d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLAsscom.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLAsscom.java
@@ -20,6 +20,9 @@ package org.apache.doris.nereids.rules.exploration.join;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.rules.exploration.OneExplorationRuleFactory;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
/**
* Rule for change inner join LAsscom (associative and commutive).
@@ -37,7 +40,7 @@ public class InnerJoinLAsscom extends
OneExplorationRuleFactory {
@Override
public Rule build() {
return innerLogicalJoin(innerLogicalJoin(), group())
-.when(topJoin -> JoinLAsscomHelper.checkInner(topJoin,
topJoin.left()))
+.when(topJoin -> check(topJoin, topJoin.left()))
.then(topJoin -> {
JoinLAsscomHelper helper = new JoinLAsscomHelper(topJoin,
topJoin.left());
if (!helper.initJoinOnCondition()) {
@@ -46,4 +49,10 @@ public class InnerJoinLAsscom extends
OneExplorationRuleFactory {
return helper.newTopJoin();
}).toRule(RuleType.LOGICAL_INNER_JOIN_LASSCOM);
}
+
+public static boolean check(LogicalJoin topJoin,
+LogicalJoin bottomJoin) {
+return !bottomJoin.getJoinReorderContext().hasCommuteZigZag()
+&& !topJoin.getJoinReorderContext().hasLAsscom();
+}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLAsscomProject.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLAsscomProject.java
index 50e4858b72..7f3fc20c68 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLAsscomProject.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLAsscomProject.java
@@ -39,10 +39,10 @@ public class InnerJoinLAsscomProject extends
OneExplorationRuleFactory {
@Override
public Rule build() {
return innerLogicalJoin(logicalProject(innerLogicalJoin()), group())
-.when(topJoin -> JoinLAsscomHelper.checkInner(topJoin,
topJoin.left().child()))
+.when(topJoin -> InnerJoinLAsscom.check(topJoin,
topJoin.left().child()))
.then(topJoin -> {
JoinLAsscomHelper helper = new JoinLAsscomHelper(topJoin,
topJoin.le
[GitHub] [doris] haohao0103 commented on issue #3217: start_fe.sh has an error in the MAC environment
haohao0103 commented on issue #3217: URL: https://github.com/apache/doris/issues/3217#issuecomment-1250518160 @stalary hello ,how to fix this ? i got the same question -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] weizhengte commented on pull request #9983: [feature-wip](statistics) Internal-query, execute SQL query statement internally (in FE)
weizhengte commented on PR #9983: URL: https://github.com/apache/doris/pull/9983#issuecomment-1250519238 > Why don't use jdbc to send query to FE itself directly? Indeed, the previous implementation was complicated... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #12602: [refactor](file scanner)Refactor file scanner.
morningman commented on code in PR #12602:
URL: https://github.com/apache/doris/pull/12602#discussion_r973836109
##
env.sh:
##
@@ -57,7 +57,7 @@ if ! ${PYTHON} --version; then
fi
if [[ -z "${DORIS_TOOLCHAIN}" ]]; then
-DORIS_TOOLCHAIN=gcc
+DORIS_TOOLCHAIN=clang
Review Comment:
change to gcc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #12714: [feature-wip](statistics) Template for building internal query SQL statements
Kikyou1997 commented on code in PR #12714:
URL: https://github.com/apache/doris/pull/12714#discussion_r973839392
##
fe/fe-core/src/main/java/org/apache/doris/statistics/util/InternalSqlTemplate.java:
##
@@ -0,0 +1,282 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.statistics.util;
+
+import org.apache.doris.common.InvalidFormatException;
+
+import com.google.common.collect.Sets;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Template for building internal query SQL statements.
+ * e.g.:
+ * - "SELECT ${column} FROM ${table}"
+ * - ${column} and ${table} will be replaced with the actual executed table
and column.
+ */
+public class InternalSqlTemplate {
+/** -- for statistics begin
-- */
+public static final String MIN_VALUE_SQL = "SELECT MIN(${column}) AS
min_value FROM ${table};";
+public static final String PARTITION_MIN_VALUE_SQL = "SELECT
MIN(${column}) AS min_value"
++ " FROM ${table} PARTITION (${partition});";
+
+public static final String MAX_VALUE_SQL = "SELECT MAX(${column}) AS
max_value FROM ${table};";
+public static final String PARTITION_MAX_VALUE_SQL = "SELECT
MAX(${column}) AS max_value FROM"
++ " ${table} PARTITION (${partition});";
+
+public static final String NDV_VALUE_SQL = "SELECT NDV(${column}) AS ndv
FROM ${table};";
+public static final String PARTITION_NDV_VALUE_SQL = "SELECT
NDV(${column}) AS ndv FROM"
++ " ${table} PARTITION (${partition});";
+
+public static final String MIN_MAX_NDV_VALUE_SQL = "SELECT MIN(${column})
AS min_value, MAX(${column})"
++ " AS max_value, NDV(${column}) AS ndv FROM ${table};";
+public static final String PARTITION_MIN_MAX_NDV_VALUE_SQL = "SELECT
MIN(${column}) AS min_value,"
++ " MAX(${column}) AS max_value, NDV(${column}) AS ndv FROM
${table} PARTITION (${partition});";
+
+public static final String ROW_COUNT_SQL = "SELECT COUNT(1) AS row_count
FROM ${table};";
+public static final String PARTITION_ROW_COUNT_SQL = "SELECT COUNT(1) AS
row_count FROM ${table} PARTITION"
++ " (${partition});";
+
+public static final String MAX_SIZE_SQL = "SELECT MAX(LENGTH(${column}))
AS max_size FROM ${table};";
+public static final String PARTITION_MAX_SIZE_SQL = "SELECT
MAX(LENGTH(${column})) AS max_size FROM"
++ " ${table} PARTITION (${partition});";
+
+public static final String AVG_SIZE_SQL = "SELECT AVG(LENGTH(${column}))
AS avg_size FROM ${table};";
+public static final String PARTITION_AVG_SIZE_SQL = "SELECT
AVG(LENGTH(${column})) AS avg_size"
++ " FROM ${table} PARTITION (${partition});";
+
+public static final String MAX_AVG_SIZE_SQL = "SELECT
MAX(LENGTH(${column})) AS max_size,"
++ " AVG(LENGTH(${column})) AS avg_size FROM ${table};";
+public static final String PARTITION_MAX_AVG_SIZE_SQL = "SELECT
MAX(LENGTH(${column}))"
++ " AS max_size, AVG(LENGTH(${column})) AS avg_size FROM ${table}
PARTITION (${partition});";
+
+public static final String NUM_NULLS_SQL = "SELECT COUNT(1) AS num_nulls
FROM ${table}"
++ " WHERE ${column} IS NULL;";
+public static final String PARTITION_NUM_NULLS_SQL = "SELECT COUNT(1) AS
num_nulls FROM"
++ " ${table} PARTITION (${partition}) WHERE ${column} IS NULL;";
+/** for statistics end
*/
+
+private static final Logger LOG =
LogManager.getLogger(InternalSqlTemplate.class);
+
+private static final Pattern PATTERN = Pattern.compile("\\$\\{\\w+\\}");
+
+/**
+ * Concatenate SQL statements based on templates and parameters. e.g.:
+ * template and parameters:
+ * 'SELECT ${col} FROM ${table} WHERE id = ${id};',
+ * parameters: {col=colName, table=tableName, id=1}
+ * result sql: 'SELECT colName FROM tableName WHERE id = 1;
+ *
+ *
+ * @param template sql template
+ *
[GitHub] [doris] github-actions[bot] commented on pull request #12712: [fix](intersect) fix intersect query failed in row storage code
github-actions[bot] commented on PR #12712: URL: https://github.com/apache/doris/pull/12712#issuecomment-1250530146 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12712: [fix](intersect) fix intersect query failed in row storage code
github-actions[bot] commented on PR #12712: URL: https://github.com/apache/doris/pull/12712#issuecomment-1250530164 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12602: [refactor](file scanner)Refactor file scanner.
github-actions[bot] commented on PR #12602: URL: https://github.com/apache/doris/pull/12602#issuecomment-1250530498 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12602: [refactor](file scanner)Refactor file scanner.
github-actions[bot] commented on PR #12602: URL: https://github.com/apache/doris/pull/12602#issuecomment-1250530521 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #12714: [feature-wip](statistics) Template for building internal query SQL statements
Kikyou1997 commented on code in PR #12714:
URL: https://github.com/apache/doris/pull/12714#discussion_r973842476
##
fe/fe-core/src/main/java/org/apache/doris/statistics/util/InternalSqlTemplate.java:
##
@@ -0,0 +1,282 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.statistics.util;
+
+import org.apache.doris.common.InvalidFormatException;
+
+import com.google.common.collect.Sets;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Template for building internal query SQL statements.
+ * e.g.:
+ * - "SELECT ${column} FROM ${table}"
+ * - ${column} and ${table} will be replaced with the actual executed table
and column.
+ */
+public class InternalSqlTemplate {
+/** -- for statistics begin
-- */
+public static final String MIN_VALUE_SQL = "SELECT MIN(${column}) AS
min_value FROM ${table};";
+public static final String PARTITION_MIN_VALUE_SQL = "SELECT
MIN(${column}) AS min_value"
++ " FROM ${table} PARTITION (${partition});";
+
+public static final String MAX_VALUE_SQL = "SELECT MAX(${column}) AS
max_value FROM ${table};";
+public static final String PARTITION_MAX_VALUE_SQL = "SELECT
MAX(${column}) AS max_value FROM"
++ " ${table} PARTITION (${partition});";
+
+public static final String NDV_VALUE_SQL = "SELECT NDV(${column}) AS ndv
FROM ${table};";
+public static final String PARTITION_NDV_VALUE_SQL = "SELECT
NDV(${column}) AS ndv FROM"
++ " ${table} PARTITION (${partition});";
+
+public static final String MIN_MAX_NDV_VALUE_SQL = "SELECT MIN(${column})
AS min_value, MAX(${column})"
++ " AS max_value, NDV(${column}) AS ndv FROM ${table};";
+public static final String PARTITION_MIN_MAX_NDV_VALUE_SQL = "SELECT
MIN(${column}) AS min_value,"
++ " MAX(${column}) AS max_value, NDV(${column}) AS ndv FROM
${table} PARTITION (${partition});";
+
+public static final String ROW_COUNT_SQL = "SELECT COUNT(1) AS row_count
FROM ${table};";
+public static final String PARTITION_ROW_COUNT_SQL = "SELECT COUNT(1) AS
row_count FROM ${table} PARTITION"
++ " (${partition});";
+
+public static final String MAX_SIZE_SQL = "SELECT MAX(LENGTH(${column}))
AS max_size FROM ${table};";
+public static final String PARTITION_MAX_SIZE_SQL = "SELECT
MAX(LENGTH(${column})) AS max_size FROM"
++ " ${table} PARTITION (${partition});";
+
+public static final String AVG_SIZE_SQL = "SELECT AVG(LENGTH(${column}))
AS avg_size FROM ${table};";
+public static final String PARTITION_AVG_SIZE_SQL = "SELECT
AVG(LENGTH(${column})) AS avg_size"
++ " FROM ${table} PARTITION (${partition});";
+
+public static final String MAX_AVG_SIZE_SQL = "SELECT
MAX(LENGTH(${column})) AS max_size,"
++ " AVG(LENGTH(${column})) AS avg_size FROM ${table};";
+public static final String PARTITION_MAX_AVG_SIZE_SQL = "SELECT
MAX(LENGTH(${column}))"
++ " AS max_size, AVG(LENGTH(${column})) AS avg_size FROM ${table}
PARTITION (${partition});";
+
+public static final String NUM_NULLS_SQL = "SELECT COUNT(1) AS num_nulls
FROM ${table}"
++ " WHERE ${column} IS NULL;";
+public static final String PARTITION_NUM_NULLS_SQL = "SELECT COUNT(1) AS
num_nulls FROM"
++ " ${table} PARTITION (${partition}) WHERE ${column} IS NULL;";
+/** for statistics end
*/
+
+private static final Logger LOG =
LogManager.getLogger(InternalSqlTemplate.class);
+
+private static final Pattern PATTERN = Pattern.compile("\\$\\{\\w+\\}");
+
+/**
+ * Concatenate SQL statements based on templates and parameters. e.g.:
+ * template and parameters:
+ * 'SELECT ${col} FROM ${table} WHERE id = ${id};',
+ * parameters: {col=colName, table=tableName, id=1}
+ * result sql: 'SELECT colName FROM tableName WHERE id = 1;
+ *
+ *
+ * @param template sql template
+ *
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #12714: [feature-wip](statistics) Template for building internal query SQL statements
Kikyou1997 commented on code in PR #12714:
URL: https://github.com/apache/doris/pull/12714#discussion_r973842476
##
fe/fe-core/src/main/java/org/apache/doris/statistics/util/InternalSqlTemplate.java:
##
@@ -0,0 +1,282 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.statistics.util;
+
+import org.apache.doris.common.InvalidFormatException;
+
+import com.google.common.collect.Sets;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Template for building internal query SQL statements.
+ * e.g.:
+ * - "SELECT ${column} FROM ${table}"
+ * - ${column} and ${table} will be replaced with the actual executed table
and column.
+ */
+public class InternalSqlTemplate {
+/** -- for statistics begin
-- */
+public static final String MIN_VALUE_SQL = "SELECT MIN(${column}) AS
min_value FROM ${table};";
+public static final String PARTITION_MIN_VALUE_SQL = "SELECT
MIN(${column}) AS min_value"
++ " FROM ${table} PARTITION (${partition});";
+
+public static final String MAX_VALUE_SQL = "SELECT MAX(${column}) AS
max_value FROM ${table};";
+public static final String PARTITION_MAX_VALUE_SQL = "SELECT
MAX(${column}) AS max_value FROM"
++ " ${table} PARTITION (${partition});";
+
+public static final String NDV_VALUE_SQL = "SELECT NDV(${column}) AS ndv
FROM ${table};";
+public static final String PARTITION_NDV_VALUE_SQL = "SELECT
NDV(${column}) AS ndv FROM"
++ " ${table} PARTITION (${partition});";
+
+public static final String MIN_MAX_NDV_VALUE_SQL = "SELECT MIN(${column})
AS min_value, MAX(${column})"
++ " AS max_value, NDV(${column}) AS ndv FROM ${table};";
+public static final String PARTITION_MIN_MAX_NDV_VALUE_SQL = "SELECT
MIN(${column}) AS min_value,"
++ " MAX(${column}) AS max_value, NDV(${column}) AS ndv FROM
${table} PARTITION (${partition});";
+
+public static final String ROW_COUNT_SQL = "SELECT COUNT(1) AS row_count
FROM ${table};";
+public static final String PARTITION_ROW_COUNT_SQL = "SELECT COUNT(1) AS
row_count FROM ${table} PARTITION"
++ " (${partition});";
+
+public static final String MAX_SIZE_SQL = "SELECT MAX(LENGTH(${column}))
AS max_size FROM ${table};";
+public static final String PARTITION_MAX_SIZE_SQL = "SELECT
MAX(LENGTH(${column})) AS max_size FROM"
++ " ${table} PARTITION (${partition});";
+
+public static final String AVG_SIZE_SQL = "SELECT AVG(LENGTH(${column}))
AS avg_size FROM ${table};";
+public static final String PARTITION_AVG_SIZE_SQL = "SELECT
AVG(LENGTH(${column})) AS avg_size"
++ " FROM ${table} PARTITION (${partition});";
+
+public static final String MAX_AVG_SIZE_SQL = "SELECT
MAX(LENGTH(${column})) AS max_size,"
++ " AVG(LENGTH(${column})) AS avg_size FROM ${table};";
+public static final String PARTITION_MAX_AVG_SIZE_SQL = "SELECT
MAX(LENGTH(${column}))"
++ " AS max_size, AVG(LENGTH(${column})) AS avg_size FROM ${table}
PARTITION (${partition});";
+
+public static final String NUM_NULLS_SQL = "SELECT COUNT(1) AS num_nulls
FROM ${table}"
++ " WHERE ${column} IS NULL;";
+public static final String PARTITION_NUM_NULLS_SQL = "SELECT COUNT(1) AS
num_nulls FROM"
++ " ${table} PARTITION (${partition}) WHERE ${column} IS NULL;";
+/** for statistics end
*/
+
+private static final Logger LOG =
LogManager.getLogger(InternalSqlTemplate.class);
+
+private static final Pattern PATTERN = Pattern.compile("\\$\\{\\w+\\}");
+
+/**
+ * Concatenate SQL statements based on templates and parameters. e.g.:
+ * template and parameters:
+ * 'SELECT ${col} FROM ${table} WHERE id = ${id};',
+ * parameters: {col=colName, table=tableName, id=1}
+ * result sql: 'SELECT colName FROM tableName WHERE id = 1;
+ *
+ *
+ * @param template sql template
+ *
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #12714: [feature-wip](statistics) Template for building internal query SQL statements
Kikyou1997 commented on code in PR #12714:
URL: https://github.com/apache/doris/pull/12714#discussion_r973842691
##
fe/fe-core/src/main/java/org/apache/doris/statistics/util/InternalSqlTemplate.java:
##
@@ -0,0 +1,282 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.statistics.util;
+
+import org.apache.doris.common.InvalidFormatException;
+
+import com.google.common.collect.Sets;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Template for building internal query SQL statements.
+ * e.g.:
+ * - "SELECT ${column} FROM ${table}"
+ * - ${column} and ${table} will be replaced with the actual executed table
and column.
+ */
+public class InternalSqlTemplate {
+/** -- for statistics begin
-- */
+public static final String MIN_VALUE_SQL = "SELECT MIN(${column}) AS
min_value FROM ${table};";
+public static final String PARTITION_MIN_VALUE_SQL = "SELECT
MIN(${column}) AS min_value"
++ " FROM ${table} PARTITION (${partition});";
+
+public static final String MAX_VALUE_SQL = "SELECT MAX(${column}) AS
max_value FROM ${table};";
+public static final String PARTITION_MAX_VALUE_SQL = "SELECT
MAX(${column}) AS max_value FROM"
++ " ${table} PARTITION (${partition});";
+
+public static final String NDV_VALUE_SQL = "SELECT NDV(${column}) AS ndv
FROM ${table};";
+public static final String PARTITION_NDV_VALUE_SQL = "SELECT
NDV(${column}) AS ndv FROM"
++ " ${table} PARTITION (${partition});";
+
+public static final String MIN_MAX_NDV_VALUE_SQL = "SELECT MIN(${column})
AS min_value, MAX(${column})"
++ " AS max_value, NDV(${column}) AS ndv FROM ${table};";
+public static final String PARTITION_MIN_MAX_NDV_VALUE_SQL = "SELECT
MIN(${column}) AS min_value,"
++ " MAX(${column}) AS max_value, NDV(${column}) AS ndv FROM
${table} PARTITION (${partition});";
+
+public static final String ROW_COUNT_SQL = "SELECT COUNT(1) AS row_count
FROM ${table};";
+public static final String PARTITION_ROW_COUNT_SQL = "SELECT COUNT(1) AS
row_count FROM ${table} PARTITION"
++ " (${partition});";
+
+public static final String MAX_SIZE_SQL = "SELECT MAX(LENGTH(${column}))
AS max_size FROM ${table};";
+public static final String PARTITION_MAX_SIZE_SQL = "SELECT
MAX(LENGTH(${column})) AS max_size FROM"
++ " ${table} PARTITION (${partition});";
+
+public static final String AVG_SIZE_SQL = "SELECT AVG(LENGTH(${column}))
AS avg_size FROM ${table};";
+public static final String PARTITION_AVG_SIZE_SQL = "SELECT
AVG(LENGTH(${column})) AS avg_size"
++ " FROM ${table} PARTITION (${partition});";
+
+public static final String MAX_AVG_SIZE_SQL = "SELECT
MAX(LENGTH(${column})) AS max_size,"
++ " AVG(LENGTH(${column})) AS avg_size FROM ${table};";
+public static final String PARTITION_MAX_AVG_SIZE_SQL = "SELECT
MAX(LENGTH(${column}))"
++ " AS max_size, AVG(LENGTH(${column})) AS avg_size FROM ${table}
PARTITION (${partition});";
+
+public static final String NUM_NULLS_SQL = "SELECT COUNT(1) AS num_nulls
FROM ${table}"
++ " WHERE ${column} IS NULL;";
+public static final String PARTITION_NUM_NULLS_SQL = "SELECT COUNT(1) AS
num_nulls FROM"
++ " ${table} PARTITION (${partition}) WHERE ${column} IS NULL;";
+/** for statistics end
*/
+
+private static final Logger LOG =
LogManager.getLogger(InternalSqlTemplate.class);
+
+private static final Pattern PATTERN = Pattern.compile("\\$\\{\\w+\\}");
+
+/**
+ * Concatenate SQL statements based on templates and parameters. e.g.:
+ * template and parameters:
+ * 'SELECT ${col} FROM ${table} WHERE id = ${id};',
+ * parameters: {col=colName, table=tableName, id=1}
+ * result sql: 'SELECT colName FROM tableName WHERE id = 1;
+ *
+ *
+ * @param template sql template
+ *
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #12714: [feature-wip](statistics) Template for building internal query SQL statements
Kikyou1997 commented on code in PR #12714:
URL: https://github.com/apache/doris/pull/12714#discussion_r973842691
##
fe/fe-core/src/main/java/org/apache/doris/statistics/util/InternalSqlTemplate.java:
##
@@ -0,0 +1,282 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.statistics.util;
+
+import org.apache.doris.common.InvalidFormatException;
+
+import com.google.common.collect.Sets;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Template for building internal query SQL statements.
+ * e.g.:
+ * - "SELECT ${column} FROM ${table}"
+ * - ${column} and ${table} will be replaced with the actual executed table
and column.
+ */
+public class InternalSqlTemplate {
+/** -- for statistics begin
-- */
+public static final String MIN_VALUE_SQL = "SELECT MIN(${column}) AS
min_value FROM ${table};";
+public static final String PARTITION_MIN_VALUE_SQL = "SELECT
MIN(${column}) AS min_value"
++ " FROM ${table} PARTITION (${partition});";
+
+public static final String MAX_VALUE_SQL = "SELECT MAX(${column}) AS
max_value FROM ${table};";
+public static final String PARTITION_MAX_VALUE_SQL = "SELECT
MAX(${column}) AS max_value FROM"
++ " ${table} PARTITION (${partition});";
+
+public static final String NDV_VALUE_SQL = "SELECT NDV(${column}) AS ndv
FROM ${table};";
+public static final String PARTITION_NDV_VALUE_SQL = "SELECT
NDV(${column}) AS ndv FROM"
++ " ${table} PARTITION (${partition});";
+
+public static final String MIN_MAX_NDV_VALUE_SQL = "SELECT MIN(${column})
AS min_value, MAX(${column})"
++ " AS max_value, NDV(${column}) AS ndv FROM ${table};";
+public static final String PARTITION_MIN_MAX_NDV_VALUE_SQL = "SELECT
MIN(${column}) AS min_value,"
++ " MAX(${column}) AS max_value, NDV(${column}) AS ndv FROM
${table} PARTITION (${partition});";
+
+public static final String ROW_COUNT_SQL = "SELECT COUNT(1) AS row_count
FROM ${table};";
+public static final String PARTITION_ROW_COUNT_SQL = "SELECT COUNT(1) AS
row_count FROM ${table} PARTITION"
++ " (${partition});";
+
+public static final String MAX_SIZE_SQL = "SELECT MAX(LENGTH(${column}))
AS max_size FROM ${table};";
+public static final String PARTITION_MAX_SIZE_SQL = "SELECT
MAX(LENGTH(${column})) AS max_size FROM"
++ " ${table} PARTITION (${partition});";
+
+public static final String AVG_SIZE_SQL = "SELECT AVG(LENGTH(${column}))
AS avg_size FROM ${table};";
+public static final String PARTITION_AVG_SIZE_SQL = "SELECT
AVG(LENGTH(${column})) AS avg_size"
++ " FROM ${table} PARTITION (${partition});";
+
+public static final String MAX_AVG_SIZE_SQL = "SELECT
MAX(LENGTH(${column})) AS max_size,"
++ " AVG(LENGTH(${column})) AS avg_size FROM ${table};";
+public static final String PARTITION_MAX_AVG_SIZE_SQL = "SELECT
MAX(LENGTH(${column}))"
++ " AS max_size, AVG(LENGTH(${column})) AS avg_size FROM ${table}
PARTITION (${partition});";
+
+public static final String NUM_NULLS_SQL = "SELECT COUNT(1) AS num_nulls
FROM ${table}"
++ " WHERE ${column} IS NULL;";
+public static final String PARTITION_NUM_NULLS_SQL = "SELECT COUNT(1) AS
num_nulls FROM"
++ " ${table} PARTITION (${partition}) WHERE ${column} IS NULL;";
+/** for statistics end
*/
+
+private static final Logger LOG =
LogManager.getLogger(InternalSqlTemplate.class);
+
+private static final Pattern PATTERN = Pattern.compile("\\$\\{\\w+\\}");
+
+/**
+ * Concatenate SQL statements based on templates and parameters. e.g.:
+ * template and parameters:
+ * 'SELECT ${col} FROM ${table} WHERE id = ${id};',
+ * parameters: {col=colName, table=tableName, id=1}
+ * result sql: 'SELECT colName FROM tableName WHERE id = 1;
+ *
+ *
+ * @param template sql template
+ *
[GitHub] [doris] liaoxin01 opened a new pull request, #12721: [feature-wip](unique-key-merge-on-write) fix that incremental clone may lead to loss of delete bitmap
liaoxin01 opened a new pull request, #12721: URL: https://github.com/apache/doris/pull/12721 # Proposed changes Issue Number: close #xxx ## Problem summary In the original implementation, there are two saves of meta. If the process is killed between two saves, the delete bitmap may be lost. So we calculate delete bitmap firstly, then save meta. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yangzhg closed issue #12711: [Bug] intersect query failed
yangzhg closed issue #12711: [Bug] intersect query failed URL: https://github.com/apache/doris/issues/12711 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yangzhg merged pull request #12712: [fix](intersect) fix intersect query failed in row storage code
yangzhg merged PR #12712: URL: https://github.com/apache/doris/pull/12712 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](intersect) fix intersect query failed in row storage code (#12712)
This is an automated email from the ASF dual-hosted git repository.
yangzhg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d68b8cce1a [fix](intersect) fix intersect query failed in row storage
code (#12712)
d68b8cce1a is described below
commit d68b8cce1a540d013de8b5a2d1909af1f0ffefe5
Author: luozenglin <37725793+luozeng...@users.noreply.github.com>
AuthorDate: Mon Sep 19 11:47:50 2022 +0800
[fix](intersect) fix intersect query failed in row storage code (#12712)
---
be/src/exec/set_operation_node.cpp | 8
be/src/exec/set_operation_node.h| 1 -
regression-test/data/query_p0/intersect/test_intersect.out | 5 +
regression-test/suites/query_p0/intersect/test_intersect.groovy | 7 +++
4 files changed, 12 insertions(+), 9 deletions(-)
diff --git a/be/src/exec/set_operation_node.cpp
b/be/src/exec/set_operation_node.cpp
index 882019b199..3458c57293 100644
--- a/be/src/exec/set_operation_node.cpp
+++ b/be/src/exec/set_operation_node.cpp
@@ -54,14 +54,6 @@ Status SetOperationNode::prepare(RuntimeState* state) {
}
_build_tuple_size = child(0)->row_desc().tuple_descriptors().size();
_build_tuple_row_size = _build_tuple_size * sizeof(Tuple*);
-_build_tuple_idx.reserve(_build_tuple_size);
-
-for (int i = 0; i < _build_tuple_size; ++i) {
-TupleDescriptor* build_tuple_desc =
child(0)->row_desc().tuple_descriptors()[i];
-auto tuple_idx = _row_descriptor.get_tuple_idx(build_tuple_desc->id());
-RETURN_IF_INVALID_TUPLE_IDX(build_tuple_desc->id(), tuple_idx);
-_build_tuple_idx.push_back(tuple_idx);
-}
_find_nulls = std::vector();
for (auto ctx : _child_expr_lists[0]) {
_find_nulls.push_back(!ctx->root()->is_slotref() ||
ctx->is_nullable());
diff --git a/be/src/exec/set_operation_node.h b/be/src/exec/set_operation_node.h
index 2d5900345f..4e2722b94d 100644
--- a/be/src/exec/set_operation_node.h
+++ b/be/src/exec/set_operation_node.h
@@ -71,7 +71,6 @@ protected:
// holds everything referenced in _hash_tbl
std::unique_ptr _build_pool;
-std::vector _build_tuple_idx;
int _build_tuple_size;
int _build_tuple_row_size;
std::vector _find_nulls;
diff --git a/regression-test/data/query_p0/intersect/test_intersect.out
b/regression-test/data/query_p0/intersect/test_intersect.out
index 4c443b8493..698b8f69f3 100644
--- a/regression-test/data/query_p0/intersect/test_intersect.out
+++ b/regression-test/data/query_p0/intersect/test_intersect.out
@@ -4,3 +4,8 @@
2
3
+-- !select --
+1
+2
+3
+
diff --git a/regression-test/suites/query_p0/intersect/test_intersect.groovy
b/regression-test/suites/query_p0/intersect/test_intersect.groovy
index 1c007b95d7..677ed5c514 100644
--- a/regression-test/suites/query_p0/intersect/test_intersect.groovy
+++ b/regression-test/suites/query_p0/intersect/test_intersect.groovy
@@ -16,6 +16,13 @@
// under the License.
suite("test_intersect") {
+sql 'set enable_vectorized_engine=false'
+qt_select """
+SELECT * FROM (SELECT k1 FROM test_query_db.baseall
+INTERSECT SELECT k1 FROM test_query_db.test) a ORDER BY k1
+ """
+
+sql 'set enable_vectorized_engine=true'
qt_select """
SELECT * FROM (SELECT k1 FROM test_query_db.baseall
INTERSECT SELECT k1 FROM test_query_db.test) a ORDER BY k1
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 opened a new issue, #12722: [Bug] Lost precision when using datetimev2
Gabriel39 opened a new issue, #12722: URL: https://github.com/apache/doris/issues/12722 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? Lost precision ### What You Expected? work well ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 opened a new pull request, #12723: [Bug](datetimev2) Fix lost precision for datetimev2
Gabriel39 opened a new pull request, #12723: URL: https://github.com/apache/doris/pull/12723 # Proposed changes Issue Number: close #12722 ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xia5271 opened a new issue, #12724: [Feature] java client支持
xia5271 opened a new issue, #12724: URL: https://github.com/apache/doris/issues/12724 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description 是否有计划提供一个标准java client？满足以下能力。 目前jdbc只能满足DDL、metadata查询、数据查询，对于数据平台常用的像大规模的数据导入、导出，没有一个标准、最佳实践，导致像spark、flink、datax等各自实现一套轮子，用户体验不好，也限制了生态成长 - 提供DDL能力 - 提供metadata查询能力 - 数据查询 - 提供数据导入、导出能力 - 其他，如查询分析、运维能力等 ### Use case - 计算引擎spark、flink等快速对接，大规模数据导入、导出、join等 - BI等查询 - 数据平台、数据中台的分析、运维能力快速对接 ### Related issues _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on pull request #9983: [feature-wip](statistics) Internal-query, execute SQL query statement internally (in FE)
Kikyou1997 commented on PR #9983: URL: https://github.com/apache/doris/pull/9983#issuecomment-1250538167 @morrySnow review plz -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on pull request #12714: [feature-wip](statistics) Template for building internal query SQL statements
Kikyou1997 commented on PR #12714: URL: https://github.com/apache/doris/pull/12714#issuecomment-1250538244 review plz @morrySnow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hf200012 commented on pull request #12238: [bugfix](join) fix coredump when build block size is 0
hf200012 commented on PR #12238:
URL: https://github.com/apache/doris/pull/12238#issuecomment-1250545854
```
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0919 11:28:12.714655 4271 env.cpp:46] Env init successfully.
*** Aborted at 1663558219 (unix time) try "date -d @1663558219" if you are
using GNU date ***
*** SIGSEGV unkown detail explain (@0x0) received by PID 4271 (TID
0x7ff098311700) from PID 0; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int,
siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:420
1# 0x7FF0DC129400 in /lib64/libc.so.6
2#
COW::intrusive_ptr::operator->() const at /root/doris/be/src/vec/common/cow.h:210
3# void
doris::vectorized::ProcessHashTableProbe,
false>::build_side_output_column(std::vector::mutable_ptr,
std::allocator::mutable_ptr
> >&, int, int, std::vector > const&, int) at
/root/doris/be/src/vec/exec/join/vhash_join_node.cpp:208
4#
doris::vectorized::ProcessHashTableProbe,
false>::do_process(doris::vectorized::SerializedHashTableContext&,
doris::vectorized::PODArray,
15ul, 16ul> const*, doris::vectorized::MutableBlock&,
doris::vectorized::Block*) at
/root/doris/be/src/vec/exec/join/vhash_join_node.cpp:399
5# auto doris::vectorized::HashJoinNode::get_next(doris::RuntimeState*,
doris::vectorized::Block*, bool*)::{lambda(auto:1&&, auto:2&&, auto:3,
auto:4)#2}::operator()&,
std::integral_constant, std::integral_constant
>(doris::vectorized::SerializedHashTableContext&,
std::integral_constant&,
std::integral_constant, std::integral_constant) const
at /root/doris/be/src/vec/exec/join/vhash_join_node.cpp:955
6# void std::__invoke_impl&,
std::integral_constant, std::integral_constant
>(std::__invoke_other,
doris::vectorized::HashJoinNode::get_next(doris::RuntimeState*,
doris::vectorized::Block*, bool*)::{lambda(auto:1&&, auto:2&&, auto:3,
auto:4)#2}&&, doris::vectorized::SerializedHashTableContext&,
std::integral_constant&,
std::integral_constant&&, std::integral_constant&&)
at /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:61
7#
std::__invoke_result&,
std::integral_constant, std::integral_constant
>::type
std::__invoke&,
std::integral_constant, std::integral_constant
>(doris::vectorized::HashJoinNode::get_next(doris::RuntimeState*,
doris::vectorized::Block*, bool*)::{lambda(auto:1&&, auto:2&&, auto:3,
auto:4)#2}&&, doris::vectorized::SerializedHashTableContext&,
std::integral_constant
&, std::integral_constant&&, std::integral_constant&&) at /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:97
8#
std::__detail::__variant::__gen_vtable_impl
(*)(doris::vectorized::HashJoinNode::get_next(doris::RuntimeState*,
doris::vectorized::Block*, bool*)::{lambda(auto:1&&, auto:2&&, auto:3,
auto:4)#2}&&, std::variant,
doris::vectorized::PrimaryTypeHashTableContext,
doris::vectorized::PrimaryTypeHashTableContext,
doris::vectorized::PrimaryTypeHashTableContext,
doris::vectorized::PrimaryTypeHashTableContext,
doris::vectorized::PrimaryTypeHashTableContext,
doris::vectorized::FixedKeyHashTableContext,
doris::vectorized::FixedKeyHashTableContext,
doris::vectorized::FixedKeyHashTableContext,
doris::ve
ctorized::FixedKeyHashTableContext,
doris::vectorized::FixedKeyHashTableContext,
doris::vectorized::FixedKeyHashTableContext
>&, std::variant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant >&, std::variant, std::integral_constant >&&,
std::variant, std::integral_constant >&&)>, std::integer_sequence
>::__visit_invoke(doris::vectorized::HashJoinNode::get_next(doris::RuntimeState*,
doris::vectorized::Block*, bool*)::{lambda(auto:1&&, auto:2&&, auto:3,
auto:4)#2}&&, std::variant,
doris::vectorized::PrimaryTypeHashTableContext,
doris::vectorized::PrimaryTypeHashTableContext,
doris::vectorized::PrimaryTypeHashTableContext,
doris::vectorized::PrimaryTypeHashTableContext,
doris::vectorized::PrimaryTypeHashTableContext,
doris::vectorized::FixedKeyHashTableContext,
doris::vectorized::FixedKeyHashTableContext,
doris::vectoriz
ed::FixedKeyHashTableContext,
doris::vectorized::FixedKeyHashTableContext,
doris::vectorized::FixedKeyHashTableContext,
doris::vectorized::FixedKeyHashTableContext
>&, std::variant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant >&, std::variant, std::integral_constant >&&,
std::variant, std::integral_constant >&&) at /var/local/ldb-toolchain/include/c++/11/variant:1015
9# decltype(auto)
std::__do_visit,
doris::vectorized::HashJoinNode::get_next(doris::RuntimeState*,
doris::vectorized::Block*, bool*)::{lambda(auto:1&&, auto:2&&, auto:3,
auto:4)#
[GitHub] [doris] github-actions[bot] commented on pull request #12714: [feature-wip](statistics) Template for building internal query SQL statements
github-actions[bot] commented on PR #12714: URL: https://github.com/apache/doris/pull/12714#issuecomment-1250548059 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #9983: [feature-wip](statistics) Internal-query, execute SQL query statement internally (in FE)
github-actions[bot] commented on PR #9983: URL: https://github.com/apache/doris/pull/9983#issuecomment-1250548525 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on pull request #12714: [feature-wip](statistics) Template for building internal query SQL statements
Kikyou1997 commented on PR #12714: URL: https://github.com/apache/doris/pull/12714#issuecomment-1250549148 Besides, I'm not sure whether simply use regex to do the mach and replace would lead to the SQL injection or this problem could be tolerant since Doris is normaly used as a internal service -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #12700: [Improvement](sort) Use heap sort to optimize sort node
HappenLee commented on code in PR #12700:
URL: https://github.com/apache/doris/pull/12700#discussion_r973839470
##
be/src/vec/columns/column.h:
##
@@ -402,6 +402,10 @@ class IColumn : public COW {
virtual int compare_at(size_t n, size_t m, const IColumn& rhs,
int nan_direction_hint) const = 0;
+virtual void compare_internal(size_t rhs_row_id, const IColumn& rhs, int
nan_direction_hint,
Review Comment:
Add a commen for the function `compare_internal`
##
be/src/vec/columns/column_decimal.cpp:
##
@@ -381,6 +381,32 @@ void ColumnDecimal::sort_column(const ColumnSorter*
sorter, EqualFlags& flags
sorter->template sort_column(static_cast(*this), flags,
perms, range, last_column);
}
+template
+void ColumnDecimal::compare_internal(size_t rhs_row_id, const IColumn& rhs,
+int nan_direction_hint, int direction,
+std::vector& cmp_res,
+uint8* __restrict filter) const {
+auto sz = this->size();
+DCHECK(cmp_res.size() == sz);
+const auto& cmp_base = assert_cast&>(rhs).get_data()[rhs_row_id];
+
+size_t begin = simd::find_zero(cmp_res, 0);
+while (begin < sz) {
+size_t end = simd::find_nonzero(cmp_res, begin + 1);
Review Comment:
we maybe need a better name for `find_nonzero`, find_one ?
##
be/src/vec/core/sort_cursor.h:
##
@@ -21,43 +21,123 @@
#pragma once
#include "vec/columns/column.h"
-#include "vec/columns/column_string.h"
-#include "vec/common/assert_cast.h"
-#include "vec/common/typeid_cast.h"
#include "vec/core/block.h"
-#include "vec/core/column_numbers.h"
#include "vec/core/sort_description.h"
#include "vec/exprs/vexpr_context.h"
-#include "vec/runtime/vdata_stream_recvr.h"
namespace doris::vectorized {
+struct HeapSortCursorBlockView {
+public:
+Block block;
+ColumnRawPtrs sort_columns;
+SortDescription& desc;
+
+HeapSortCursorBlockView(Block&& cur_block, SortDescription& sort_desc)
+: block(cur_block), desc(sort_desc) {
+_reset();
+}
+
+void filter_block(IColumn::Filter& filter) {
+Block::filter_block_internal(&block, filter, block.columns());
+_reset();
+}
+
+private:
+void _reset() {
+sort_columns.clear();
+auto columns = block.get_columns();
+for (size_t j = 0, size = desc.size(); j < size; ++j) {
+auto& column_desc = desc[j];
+size_t column_number = !column_desc.column_name.empty()
+ ?
block.get_position_by_name(column_desc.column_name)
+ : column_desc.column_number;
+sort_columns.push_back(columns[column_number].get());
+}
+}
+};
+
+struct HeapSortCursorImpl {
+public:
+HeapSortCursorImpl() : _row_id(0), _block_view(nullptr) {}
+HeapSortCursorImpl(int row_id, std::shared_ptr
block_view)
+: _row_id(row_id), _block_view(block_view) {}
+
+HeapSortCursorImpl(const HeapSortCursorImpl& other) {
+_row_id = other._row_id;
+_block_view = other._block_view;
+}
+
+HeapSortCursorImpl(HeapSortCursorImpl&& other) {
+_row_id = other._row_id;
+_block_view = std::move(other._block_view);
+}
+
+HeapSortCursorImpl& operator=(HeapSortCursorImpl&& other) {
+std::swap(_row_id, other._row_id);
+std::swap(_block_view, other._block_view);
+return *this;
+}
+
+~HeapSortCursorImpl() = default;
+const size_t row_id() const { return _row_id; }
+
+const ColumnRawPtrs& sort_columns() const { return
_block_view->sort_columns; }
+
+const Block* block() const { return &_block_view->block; }
+
+const SortDescription& sort_desc() const { return _block_view->desc; }
+
+bool operator<(const HeapSortCursorImpl& rhs) const {
+for (size_t i = 0; i < sort_desc().size(); ++i) {
+int direction = sort_desc()[i].direction;
+int nulls_direction = sort_desc()[i].nulls_direction;
+int res = direction * sort_columns()[i]->compare_at(row_id(),
rhs.row_id(),
+
*(rhs.sort_columns()[i]),
+
nulls_direction);
+// ASC: direction == 1. If bigger, res > 0. So we return true.
+if (res < 0) {
+return true;
+}
+if (res > 0) {
+return false;
+}
+}
+return false;
+}
+
+private:
+size_t _row_id;
+std::shared_ptr _block_view;
Review Comment:
Smart pointers introduce additional memory and performance overhead,
consider whether we can circumvent its use with other mechanisms to gain
performance gains
##
be/src/vec/common/sort/topn_sorter.cpp:
#
