[GitHub] [doris] morningman commented on a diff in pull request #13096: [DOC](storage policy) add cold and hot separation docs
morningman commented on code in PR #13096:
URL: https://github.com/apache/doris/pull/13096#discussion_r990960425
##
docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-RESOURCE.md:
##
@@ -50,6 +50,22 @@ ALTER RESOURCE 'spark0' PROPERTIES ("working_dir" =
"hdfs://127.0.0.1:1/tmp/
```sql
ALTER RESOURCE 'remote_s3' PROPERTIES ("s3_max_connections" = "100");
```
+3. 修改冷热分离S3资源相关信息
+- 支持修改项
+ - s3_max_connections s3最大连接数，默认50
Review Comment:
另
1. 需要在 `docs/sidebars.json` 中添加所有新增文档的链接，否则网站不会显示。
2. `docs/zh-CN/docs/advanced/cold_hot_separation.md` 需要有英文版本。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13229: Fully resolved Mysql external table issues
github-actions[bot] commented on PR #13229: URL: https://github.com/apache/doris/pull/13229#issuecomment-1272877136 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13229: Fully resolved Mysql external table issues
github-actions[bot] commented on PR #13229: URL: https://github.com/apache/doris/pull/13229#issuecomment-1272877178 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xiaojunjie opened a new pull request, #13244: [improvement](load) support function from_unixtime with specified timezome
xiaojunjie opened a new pull request, #13244: URL: https://github.com/apache/doris/pull/13244 # Proposed changes Issue Number: none ## Problem summary function from_unixtime use default fixed time zone when vectorized engine enable. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-flink-connector] dinggege1024 commented on a diff in pull request #71: [Enhancement] ADD RowSerializer for doris flink connector
dinggege1024 commented on code in PR #71:
URL:
https://github.com/apache/doris-flink-connector/pull/71#discussion_r990977828
##
flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/RowSerializer.java:
##
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.flink.sink.writer;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.doris.flink.deserialization.converter.DorisRowConverter;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.conversion.RowRowConverter;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.types.Row;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.StringJoiner;
+
+import static org.apache.doris.flink.sink.writer.LoadConstants.CSV;
+import static
org.apache.doris.flink.sink.writer.LoadConstants.DORIS_DELETE_SIGN;
+import static org.apache.doris.flink.sink.writer.LoadConstants.JSON;
+import static org.apache.doris.flink.sink.writer.LoadConstants.NULL_VALUE;
+
+/**
+ * Serializer for {@link Row}.
+ */
+public class RowSerializer implements DorisRecordSerializer {
+String[] fieldNames;
+String type;
+private ObjectMapper objectMapper;
+private final String fieldDelimiter;
+private final boolean enableDelete;
+private final DorisRowConverter rowConverter;
+private final DataType[] dataTypes;
+
+private RowSerializer(String[] fieldNames, DataType[] dataTypes, String
type, String fieldDelimiter, boolean enableDelete) {
+this.fieldNames = fieldNames;
+this.dataTypes = dataTypes;
+this.type = type;
+this.fieldDelimiter = fieldDelimiter;
+this.enableDelete = enableDelete;
+if (JSON.equals(type)) {
+objectMapper = new ObjectMapper();
+}
+this.rowConverter = new DorisRowConverter(dataTypes);
+}
+
+@Override
+public byte[] serialize(Row record) throws IOException{
+RowData rowDataRecord =
RowRowConverter.create(DataTypes.ROW(dataTypes)).toInternal(record);
+int maxIndex = Math.min(record.getArity(), fieldNames.length);
+String valString;
+if (JSON.equals(type)) {
+valString = buildJsonString(rowDataRecord, maxIndex);
+} else if (CSV.equals(type)) {
+valString = buildCSVString(rowDataRecord, maxIndex);
+} else {
+throw new IllegalArgumentException("The type " + type + " is not
supported!");
+}
+return valString.getBytes(StandardCharsets.UTF_8);
+}
Review Comment:
> How about calling `RowDataSerializer.serialize(record)` directly?
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xiaokang commented on a diff in pull request #13129: jsonb parse function and load
xiaokang commented on code in PR #13129:
URL: https://github.com/apache/doris/pull/13129#discussion_r990984939
##
fe/fe-core/src/main/java/org/apache/doris/planner/LoadScanNode.java:
##
@@ -206,7 +206,23 @@ protected void finalizeParams(Map
slotDescByName,
expr = new ArithmeticExpr(ArithmeticExpr.Operator.MULTIPLY,
expr, new IntLiteral(-1));
expr.analyze(analyzer);
}
-expr = castToSlot(destSlotDesc, expr);
+
+PrimitiveType dstType = destSlotDesc.getType().getPrimitiveType();
+PrimitiveType srcType = expr.getType().getPrimitiveType();
+if (srcType == PrimitiveType.VARCHAR && dstType ==
PrimitiveType.JSONB) {
+List args = Lists.newArrayList();
+args.add(expr);
+String nullable = "notnull";
+if (destSlotDesc.getIsNullable() || expr.isNullable()) {
+nullable = "nullable";
+}
+String name = "jsonb_parse_" + nullable + "_error_to_invalid";
+expr = new FunctionCallExpr(name, args);
+expr.analyze(analyzer);
+System.out.println("xk debug use function " + name + " " +
expr);
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring opened a new pull request, #13245: [improvement](tcmalloc) increase tcmalloc upper limit to 90%
dataroaring opened a new pull request, #13245: URL: https://github.com/apache/doris/pull/13245 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt opened a new pull request, #13246: [Feature](runtime-filter) add runtime filter breaking change adapt
BiteThet opened a new pull request, #13246: URL: https://github.com/apache/doris/pull/13246 # Proposed changes make runtime filter breaking change(https://github.com/apache/doris/pull/12856) support rolling upgrade. ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-flink-connector] JNSimba commented on a diff in pull request #71: [Enhancement] ADD RowSerializer for doris flink connector
JNSimba commented on code in PR #71:
URL:
https://github.com/apache/doris-flink-connector/pull/71#discussion_r990999551
##
flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/RowSerializer.java:
##
@@ -0,0 +1,117 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.flink.sink.writer;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.conversion.RowRowConverter;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.types.Row;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+import java.io.IOException;
+import static org.apache.doris.flink.sink.writer.LoadConstants.CSV;
+import static org.apache.doris.flink.sink.writer.LoadConstants.JSON;
+
+/**
+ * Serializer for {@link Row}.
+ * Quick way to support RowSerializer on existing code
+ * TODO: support original Doris to Row serializer
+ */
+public class RowSerializer implements DorisRecordSerializer {
+/**
+ * converter {@link Row} to {@link RowData}
+ */
+private final RowRowConverter rowRowConverter;
+private final RowDataSerializer rowDataSerializer;
+
+private RowSerializer(String[] fieldNames, DataType[] dataTypes, String
type, String fieldDelimiter,
+ boolean enableDelete) {
+this.rowRowConverter =
RowRowConverter.create(DataTypes.ROW(dataTypes));
+this.rowDataSerializer = RowDataSerializer.builder()
+.setFieldNames(fieldNames)
+.setFieldType(dataTypes)
+.setType(type)
+.setFieldDelimiter(fieldDelimiter)
+.enableDelete(enableDelete)
+.build();
+}
+
+@Override
+public byte[] serialize(Row record) throws IOException{
+RowData rowDataRecord = this.rowRowConverter.toInternal(record);
+return this.rowDataSerializer.serialize(rowDataRecord);
+}
+
+public static Builder builder() {
+return new Builder();
+}
+
+public String parseDeleteSign(RowKind rowKind) {
+if (RowKind.INSERT.equals(rowKind) ||
RowKind.UPDATE_AFTER.equals(rowKind)) {
+return "0";
+} else if (RowKind.DELETE.equals(rowKind) ||
RowKind.UPDATE_BEFORE.equals(rowKind)) {
+return "1";
+} else {
+throw new IllegalArgumentException("Unrecognized row kind:" +
rowKind.toString());
+}
+}
Review Comment:
This function seems to be useless?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #13220: [Bug](replace function) fix be infinite loop and oom when use replace with an empty old str
dataroaring merged PR #13220: URL: https://github.com/apache/doris/pull/13220 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (e094e6ca71 -> 5757bbc9f3)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from e094e6ca71 [typo](docs)add hive-bitmap compile and package des #13237 add 5757bbc9f3 fix be oom when replace with an empty old str (#13220) No new revisions were added by this update. Summary of changes: be/src/vec/functions/function_string.h | 3 +++ .../query_p0/sql_functions/string_functions/test_string_function.out | 3 +++ .../sql_functions/string_functions/test_string_function.groovy | 1 + 3 files changed, 7 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #13234: [improvement](test) set default value of parallel config items to 10
dataroaring merged PR #13234: URL: https://github.com/apache/doris/pull/13234 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (5757bbc9f3 -> 0e3522c088)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 5757bbc9f3 fix be oom when replace with an empty old str (#13220) add 0e3522c088 [improvement](test) set default value of parallel config items to 10 (#13234) No new revisions were added by this update. Summary of changes: .../src/main/groovy/org/apache/doris/regression/Config.groovy | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 opened a new pull request, #13247: [fix](agg)count function should return 0 for null value
starocean999 opened a new pull request, #13247: URL: https://github.com/apache/doris/pull/13247 # Proposed changes Issue Number: close #xxx ## Problem summary count(null) should return 0 instead of 1, the streaming_agg_serialize_to_column function didn't handle if the input value is null, this pr fix it. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-flink-connector] dinggege1024 commented on a diff in pull request #71: [Enhancement] ADD RowSerializer for doris flink connector
dinggege1024 commented on code in PR #71:
URL:
https://github.com/apache/doris-flink-connector/pull/71#discussion_r991006903
##
flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/RowSerializer.java:
##
@@ -0,0 +1,117 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.flink.sink.writer;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.conversion.RowRowConverter;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.types.Row;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+import java.io.IOException;
+import static org.apache.doris.flink.sink.writer.LoadConstants.CSV;
+import static org.apache.doris.flink.sink.writer.LoadConstants.JSON;
+
+/**
+ * Serializer for {@link Row}.
+ * Quick way to support RowSerializer on existing code
+ * TODO: support original Doris to Row serializer
+ */
+public class RowSerializer implements DorisRecordSerializer {
+/**
+ * converter {@link Row} to {@link RowData}
+ */
+private final RowRowConverter rowRowConverter;
+private final RowDataSerializer rowDataSerializer;
+
+private RowSerializer(String[] fieldNames, DataType[] dataTypes, String
type, String fieldDelimiter,
+ boolean enableDelete) {
+this.rowRowConverter =
RowRowConverter.create(DataTypes.ROW(dataTypes));
+this.rowDataSerializer = RowDataSerializer.builder()
+.setFieldNames(fieldNames)
+.setFieldType(dataTypes)
+.setType(type)
+.setFieldDelimiter(fieldDelimiter)
+.enableDelete(enableDelete)
+.build();
+}
+
+@Override
+public byte[] serialize(Row record) throws IOException{
+RowData rowDataRecord = this.rowRowConverter.toInternal(record);
+return this.rowDataSerializer.serialize(rowDataRecord);
+}
+
+public static Builder builder() {
+return new Builder();
+}
+
+public String parseDeleteSign(RowKind rowKind) {
+if (RowKind.INSERT.equals(rowKind) ||
RowKind.UPDATE_AFTER.equals(rowKind)) {
+return "0";
+} else if (RowKind.DELETE.equals(rowKind) ||
RowKind.UPDATE_BEFORE.equals(rowKind)) {
+return "1";
+} else {
+throw new IllegalArgumentException("Unrecognized row kind:" +
rowKind.toString());
+}
+}
Review Comment:
> This function seems to be useless?
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz opened a new pull request, #13248: [branch-1.1-lts](hot) Fix old mem tracker cancel
xinyiZzz opened a new pull request, #13248: URL: https://github.com/apache/doris/pull/13248 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #13248: [branch-1.1-lts](hot) Fix old mem tracker cancel
dataroaring merged PR #13248: URL: https://github.com/apache/doris/pull/13248 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.1-lts updated: [branch-1.1-lts](hot) Fix old mem tracker cancel (#13248)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-1.1-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.1-lts by this push:
new 579c60949c [branch-1.1-lts](hot) Fix old mem tracker cancel (#13248)
579c60949c is described below
commit 579c60949c275b15f6f9bc8ba5806a71a6ad08db
Author: Xinyi Zou
AuthorDate: Mon Oct 10 16:08:09 2022 +0800
[branch-1.1-lts](hot) Fix old mem tracker cancel (#13248)
---
be/src/runtime/mem_tracker.h | 10 +-
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/be/src/runtime/mem_tracker.h b/be/src/runtime/mem_tracker.h
index a8d7a7f146..e7a14c1ac7 100644
--- a/be/src/runtime/mem_tracker.h
+++ b/be/src/runtime/mem_tracker.h
@@ -186,7 +186,7 @@ public:
for (i = all_trackers_.size() - 1; i >= 0; --i) {
MemTracker* tracker = all_trackers_[i];
const int64_t limit = tracker->GetLimit(mode);
-if (limit < 0 || !config::enable_cancel_query) {
+if (limit < 0) {
tracker->consumption_->add(bytes); // No limit at this tracker.
} else {
// If TryConsume fails, we can try to GC, but we may need to
try several times if
@@ -262,7 +262,7 @@ public:
/// exceeded.
bool AnyLimitExceeded(MemLimit mode) {
for (const auto& tracker : limit_trackers_) {
-if (tracker->LimitExceeded(mode) && config::enable_cancel_query) {
+if (tracker->LimitExceeded(mode)) {
return true;
}
}
@@ -281,7 +281,7 @@ public:
// Return limit exceeded tracker or null
MemTracker* find_limit_exceeded_tracker() {
for (const auto& tracker : limit_trackers_) {
-if (tracker->limit_exceeded() && config::enable_cancel_query) {
+if (tracker->limit_exceeded()) {
return tracker;
}
}
@@ -298,7 +298,7 @@ public:
void RefreshConsumptionFromMetric();
// TODO(yingchun): following functions are old style which have no
MemLimit parameter
-bool limit_exceeded() const { return limit_ >= 0 && limit_ < consumption()
&& config::enable_cancel_query; }
+bool limit_exceeded() const { return limit_ >= 0 && limit_ <
consumption(); }
int64_t limit() const { return limit_; }
bool has_limit() const { return limit_ >= 0; }
@@ -396,7 +396,7 @@ public:
static bool limit_exceeded(const std::vector>&
trackers) {
for (const auto& tracker : trackers) {
-if (tracker->limit_exceeded() && config::enable_cancel_query) {
+if (tracker->limit_exceeded()) {
// TODO: remove logging
LOG(WARNING) << "exceeded limit: limit=" << tracker->limit()
<< " consumption=" << tracker->consumption();
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] lsy3993 commented on a diff in pull request #13132: [function](date function) add new date function 'first_month_day'
lsy3993 commented on code in PR #13132:
URL: https://github.com/apache/doris/pull/13132#discussion_r991017597
##
be/src/vec/functions/function_timestamp.cpp:
##
@@ -478,6 +478,149 @@ class FunctionUnixTimestamp : public IFunction {
}
};
+template
+class FirstMonthDay : public IFunction {
+public:
+static constexpr auto name = "first_month_day";
+static FunctionPtr create() { return
std::make_shared>(); }
+
+String get_name() const override { return name; }
+
+bool use_default_implementation_for_nulls() const override { return true; }
+
+size_t get_number_of_arguments() const override { return 1; }
+
+bool is_variadic() const override { return true; }
+
+DataTypePtr get_return_type_impl(const ColumnsWithTypeAndName& arguments)
const override {
+return Impl::get_return_type_impl(arguments);
+}
+
+DataTypes get_variadic_argument_types_impl() const override {
+return Impl::get_variadic_argument_types();
+}
+
+Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
+size_t result, size_t input_rows_count) override {
+return Impl::execute_impl(context, block, arguments, result,
input_rows_count);
+}
+};
+
+template
+struct FirstMonthDayImpl {
+static DataTypes get_variadic_argument_types() {
+if constexpr (std::is_same_v) {
+return {std::make_shared()};
+} else if constexpr (std::is_same_v) {
+return {std::make_shared()};
+} else {
+return {std::make_shared()};
+}
+}
+
+// input DateTime and Date, return Date
+// input DateTimeV2 and DateV2, return DateV2
+static DataTypePtr get_return_type_impl(const ColumnsWithTypeAndName&
arguments) {
+if constexpr (std::is_same_v) {
Review Comment:
return DataTypeDate when DateType is DataTypeDate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 commented on pull request #13227: [fix](join) should call getOutputTblRefIds to get child's tuple info
starocean999 commented on PR #13227: URL: https://github.com/apache/doris/pull/13227#issuecomment-1272969041 > Does this pr solve the problem that #12261 wants to fix？ yes, originally, the old pr just want to make the computeOutputTuple and computeIntermediateTuple both call getOutputTupleIds to keep consistent, but it's wrong. The two function should call diffrent method to get child's tuple info. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] mrhhsg opened a new issue, #13249: [Bug] coredump caused by like predicate with nullable column
mrhhsg opened a new issue, #13249: URL: https://github.com/apache/doris/issues/13249 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? ``` *** Query id: 0-0 *** *** Aborted at 1665391257 (unix time) try "date -d @1665391257" if you are using GNU date *** *** Current BE git commitID: 935ef5a59 *** *** SIGSEGV address not mapped to object (@0x204) received by PID 124332 (TID 0x7f0f30bdd700) from PID 516; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk/hushenggang/doris/be/src/common/signal_handler.h:420 1# 0x7F0F43650400 in /lib64/libc.so.6 2# doris::VolnitskyBase >::search(unsigned char const*, unsigned long) const at /mnt/disk/hushenggang/doris/be/src/vec/common/volnitsky.h:214 3# doris::vectorized::FunctionLikeBase::constant_substring_fn(doris::vectorized::LikeSearchState*, doris::StringValue const&, doris::StringValue const&, unsigned char*) at /mnt/disk/hushenggang/doris/be/src/vec/functions/like.cpp:94 4# std::_Function_handler::_M_invoke(std::_Any_data const&, doris::vectorized::LikeSearchState*&&, doris::StringValue const&, doris::StringValue const&, unsigned char*&&) at /mnt/disk/hushenggang/.local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 5# doris::LikeColumnPredicate::evaluate(doris::vectorized::IColumn const&, unsigned short*, unsigned short) const at /mnt/disk/hushenggang/doris/be/src/olap/like_column_predicate.cpp:111 6# doris::segment_v2::SegmentIterator::_evaluate_short_circuit_predicate(unsigned short*, unsigned short) at /mnt/disk/hushenggang/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1037 7# doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) in /mnt/disk/hushenggang/doris/be/output/lib/doris_be 8# doris::BetaRowsetReader::next_block(doris::vectorized::Block*) at /mnt/disk/hushenggang/doris/be/src/olap/rowset/beta_rowset_reader.cpp:277 9# doris::vectorized::VCollectIterator::Level0Iterator::next(doris::vectorized::Block*) at /mnt/disk/hushenggang/doris/be/src/vec/olap/vcollect_iterator.cpp:262 10# doris::vectorized::VCollectIterator::Level1Iterator::_normal_next(doris::vectorized::Block*) at /mnt/disk/hushenggang/doris/be/src/vec/olap/vcollect_iterator.cpp:528 11# doris::vectorized::VCollectIterator::Level1Iterator::next(doris::vectorized::Block*) at /mnt/disk/hushenggang/doris/be/src/vec/olap/vcollect_iterator.cpp:355 12# doris::vectorized::VCollectIterator::next(doris::vectorized::Block*) at /mnt/disk/hushenggang/doris/be/src/vec/olap/vcollect_iterator.cpp:185 13# doris::vectorized::BlockReader::_direct_next_block(doris::vectorized::Block*, doris::MemPool*, doris::ObjectPool*, bool*) at /mnt/disk/hushenggang/doris/be/src/vec/olap/block_reader.cpp:175 14# doris::vectorized::BlockReader::next_block_with_aggregation(doris::vectorized::Block*, doris::MemPool*, doris::ObjectPool*, bool*) at /mnt/disk/hushenggang/doris/be/src/vec/olap/block_reader.h:45 15# doris::vectorized::VOlapScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /mnt/disk/hushenggang/doris/be/src/vec/exec/volap_scanner.cpp:344 16# doris::vectorized::VOlapScanNode::scanner_thread(doris::vectorized::VOlapScanner*) at /mnt/disk/hushenggang/doris/be/src/vec/exec/volap_scan_node.cpp:486 17# doris::ThreadPool::dispatch_thread() at /mnt/disk/hushenggang/doris/be/src/util/threadpool.cpp:548 18# doris::Thread::supervise_thread(void*) at /mnt/disk/hushenggang/doris/be/src/util/thread.cpp:426 19# start_thread in /lib64/libpthread.so.0 20# clone in /lib64/libc.so.6 ``` ### What You Expected? query run successfully ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-flink-connector] JNSimba merged pull request #71: [Enhancement] ADD RowSerializer for doris flink connector
JNSimba merged PR #71: URL: https://github.com/apache/doris-flink-connector/pull/71 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-flink-connector] branch master updated: [Enhancement] ADD RowSerializer for doris flink connector (#71)
This is an automated email from the ASF dual-hosted git repository.
diwu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-flink-connector.git
The following commit(s) were added to refs/heads/master by this push:
new 8d7b7b7 [Enhancement] ADD RowSerializer for doris flink connector
(#71)
8d7b7b7 is described below
commit 8d7b7b7db59f142cd19683e91c686cc2ed96008d
Author: DingGeGe <109070189+dinggege1...@users.noreply.github.com>
AuthorDate: Mon Oct 10 16:44:19 2022 +0800
[Enhancement] ADD RowSerializer for doris flink connector (#71)
* [Enhancement] ADD RowSerializer for doris flink connector
---
.../doris/flink/sink/writer/RowSerializer.java | 107 +
.../doris/flink/sink/writer/TestRowSerializer.java | 97 +++
2 files changed, 204 insertions(+)
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/RowSerializer.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/RowSerializer.java
new file mode 100644
index 000..3a07951
--- /dev/null
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/RowSerializer.java
@@ -0,0 +1,107 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.flink.sink.writer;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.conversion.RowRowConverter;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.types.Row;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Preconditions;
+import java.io.IOException;
+import static org.apache.doris.flink.sink.writer.LoadConstants.CSV;
+import static org.apache.doris.flink.sink.writer.LoadConstants.JSON;
+
+/**
+ * Serializer for {@link Row}.
+ * Quick way to support RowSerializer on existing code
+ * TODO: support original Doris to Row serializer
+ */
+public class RowSerializer implements DorisRecordSerializer {
+/**
+ * converter {@link Row} to {@link RowData}
+ */
+private final RowRowConverter rowRowConverter;
+private final RowDataSerializer rowDataSerializer;
+
+private RowSerializer(String[] fieldNames, DataType[] dataTypes, String
type, String fieldDelimiter,
+ boolean enableDelete) {
+this.rowRowConverter =
RowRowConverter.create(DataTypes.ROW(dataTypes));
+this.rowDataSerializer = RowDataSerializer.builder()
+.setFieldNames(fieldNames)
+.setFieldType(dataTypes)
+.setType(type)
+.setFieldDelimiter(fieldDelimiter)
+.enableDelete(enableDelete)
+.build();
+}
+
+@Override
+public byte[] serialize(Row record) throws IOException{
+RowData rowDataRecord = this.rowRowConverter.toInternal(record);
+return this.rowDataSerializer.serialize(rowDataRecord);
+}
+
+public static Builder builder() {
+return new Builder();
+}
+
+/**
+ * Builder for RowSerializer.
+ */
+public static class Builder {
+private String[] fieldNames;
+private DataType[] dataTypes;
+private String type;
+private String fieldDelimiter;
+private boolean deletable;
+
+public Builder setFieldNames(String[] fieldNames) {
+this.fieldNames = fieldNames;
+return this;
+}
+
+public Builder setFieldType(DataType[] dataTypes) {
+this.dataTypes = dataTypes;
+return this;
+}
+
+public Builder setType(String type) {
+this.type = type;
+return this;
+}
+
+public Builder setFieldDelimiter(String fieldDelimiter) {
+this.fieldDelimiter = fieldDelimiter;
+return this;
+}
+
+public Builder enableDelete(boolean deletable) {
+this.deletable = deletable;
+return this;
+}
+
+public RowSerializer build() {
+Preconditions.checkState(CSV.equals(type) && fieldDelimiter !=
null || JSON.equals(type));
+Preconditions.checkNotNull(dataType
[GitHub] [doris] mrhhsg closed issue #13249: [Bug] coredump caused by like predicate with nullable column
mrhhsg closed issue #13249: [Bug] coredump caused by like predicate with nullable column URL: https://github.com/apache/doris/issues/13249 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hf200012 merged pull request #13229: [improvement](mysql-to-doris)Fully resolved Mysql external table issues
hf200012 merged PR #13229: URL: https://github.com/apache/doris/pull/13229 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improvement](mysql-to-doris)Fully resolved Mysql external table issues (#13229)
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 54e6f12110 [improvement](mysql-to-doris)Fully resolved Mysql external
table issues (#13229)
54e6f12110 is described below
commit 54e6f12110e0aae1caf45b777f05d4912962b764
Author: toms <94617906+toms1...@users.noreply.github.com>
AuthorDate: Mon Oct 10 16:48:52 2022 +0800
[improvement](mysql-to-doris)Fully resolved Mysql external table issues
(#13229)
Fully resolved Mysql external table issues
---
extension/mysql_to_doris/README.md | 12 +-
extension/mysql_to_doris/all_tables.sh | 154 -
extension/mysql_to_doris/bin/e_auto.sh | 52 +++
extension/mysql_to_doris/bin/e_mysql_to_doris.sh | 97 +
.../mysql_to_doris/conf/{tables => doris_tables} | 28 +---
.../mysql_to_doris/conf/{tables => mysql_tables} | 25 +---
extension/mysql_to_doris/lib/mysql_to_doris.sh | 61
extension/mysql_to_doris/user_define_tables.sh | 154 -
8 files changed, 225 insertions(+), 358 deletions(-)
diff --git a/extension/mysql_to_doris/README.md
b/extension/mysql_to_doris/README.md
index 049dafcdf3..351f2b439f 100644
--- a/extension/mysql_to_doris/README.md
+++ b/extension/mysql_to_doris/README.md
@@ -22,11 +22,13 @@ How to do?
1、To configure mysql.conf and doris.conf in the conf directory,the conf
including
host、port and password
-2、sh all_tables.sh and give it two args,it is mysql database and doris database
+2、To configure mysql_tables and doris_tables in the conf directory,the conf is
user need to synchronization tables and want to get table name
-In addition
+3、To execute e_mysql_to_doris.sh by sh e_mysql_to_doris.sh
-user can add mysql tables to the conf directory tables by sh
user_define_tables.sh
-can also be export mysql tables to doris
+4、To execute e_auto.sh by nohup sh e_auto.sh &
+
+What do you get?
+
+A simple configuration synchronizes all configured tables and Monitor Mysql
metadata changes in real time
-just do it
diff --git a/extension/mysql_to_doris/all_tables.sh
b/extension/mysql_to_doris/all_tables.sh
deleted file mode 100644
index 634ece0f71..00
--- a/extension/mysql_to_doris/all_tables.sh
+++ /dev/null
@@ -1,154 +0,0 @@
-#!/bin/bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-
-
-# This script is used to will mysql databases import doris by external
-
-
-#reference configuration file
-source ./conf/mysql.conf
-source ./conf/doris.conf
-
-#define mysql database and doris database
-d_mysql=$1
-d_doris=$2
-
-#check args
-if [ ! -n "$1" ];then
-echo "please check source database"
-exit
-fi
-if [ ! -n "$2" ];then
-echo "please check sink database"
-exit
-fi
-
-#mkdir files to store tables and tables.sql
-mkdir -p files
-rm -rf ./files/tables
-rm -rf ./files/tables.sql
-#get tables from mysql databases
-echo "use $d_mysql; show tables;" |mysql -h$mysql_host -uroot
-p$mysql_password 2>/dev/null >> ./files/tables
-
-#delete tables first line
-sed -i '1d' ./files/tables
-
-#reference tables to create tables.sql
-for table in $(awk -F '\n' '{print $1}' ./files/tables)
-do
-sed -i "/${table}view/d" ./files/tables
-echo "use $d_mysql; show create table ${table};" |mysql -h$mysql_host
-uroot -p$mysql_password 2>/dev/null >> ./files/tables.sql
-echo "print ${table} sql to tables.sql in the file dir"
-
-done
-
-echo '==start to transform mysql table for doris
extral table=='
-#adjust sql
-awk -F '\t' '{print $2}' ./files/tables.sql |awk '!(NR%2)' |awk '{print $0
";"}' > ./files/tables1.sql
-sed -i 's/\\n/\n/g' ./files/tables1.sql
-sed -n '/CREATE TABLE/,/ENGINE\=/p' ./files/tables1.sql > ./files/tables2.sql
-#delete tables special struct
-sed -i '/^ CON/d' ./files/tables2.sql
-sed -i '/^ KEY/d' ./files/tables2.sql
-rm -
[GitHub] [doris] 345362231 closed issue #6574: [Bug] doris 0.14 backup/restore问题
345362231 closed issue #6574: [Bug] doris 0.14 backup/restore问题 URL: https://github.com/apache/doris/issues/6574 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] mrhhsg opened a new pull request, #13250: [fix](olap) fix core dump caused by LikeColumnPredicate with nullable…
mrhhsg opened a new pull request, #13250: URL: https://github.com/apache/doris/pull/13250 … column # Proposed changes Issue Number: close #13249 ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13226: [refactor](datax)Refactoring doris writer code
github-actions[bot] commented on PR #13226: URL: https://github.com/apache/doris/pull/13226#issuecomment-1272990668 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13226: [refactor](datax)Refactoring doris writer code
github-actions[bot] commented on PR #13226: URL: https://github.com/apache/doris/pull/13226#issuecomment-1272990738 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 345362231 opened a new issue, #13251: [Bug] BE日志告警UpdateDerivedVars is too busy!服务器卡住
345362231 opened a new issue, #13251: URL: https://github.com/apache/doris/issues/13251 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version trunk-f286bde ### What's Wrong?  日志报错关键字： W1010 14:54:32.531973 14786 server.cpp:336] UpdateDerivedVars is too busy! W1010 14:54:53.548692 14276 sampler.cpp:189] bvar is busy at sampling for 2 seconds! 只要一有出现，和其他服务器通信出现超时，监控也无法获取参数。服务器上操作有延迟。 在集群中就这台BE表现异常。 ### What You Expected? 如何解决该繁忙情况？不在日志中报错busy。 ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] levy5307 commented on pull request #12919: [fix](memory): avoid coredump when list pointer is null
levy5307 commented on PR #12919: URL: https://github.com/apache/doris/pull/12919#issuecomment-1273014442 > Could you add a regression test? It's hard to add a regression test because this bug is not a problem bound to arise. So I add a unit test for this function instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] carlvinhust2012 commented on pull request #13225: [fix](array-type) fix get_data_at for zero element array
carlvinhust2012 commented on PR #13225: URL: https://github.com/apache/doris/pull/13225#issuecomment-1273020872 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #13218: [fix](statstics) Incorrectly using the number of buckets to determine whether the table is partitioned
morrySnow merged PR #13218: URL: https://github.com/apache/doris/pull/13218 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (54e6f12110 -> f007e0aed0)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 54e6f12110 [improvement](mysql-to-doris)Fully resolved Mysql external table issues (#13229) add f007e0aed0 [fix](statstics) Incorrectly using the number of buckets to determine whether the table is partitioned (#13218) No new revisions were added by this update. Summary of changes: .../main/java/org/apache/doris/analysis/AlterColumnStatsStmt.java| 5 +++-- .../src/main/java/org/apache/doris/statistics/StatisticsManager.java | 4 +++- 2 files changed, 6 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #13243: [Enhancement](nereids) Remove redundant log when fall back to stale parser
morrySnow commented on code in PR #13243:
URL: https://github.com/apache/doris/pull/13243#discussion_r991085964
##
fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java:
##
@@ -266,8 +266,8 @@ private void handleQuery() {
} catch (Exception e) {
nereidsParseException = e;
// TODO: We should catch all exception here until we
support all query syntax.
-LOG.warn(" Fallback to stale planner."
-+ " Nereids cannot process this statement:
\"{}\".", originStmt, e);
+LOG.info(" Fallback to stale planner."
++ " Nereids cannot process this statement:
\"{}\".", originStmt.toString());
Review Comment:
```suggestion
+ " Nereids cannot process this statement:
\"{}\".", originStmt);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 opened a new pull request, #13252: [DMG](planner) Log error when choosing best plan
Kikyou1997 opened a new pull request, #13252: URL: https://github.com/apache/doris/pull/13252 # Proposed changes Issue Number: noissue ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] siriume opened a new issue, #13253: [Feature] to_bitmap function support date/datev2.
siriume opened a new issue, #13253: URL: https://github.com/apache/doris/issues/13253 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description to_bitmap function on only support int type, Please support date/datev2 type. ### Use case case 1: ```sql create table id_date_bitmap ( `id` int(11) NOT NULL COMMENT 'id', `date` bitmap BITMAP_UNION COMMENT 'date' )ENGINE=OLAP AGGREGATE KEY(`id`) COMMENT "bitmap test" DISTRIBUTED BY HASH(`id`) BUCKETS 4; insert into id_date_bitmap select id,account_id,to_bitmap(`date`) `date` from where `cost` > 0; ``` case 2: ```sql create table tb_account ( `id` int(11) NOT NULL COMMENT 'id', `account_id` varchar(90) NOT NULL COMMENT 'account_id', `date` date COMMENT 'date' )ENGINE=OLAP AGGREGATE KEY(`id`, `account_id`) COMMENT "bitmap test" DISTRIBUTED BY HASH(`id`) BUCKETS 4; create materialized view tb_account_id as select id, bitmap_union(to_bitmap(`date`)) from id_date_bitmap group by id; create materialized view tb_account_account_id as select account_id, bitmap_union(to_bitmap(`date`)) from id_date_bitmap group by account_id; ``` ### Related issues _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13250: [fix](olap) fix core dump caused by LikeColumnPredicate with nullable…
github-actions[bot] commented on PR #13250: URL: https://github.com/apache/doris/pull/13250#issuecomment-127398 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13250: [fix](olap) fix core dump caused by LikeColumnPredicate with nullable…
github-actions[bot] commented on PR #13250: URL: https://github.com/apache/doris/pull/13250#issuecomment-1273111252 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #13163: [regression](load)Open broker load regression test
dataroaring merged PR #13163: URL: https://github.com/apache/doris/pull/13163 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [regression](load)Open broker load regression test (#13163)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 7c0695c793 [regression](load)Open broker load regression test (#13163)
7c0695c793 is described below
commit 7c0695c7934ec60a8906b36bae3fd9a95e7ecfc2
Author: Jibing-Li <64681310+jibing...@users.noreply.github.com>
AuthorDate: Mon Oct 10 18:49:44 2022 +0800
[regression](load)Open broker load regression test (#13163)
---
regression-test/conf/regression-conf.groovy| 2 +-
.../load_p0/broker_load/test_broker_load.groovy| 54 +++---
2 files changed, 29 insertions(+), 27 deletions(-)
diff --git a/regression-test/conf/regression-conf.groovy
b/regression-test/conf/regression-conf.groovy
index 506cf89bf7..4577d0fe08 100644
--- a/regression-test/conf/regression-conf.groovy
+++ b/regression-test/conf/regression-conf.groovy
@@ -62,6 +62,6 @@ hdfsPasswd = ""
brokerName = "broker_name"
// broker load test config
-enableBrokerLoad=false
+enableBrokerLoad=true
ak=""
sk=""
diff --git a/regression-test/suites/load_p0/broker_load/test_broker_load.groovy
b/regression-test/suites/load_p0/broker_load/test_broker_load.groovy
index 681502ade6..fa4d2e0c07 100644
--- a/regression-test/suites/load_p0/broker_load/test_broker_load.groovy
+++ b/regression-test/suites/load_p0/broker_load/test_broker_load.groovy
@@ -39,27 +39,27 @@ suite("test_broker_load", "p0") {
"parquet_s3_case7", // col5 will be ignored, load normally
"parquet_s3_case8" // first column in table is not
specified, will load default value for it.
]
-def paths = ["s3://doris-community-test-1308700295/load/data/part*",
- "s3://doris-community-test-1308700295/load/data/part*",
- "s3://doris-community-test-1308700295/load/data/part*",
- "s3://doris-community-test-1308700295/load/data/part*",
- "s3://doris-community-test-1308700295/load/data/part*",
- "s3://doris-community-test-1308700295/load/data/part*",
- "s3://doris-community-test-1308700295/load/data/part*",
- "s3://doris-community-test-1308700295/load/data/part*",
- "s3://doris-community-test-1308700295/load/data/part*",
- "s3://doris-community-test-1308700295/load/data/part*",
- "s3://doris-community-test-1308700295/load/data/part*",
- "s3://doris-community-test-1308700295/load/data/part*",
- "s3://doris-community-test-1308700295/load/data/path/*/part*",
- "s3://doris-community-test-1308700295/load/data/part*",
- "s3://doris-community-test-1308700295/load/data/part*",
- "s3://doris-community-test-1308700295/load/data/part*",
- "s3://doris-community-test-1308700295/load/data/part*",
- "s3://doris-community-test-1308700295/load/data/part*",
- "s3://doris-community-test-1308700295/load/data/part*",
- "s3://doris-community-test-1308700295/load/data/part*",
- "s3://doris-community-test-1308700295/load/data/part*",
+def paths = ["s3://doris-build-hk-1308700295/regression/load/data/part*",
+ "s3://doris-build-hk-1308700295/regression/load/data/part*",
+ "s3://doris-build-hk-1308700295/regression/load/data/part*",
+ "s3://doris-build-hk-1308700295/regression/load/data/part*",
+ "s3://doris-build-hk-1308700295/regression/load/data/part*",
+ "s3://doris-build-hk-1308700295/regression/load/data/part*",
+ "s3://doris-build-hk-1308700295/regression/load/data/part*",
+ "s3://doris-build-hk-1308700295/regression/load/data/part*",
+ "s3://doris-build-hk-1308700295/regression/load/data/part*",
+ "s3://doris-build-hk-1308700295/regression/load/data/part*",
+ "s3://doris-build-hk-1308700295/regression/load/data/part*",
+ "s3://doris-build-hk-1308700295/regression/load/data/part*",
+
"s3://doris-build-hk-1308700295/regression/load/data/path/*/part*",
+ "s3://doris-build-hk-1308700295/regression/load/data/part*",
+ "s3://doris-build-hk-1308700295/regression/load/data/part*",
+ "s3://doris-build-hk-1308700295/regression/load/data/part*",
+ "s3://doris-build-hk-1308700295/regression/load/data/part*",
+ "s3://doris-build-hk-1308700295/regression/load/data/part*",
+ "s3://doris-build-hk-1308700295/regression/load/data/part*",
+ "s3://doris-build-hk-1308700295/regression/load/data/part*",
+ "s3://doris-build-hk-1308700295/regression/load/data/part*",
[GitHub] [doris] pengxiangyu opened a new issue, #13254: [Feature] Add hide config to hide config in webserver for safety.
pengxiangyu opened a new issue, #13254: URL: https://github.com/apache/doris/issues/13254 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description Webserver on BE has no authorization verification. Anyone can get the config for be. It is not safe for online system. So I need to add a parameter to hide the config. ### Use case call webserver on BE, config will be returned. ### Related issues 1.12 ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] pengxiangyu opened a new pull request, #13255: [feature](config)Add hide config to hide config in webserver for safety.
pengxiangyu opened a new pull request, #13255: URL: https://github.com/apache/doris/pull/13255 # Proposed changes Issue Number: close #13254 ## Problem summary Webserver on BE has no authorization verification. Anyone can get the config for be. It is not safe for online system. So I need to add a parameter to hide the config. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt opened a new pull request, #13256: [Enhancement](be-exec-version) add process when pblock.has_be_exec_version()=false
BiteThet opened a new pull request, #13256: URL: https://github.com/apache/doris/pull/13256 # Proposed changes add process when pblock.has_be_exec_version()=false ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] AshinGau opened a new pull request, #13257: [feature-wip](multi-catalog) optimize parquet profile & add null map timer
AshinGau opened a new pull request, #13257: URL: https://github.com/apache/doris/pull/13257 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [x] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on a diff in pull request #10170: [Enhancement](optimizer) Support select table sample
xinyiZzz commented on code in PR #10170:
URL: https://github.com/apache/doris/pull/10170#discussion_r991185692
##
fe/fe-core/src/main/java/org/apache/doris/analysis/TupleDescriptor.java:
##
@@ -159,6 +164,80 @@ public void setTable(TableIf tbl) {
table = tbl;
}
+public Set getSampleTabletIds() {
+return sampleTabletIds;
+}
+
+/**
+ * First, determine how many rows to sample from each partition according
to the number of partitions.
+ * Then determine the number of Tablets to be selected for each partition
according to the average number
+ * of rows of Tablet,
+ * If seek is not specified, the specified number of Tablets are
pseudo-randomly selected from each partition.
+ * If seek is specified, it will be selected sequentially from the seek
tablet of the partition.
+ * And add the manually specified Tablet id to the selected Tablet.
+ * simpleTabletNums = simpleRows / partitionNums / (partitionRows /
partitionTabletNums)
+ */
+public void computeSampleTabletIds(List tabletIds, TableSample
tableSample) {
+if (table.getType() != TableType.OLAP) {
+return;
+}
+sampleTabletIds.addAll(tabletIds);
+if (tableSample == null) {
+return;
+}
+OlapTable olapTable = (OlapTable) table;
+long sampleRows; // The total number of sample rows
+long hitRows = 1; // The total number of rows hit by the tablet
+long totalRows = 0; // The total number of partition rows hit
+long totalTablet = 0; // The total number of tablets in the hit
partition
+if (tableSample.isPercent()) {
+sampleRows = (long) Math.max(olapTable.getRowCount() *
(tableSample.getSampleValues() / 100.0), 1);
+} else {
+sampleRows = Math.max(tableSample.getSampleValues(), 1);
+}
+
+// calculate the number of tablets by each partition
+long avgRowsPerPartition = sampleRows /
Math.max(olapTable.getPartitions().size(), 1);
+
+for (Partition p : olapTable.getPartitions()) {
+List ids = p.getBaseIndex().getTabletIdsInOrder();
+
+if (ids.isEmpty()) {
+continue;
+}
+
+if (p.getBaseIndex().getRowCount() < (avgRowsPerPartition / 2)) {
Review Comment:
sorry, totalRows will not be less than sampleRows
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on a diff in pull request #10170: [Enhancement](optimizer) Support select table sample
xinyiZzz commented on code in PR #10170:
URL: https://github.com/apache/doris/pull/10170#discussion_r991197443
##
fe/fe-core/src/main/java/org/apache/doris/analysis/TableSample.java:
##
@@ -0,0 +1,101 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.common.AnalysisException;
+
+/*
+ * To represent following stmt:
+ * TABLESAMPLE (10 PERCENT)
+ * TABLESAMPLE (100 ROWS)
+ * TABLESAMPLE (10 PERCENT) REPEATABLE (123)
+ * TABLESAMPLE (100 ROWS) REPEATABLE (123)R
+ *
+ * references:
+ *
https://simplebiinsights.com/sql-server-tablesample-retrieving-random-data-from-sql-server/
+ * https://sqlrambling.net/2018/01/24/tablesample-basic-examples/
+ */
+public class TableSample implements ParseNode {
+
+private final Long sampleValue;
+private final boolean isPercent;
+private final Long seek;
+
+public TableSample(boolean isPercent, Long sampleValue) {
+this.sampleValue = sampleValue;
+this.isPercent = isPercent;
+this.seek = -1L;
Review Comment:
It is also set in another init func~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 opened a new pull request, #13258: [config](datev2) set `enable_date_conversion = true` by default
Gabriel39 opened a new pull request, #13258: URL: https://github.com/apache/doris/pull/13258 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on a diff in pull request #10170: [Enhancement](optimizer) Support select table sample
xinyiZzz commented on code in PR #10170:
URL: https://github.com/apache/doris/pull/10170#discussion_r990934807
##
fe/fe-core/src/main/java/org/apache/doris/analysis/TableSample.java:
##
@@ -0,0 +1,101 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.common.AnalysisException;
+
+/*
+ * To represent following stmt:
+ * TABLESAMPLE (10 PERCENT)
+ * TABLESAMPLE (100 ROWS)
+ * TABLESAMPLE (10 PERCENT) REPEATABLE (123)
+ * TABLESAMPLE (100 ROWS) REPEATABLE (123)R
+ *
+ * references:
+ *
https://simplebiinsights.com/sql-server-tablesample-retrieving-random-data-from-sql-server/
+ * https://sqlrambling.net/2018/01/24/tablesample-basic-examples/
+ */
+public class TableSample implements ParseNode {
+
+private final Long sampleValue;
+private final boolean isPercent;
+private final Long seek;
+
+public TableSample(boolean isPercent, Long sampleValue) {
+this.sampleValue = sampleValue;
+this.isPercent = isPercent;
+this.seek = -1L;
+}
+
+public TableSample(boolean isPercent, Long sampleValue, Long seek) {
+this.sampleValue = sampleValue;
+this.isPercent = isPercent;
+this.seek = seek;
+}
+
+public TableSample(TableSample other) {
+this.sampleValue = other.sampleValue;
+this.isPercent = other.isPercent;
+this.seek = other.seek;
+}
+
+public Long getSampleValue() {
+return sampleValue;
+}
+
+public boolean isPercent() {
+return isPercent;
+}
+
+public Long getSeek() {
+return seek;
+}
+
+@Override
+public void analyze(Analyzer analyzer) throws AnalysisException {
+if (sampleValue <= 0 || (isPercent && sampleValue > 100)) {
+throw new AnalysisException("table sample value must be greater
than 0, percent need less than 100.");
+}
+}
+
+@Override
+public String toSql() {
+if (sampleValue == null) {
+return "";
+}
+StringBuilder sb = new StringBuilder();
+sb.append("TABLESAMPLE ( ");
+sb.append(sampleValue);
+if (isPercent) {
+sb.append(" PERCENT ");
+} else {
+sb.append(" ROWS ");
+}
+sb.append(")");
+if (seek != 0) {
+sb.append(" REPEATABLE ");
+sb.append(seek);
Review Comment:
no parentheses, wrong description in pr comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on a diff in pull request #10170: [Enhancement](optimizer) Support select table sample
xinyiZzz commented on code in PR #10170:
URL: https://github.com/apache/doris/pull/10170#discussion_r991268951
##
fe/fe-core/src/main/java/org/apache/doris/analysis/TupleDescriptor.java:
##
@@ -159,6 +164,84 @@ public void setTable(TableIf tbl) {
table = tbl;
}
+public Set getSampleTabletIds() {
+return sampleTabletIds;
+}
+
+/**
+ * First, determine how many rows to sample from each partition according
to the number of partitions.
+ * Then determine the number of Tablets to be selected for each partition
according to the average number
+ * of rows of Tablet,
+ * If seek is not specified, the specified number of Tablets are
pseudo-randomly selected from each partition.
+ * If seek is specified, it will be selected sequentially from the seek
tablet of the partition.
+ * And add the manually specified Tablet id to the selected Tablet.
+ * simpleTabletNums = simpleRows / partitionNums / (partitionRows /
partitionTabletNums)
+ */
+public void computeSampleTabletIds(List tabletIds, TableSample
tableSample) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on a diff in pull request #10170: [Enhancement](optimizer) Support select table sample
xinyiZzz commented on code in PR #10170:
URL: https://github.com/apache/doris/pull/10170#discussion_r991269203
##
fe/fe-core/src/main/java/org/apache/doris/analysis/TupleDescriptor.java:
##
@@ -159,6 +164,80 @@ public void setTable(TableIf tbl) {
table = tbl;
}
+public Set getSampleTabletIds() {
+return sampleTabletIds;
+}
+
+/**
+ * First, determine how many rows to sample from each partition according
to the number of partitions.
+ * Then determine the number of Tablets to be selected for each partition
according to the average number
+ * of rows of Tablet,
+ * If seek is not specified, the specified number of Tablets are
pseudo-randomly selected from each partition.
+ * If seek is specified, it will be selected sequentially from the seek
tablet of the partition.
+ * And add the manually specified Tablet id to the selected Tablet.
+ * simpleTabletNums = simpleRows / partitionNums / (partitionRows /
partitionTabletNums)
+ */
+public void computeSampleTabletIds(List tabletIds, TableSample
tableSample) {
+if (table.getType() != TableType.OLAP) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on pull request #10170: [Enhancement](optimizer) Support select table sample
xinyiZzz commented on PR #10170: URL: https://github.com/apache/doris/pull/10170#issuecomment-1273290477 @morrySnow @weizhengte PTAL~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen opened a new pull request, #13259: test benchmark pipeline
hello-stephen opened a new pull request, #13259: URL: https://github.com/apache/doris/pull/13259 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13259: test benchmark pipeline
github-actions[bot] commented on PR #13259:
URL: https://github.com/apache/doris/pull/13259#issuecomment-1273330241
`sh-checker report`
To get the full details, please check in the
[job]("https://github.com/apache/doris/actions/runs/3219583062";) output.
shellcheck errors
```
'shellcheck ' returned error 1 finding the following syntactical issues:
--
In benchmark/clickbench/prepare.sh line 9:
checkout_home=$(pwd)
^---^ SC2034 (warning): checkout_home appears unused. Verify use (or
export if used externally).
In benchmark/clickbench/prepare.sh line 14:
cd "$prepare_home"
^---^ SC2250 (style): Prefer putting braces around variable
references even when not strictly required.
Did you mean:
cd "${prepare_home}"
In benchmark/clickbench/prepare.sh line 15:
if [[ ! -d "$prepare_home/selectdb-qa" ]]; then
^---^ SC2250 (style): Prefer putting braces around
variable references even when not strictly required.
Did you mean:
if [[ ! -d "${prepare_home}/selectdb-qa" ]]; then
In benchmark/clickbench/prepare.sh line 19:
qa_home="$prepare_home/selectdb-qa"
^---^ SC2250 (style): Prefer putting braces around variable
references even when not strictly required.
Did you mean:
qa_home="${prepare_home}/selectdb-qa"
In benchmark/clickbench/prepare.sh line 20:
cd "$qa_home" && git stash && git checkout main && git pull && cd -
^--^ SC2250 (style): Prefer putting braces around variable
references even when not strictly required.
Did you mean:
cd "${qa_home}" && git stash && git checkout main && git pull && cd -
For more information:
https://www.shellcheck.net/wiki/SC2034 -- checkout_home appears unused.
Ver...
https://www.shellcheck.net/wiki/SC2250 -- Prefer putting braces around
vari...
--
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=
above the line that contains the issue, where is the error code;
3. Add '-e ' to the SHELLCHECK_OPTS setting in your .yml action file.
```
shfmt errors
```
'shfmt ' returned error 1 finding the following formatting issues:
--
--- benchmark/clickbench/prepare.sh.orig
+++ benchmark/clickbench/prepare.sh
@@ -2,14 +2,11 @@
echo 'check if skip'
-
set -ex
prepare_home=/home/ec2-user/
checkout_home=$(pwd)
-
-
echo 'prepare'
cd "$prepare_home"
if [[ ! -d "$prepare_home/selectdb-qa" ]]; then
@@ -18,5 +15,3 @@
echo 'update scripts from g...@github.com:selectdb/selectdb-qa.git'
qa_home="$prepare_home/selectdb-qa"
cd "$qa_home" && git stash && git checkout main && git pull && cd -
-
-
--
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12781: [Enhancement] Doris support obs broker load
github-actions[bot] commented on PR #12781: URL: https://github.com/apache/doris/pull/12781#issuecomment-1273345173 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13225: [fix](array-type) fix get_data_at for zero element array
github-actions[bot] commented on PR #13225: URL: https://github.com/apache/doris/pull/13225#issuecomment-1273348777 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13225: [fix](array-type) fix get_data_at for zero element array
github-actions[bot] commented on PR #13225: URL: https://github.com/apache/doris/pull/13225#issuecomment-1273348827 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Jibing-Li opened a new pull request, #13260: [Enhancement](broker)Support obs broker load
Jibing-Li opened a new pull request, #13260: URL: https://github.com/apache/doris/pull/13260 # Proposed changes Issue Number: close #xxx ## Problem summary 1. Upgrade fs_broker module hadoop2.7.3->hadoop2.8.3 2. Support obs broker load This pr is based on https://github.com/apache/doris/pull/12781, added the maven dependencies to pom, avoid manually copy the jar file to broker lib. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13247: [fix](agg)count function should return 0 for null value
github-actions[bot] commented on PR #13247: URL: https://github.com/apache/doris/pull/13247#issuecomment-1273517959 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13247: [fix](agg)count function should return 0 for null value
github-actions[bot] commented on PR #13247: URL: https://github.com/apache/doris/pull/13247#issuecomment-1273518007 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xy720 opened a new pull request, #13261: [Fix] Disable schema change between array type columns
xy720 opened a new pull request, #13261: URL: https://github.com/apache/doris/pull/13261 # Proposed changes Currently, we do not support schema change between array type columns. We should forbid users from doing this operation. ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz opened a new pull request, #13262: [enhancement](memtracker) Add independent and unique scanner mem tracker for each query
xinyiZzz opened a new pull request, #13262: URL: https://github.com/apache/doris/pull/13262 # Proposed changes Issue Number: close #xxx ## Problem summary Count the memory consumption of Scanner, independent and unique for each query, this means that scnner memory does not count into query mem tracker. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on pull request #13262: [enhancement](memtracker) Add independent and unique scanner mem tracker for each query
xinyiZzz commented on PR #13262: URL: https://github.com/apache/doris/pull/13262#issuecomment-1273591675 @morningman CC -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #13263: [fix](new-scann) scanner may be marked close twice
morningman opened a new pull request, #13263: URL: https://github.com/apache/doris/pull/13263 # Proposed changes Issue Number: close #xxx ## Problem summary See comment in code for details. This bug will cause some scanners to exit early, resulting in missing query results. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hf200012 merged pull request #13226: [refactor](datax)Refactoring doris writer code
hf200012 merged PR #13226: URL: https://github.com/apache/doris/pull/13226 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (7c0695c793 -> 6ee150755a)
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 7c0695c793 [regression](load)Open broker load regression test (#13163)
add 6ee150755a [refactor](datax)Refactoring doris writer code (#13226)
No new revisions were added by this update.
Summary of changes:
docs/en/docs/ecosystem/datax.md| 63 +++--
docs/zh-CN/docs/ecosystem/datax.md | 64 +++--
extension/DataX/doriswriter/doc/doriswriter.md | 171 ++--
extension/DataX/doriswriter/doc/mysql2doris.json | 19 +-
extension/DataX/doriswriter/pom.xml| 1 -
.../plugin/writer/doriswriter/DelimiterParser.java | 71 +
.../{EscapeHandler.java => DorisBaseCodec.java}| 34 ++-
.../plugin/writer/doriswriter/DorisCodec.java | 55 +---
.../{EscapeHandler.java => DorisCodecFactory.java} | 30 +--
.../plugin/writer/doriswriter/DorisCsvCodec.java | 31 +--
.../plugin/writer/doriswriter/DorisFlushBatch.java | 66 -
.../plugin/writer/doriswriter/DorisJsonCodec.java | 29 +-
.../doriswriter/DorisStreamLoadObserver.java | 250 ++
.../datax/plugin/writer/doriswriter/DorisUtil.java | 122 +
.../plugin/writer/doriswriter/DorisWriter.java | 251 ++
.../writer/doriswriter/DorisWriterEmitter.java | 292 -
.../writer/doriswriter/DorisWriterExcetion.java| 36 ++-
.../writer/doriswriter/DorisWriterManager.java | 209 +++
.../datax/plugin/writer/doriswriter/Key.java | 191 --
.../datax/plugin/writer/doriswriter/Keys.java | 194 ++
.../plugin/writer/doriswriter/WriterTuple.java | 26 +-
.../doriswriter/src/main/resources/plugin.json | 4 +-
.../src/main/resources/plugin_job_template.json| 14 +-
.../writer/doriswriter/TestDorisWriterLoad.java| 88 +++
24 files changed, 1281 insertions(+), 1030 deletions(-)
create mode 100644
extension/DataX/doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/DelimiterParser.java
copy
extension/DataX/doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/{EscapeHandler.java
=> DorisBaseCodec.java} (54%)
rename
extension/DataX/doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/{EscapeHandler.java
=> DorisCodecFactory.java} (53%)
delete mode 100644
extension/DataX/doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/DorisFlushBatch.java
create mode 100644
extension/DataX/doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/DorisStreamLoadObserver.java
create mode 100644
extension/DataX/doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/DorisUtil.java
delete mode 100644
extension/DataX/doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/DorisWriterEmitter.java
copy fe/fe-core/src/main/java/org/apache/doris/journal/bdbje/Timestamp.java =>
extension/DataX/doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/DorisWriterExcetion.java
(54%)
create mode 100644
extension/DataX/doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/DorisWriterManager.java
delete mode 100644
extension/DataX/doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/Key.java
create mode 100644
extension/DataX/doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/Keys.java
copy fe/fe-core/src/main/java/org/apache/doris/analysis/ImportColumnsStmt.java
=>
extension/DataX/doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/WriterTuple.java
(61%)
create mode 100644
extension/DataX/doriswriter/src/test/java/com/alibaba/datax/plugin/writer/doriswriter/TestDorisWriterLoad.java
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] liaoxin01 commented on a diff in pull request #13224: [fix](rowset) fix that rowset writer doesn't process the return value which may result in data loss
liaoxin01 commented on code in PR #13224:
URL: https://github.com/apache/doris/pull/13224#discussion_r991747785
##
be/src/olap/rowset/beta_rowset_writer.cpp:
##
@@ -173,7 +173,11 @@ OLAPStatus
BetaRowsetWriter::flush_single_memtable(MemTable* memtable, int64_t*
RowsetSharedPtr BetaRowsetWriter::build() {
// TODO(lingbin): move to more better place, or in a CreateBlockBatch?
for (auto& wblock : _wblocks) {
Review Comment:
Now the caller handle the error according to whether the pointer is nullptr.
So it‘s ok to return nullptr, although it is not a complete way. There will be
a lot of changes if return Status.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] lsy3993 closed pull request #13132: [function](date function) add new date function 'first_month_day'
lsy3993 closed pull request #13132: [function](date function) add new date function 'first_month_day' URL: https://github.com/apache/doris/pull/13132 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 opened a new pull request, #13266: [Regression](javaudf) add regression test for javaudf
Gabriel39 opened a new pull request, #13266: URL: https://github.com/apache/doris/pull/13266 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13266: [Regression](javaudf) add regression test for javaudf
github-actions[bot] commented on PR #13266:
URL: https://github.com/apache/doris/pull/13266#issuecomment-1273990059
`sh-checker report`
To get the full details, please check in the
[job]("https://github.com/apache/doris/actions/runs/3223608884";) output.
shellcheck errors
```
'shellcheck ' returned error 1 finding the following syntactical issues:
--
In run-regression-test.sh line 157:
mkdir -p ${DORIS_HOME}/regression-test/suites/javaudf_p0/jars
^---^ SC2086 (info): Double quote to prevent globbing
and word splitting.
Did you mean:
mkdir -p "${DORIS_HOME}"/regression-test/suites/javaudf_p0/jars
In run-regression-test.sh line 158:
cd ${DORIS_HOME}/samples/doris-demo/java-udf-demo
^---^ SC2086 (info): Double quote to prevent globbing and
word splitting.
Did you mean:
cd "${DORIS_HOME}"/samples/doris-demo/java-udf-demo
In run-regression-test.sh line 160:
cp target/java-udf-demo-jar-with-dependencies.jar
${DORIS_HOME}/regression-test/suites/javaudf_p0/jars/
^---^ SC2086
(info): Double quote to prevent globbing and word splitting.
Did you mean:
cp target/java-udf-demo-jar-with-dependencies.jar
"${DORIS_HOME}"/regression-test/suites/javaudf_p0/jars/
For more information:
https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing
...
--
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=
above the line that contains the issue, where is the error code;
3. Add '-e ' to the SHELLCHECK_OPTS setting in your .yml action file.
```
shfmt errors
```
'shfmt ' found no issues.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on a diff in pull request #10170: [Enhancement](optimizer) Support select table sample
xinyiZzz commented on code in PR #10170:
URL: https://github.com/apache/doris/pull/10170#discussion_r991268951
##
fe/fe-core/src/main/java/org/apache/doris/analysis/TupleDescriptor.java:
##
@@ -159,6 +164,84 @@ public void setTable(TableIf tbl) {
table = tbl;
}
+public Set getSampleTabletIds() {
+return sampleTabletIds;
+}
+
+/**
+ * First, determine how many rows to sample from each partition according
to the number of partitions.
+ * Then determine the number of Tablets to be selected for each partition
according to the average number
+ * of rows of Tablet,
+ * If seek is not specified, the specified number of Tablets are
pseudo-randomly selected from each partition.
+ * If seek is specified, it will be selected sequentially from the seek
tablet of the partition.
+ * And add the manually specified Tablet id to the selected Tablet.
+ * simpleTabletNums = simpleRows / partitionNums / (partitionRows /
partitionTabletNums)
+ */
+public void computeSampleTabletIds(List tabletIds, TableSample
tableSample) {
Review Comment:
done, I moved `computeSampleTabletIds` to `analysis::olapScanNode`.
##
fe/fe-core/src/main/java/org/apache/doris/analysis/TupleDescriptor.java:
##
@@ -159,6 +164,80 @@ public void setTable(TableIf tbl) {
table = tbl;
}
+public Set getSampleTabletIds() {
+return sampleTabletIds;
+}
+
+/**
+ * First, determine how many rows to sample from each partition according
to the number of partitions.
+ * Then determine the number of Tablets to be selected for each partition
according to the average number
+ * of rows of Tablet,
+ * If seek is not specified, the specified number of Tablets are
pseudo-randomly selected from each partition.
+ * If seek is specified, it will be selected sequentially from the seek
tablet of the partition.
+ * And add the manually specified Tablet id to the selected Tablet.
+ * simpleTabletNums = simpleRows / partitionNums / (partitionRows /
partitionTabletNums)
+ */
+public void computeSampleTabletIds(List tabletIds, TableSample
tableSample) {
+if (table.getType() != TableType.OLAP) {
Review Comment:
done, I moved `computeSampleTabletIds` to `analysis::olapScanNode`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yangzhg opened a new pull request, #13267: [fix](DynamicPartition) Not check max_dynamic_partition_num when disable DynamicPartition
yangzhg opened a new pull request, #13267:
URL: https://github.com/apache/doris/pull/13267
# Proposed changes
Disable max_dynamic_partition_num check when disable DynamicPartition by
`ALTER TABLE tbl_name SET ("dynamic_partition.enable" = "false")`, when
`max_dynamic_partition_num` changed to larger and then changed to a lower
value, the actual dynamic partition num may larger than
`max_dynamic_partition_num`, and cannot disable DynamicPartition
## Problem summary
Describe your changes.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [x] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [x] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [x] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [x] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [x] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 opened a new issue, #13268: [Bug](be-ut)(aarch64) "sched_to itself" errors occur when running BE UT.
adonis0147 opened a new issue, #13268: URL: https://github.com/apache/doris/issues/13268 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version Doris version: master OS: Ubuntu 22.04.01 LTS aarch64 Compiler: Clang ### What's Wrong?  ### What You Expected? Run BE UT successfully. ### How to Reproduce? ```shell # 1. Build third-parties by Clang DORIS_TOOLCHAIN=clang ./build-thirdparty.sh # 2. Run BE UT DORIS_TOOLCHAIN=clang ./run-be-ut.sh --run ``` ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.1-lts updated: 1.1.3-rc02
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch branch-1.1-lts in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-1.1-lts by this push: new b4364b451e 1.1.3-rc02 b4364b451e is described below commit b4364b451ef37dbe15db5674551851733645ffbb Author: yiguolei AuthorDate: Tue Oct 11 10:26:15 2022 +0800 1.1.3-rc02 --- gensrc/script/gen_build_version.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gensrc/script/gen_build_version.sh b/gensrc/script/gen_build_version.sh index ece4e39824..5528ef9e31 100755 --- a/gensrc/script/gen_build_version.sh +++ b/gensrc/script/gen_build_version.sh @@ -25,7 +25,7 @@ # contains the build version based on the git hash or svn revision. ## -build_version="1.1.3-rc01" +build_version="1.1.3-rc02" unset LANG unset LC_CTYPE - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] liaoxin01 opened a new pull request, #13269: [enhancement](storage) set the segment cache capacity according to the open file limit of the process
liaoxin01 opened a new pull request, #13269: URL: https://github.com/apache/doris/pull/13269 # Proposed changes Issue Number: close #xxx ## Problem summary Now segment_cache_capacity is 100, which usually exceeds open file limit. A large number of file descriptor will not be eliminated because of segment_cache_capacity is so large, resulting in too many open files errors. The client will receive -3109 error. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] tag 1.1.3-rc02 created (now b4364b451e)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to tag 1.1.3-rc02 in repository https://gitbox.apache.org/repos/asf/doris.git at b4364b451e (commit) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 opened a new pull request, #13270: [Bug](aarch64) Fix the BRPC "sched_to itself" error when building BE by Clang on Linux aarch64
adonis0147 opened a new pull request, #13270: URL: https://github.com/apache/doris/pull/13270 # Proposed changes Issue Number: close #13268 ## Problem summary Upstream BRPC fixed this bug for GCC compiler. (commit: [c271e2e0](https://github.com/apache/incubator-brpc/commit/c271e2e0cc589047d832f72a381a6efadb6eeed8), issue: [apache/incubator-brpc#845](https://github.com/apache/incubator-brpc/issues/845)) This PR is for Clang compiler. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring commented on a diff in pull request #13269: [enhancement](storage) set the segment cache capacity according to the open file limit of the process
dataroaring commented on code in PR #13269:
URL: https://github.com/apache/doris/pull/13269#discussion_r991762793
##
be/src/runtime/exec_env_init.cpp:
##
@@ -287,7 +287,20 @@ Status ExecEnv::_init_mem_tracker() {
<< PrettyPrinter::print(storage_cache_limit, TUnit::BYTES)
<< ", origin config value: " << config::storage_page_cache_limit;
-SegmentLoader::create_global_instance(config::segment_cache_capacity);
+uint64_t fd_number = config::min_file_descriptor_number;
+struct rlimit l;
+int ret = getrlimit(RLIMIT_NOFILE, &l);
+if (ret != 0) {
+LOG(WARNING) << "call getrlimit() failed. errno=" << strerror(errno)
+ << ", use default configuration instead.";
+} else {
+fd_number = static_cast(l.rlim_cur);
+}
+// SegmentLoader caches segments in rowset granularity. So the size of
+// opened files will greater than segment_cache_capacity.
+uint64_t segment_cache_capacity = fd_number / 3 * 2;
+LOG(INFO) << "segment cache capacity: " << segment_cache_capacity;
Review Comment:
log out fd_number here and print segment_cache_capacity = fd_number / 3 * 2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] pengxiangyu commented on a diff in pull request #12897: [feature](remote) support local cache GC by disk usage
pengxiangyu commented on code in PR #12897:
URL: https://github.com/apache/doris/pull/12897#discussion_r991766090
##
be/src/io/cache/file_cache_manager.cpp:
##
@@ -56,92 +82,101 @@ void FileCacheManager::remove_file_cache(const
std::string& cache_path) {
}
}
-void FileCacheManager::clean_timeout_caches() {
-std::shared_lock rdlock(_cache_map_lock);
-for (std::map::const_iterator iter =
_file_cache_map.cbegin();
- iter != _file_cache_map.cend(); ++iter) {
-if (iter->second == nullptr) {
-continue;
+void
FileCacheManager::_add_file_cache_for_gc_by_disk(std::vector&
contexts,
+ FileCachePtr file_cache)
{
+// sort file cache by last match time
+if (config::file_cache_max_size_per_disk > 0) {
+auto file_size = file_cache->cache_file_size();
+if (file_size <= 0) {
+return;
+}
+for (size_t i = 0; i < contexts.size(); ++i) {
+if (contexts[i].try_add_file_cache(file_cache, file_size)) {
+break;
+}
}
-iter->second->clean_timeout_cache();
}
}
-void FileCacheManager::clean_timeout_file_not_in_mem(const std::string&
cache_path) {
-time_t now = time(nullptr);
-std::shared_lock rdlock(_cache_map_lock);
-// Deal with caches not in _file_cache_map
-if (_file_cache_map.find(cache_path) == _file_cache_map.end()) {
-std::vector cache_file_names;
-if (io::global_local_filesystem()->list(cache_path,
&cache_file_names).ok()) {
-std::map cache_names;
-std::list done_names;
-for (Path cache_file_name : cache_file_names) {
-std::string filename = cache_file_name.native();
-if (!ends_with(filename, CACHE_DONE_FILE_SUFFIX)) {
-cache_names[filename] = true;
-continue;
-}
-done_names.push_back(filename);
-std::stringstream done_file_ss;
-done_file_ss << cache_path << "/" << filename;
-std::string done_file_path = done_file_ss.str();
-time_t m_time;
-if (!FileUtils::mtime(done_file_path, &m_time).ok()) {
-continue;
-}
-if (now - m_time < config::file_cache_alive_time_sec) {
-continue;
-}
-std::string cache_file_path =
-StringReplace(done_file_path, CACHE_DONE_FILE_SUFFIX,
"", true);
-LOG(INFO) << "Delete timeout done_cache_path: " <<
done_file_path
- << ", cache_file_path: " << cache_file_path << ",
m_time: " << m_time;
-if
(!io::global_local_filesystem()->delete_file(done_file_path).ok()) {
-LOG(ERROR) << "delete_file failed: " << done_file_path;
-continue;
-}
-if
(!io::global_local_filesystem()->delete_file(cache_file_path).ok()) {
-LOG(ERROR) << "delete_file failed: " << cache_file_path;
-continue;
-}
-}
-// find cache file without done file.
-for (std::list::iterator itr = done_names.begin();
itr != done_names.end();
- ++itr) {
-std::string cache_filename = StringReplace(*itr,
CACHE_DONE_FILE_SUFFIX, "", true);
-if (cache_names.find(cache_filename) != cache_names.end()) {
-cache_names.erase(cache_filename);
-}
-}
-// remove cache file without done file
-for (std::map::iterator itr =
cache_names.begin();
- itr != cache_names.end(); ++itr) {
-std::stringstream cache_file_ss;
-cache_file_ss << cache_path << "/" << itr->first;
-std::string cache_file_path = cache_file_ss.str();
-time_t m_time;
-if (!FileUtils::mtime(cache_file_path, &m_time).ok()) {
+void FileCacheManager::_gc_unused_file_caches(std::list& result)
{
+std::vector tablets =
+StorageEngine::instance()->tablet_manager()->get_all_tablet();
+for (const auto& tablet : tablets) {
+std::vector seg_file_paths;
+if (io::global_local_filesystem()->list(tablet->tablet_path(),
&seg_file_paths).ok()) {
+for (Path seg_file : seg_file_paths) {
+std::string seg_filename = seg_file.native();
+// check if it is a dir name
+if (ends_with(seg_filename, ".dat")) {
continue;
}
-if (now - m_time < config::file_cache_alive_time_sec) {
+// skip file cache already in memory
+std::stringstream ss;
+ss << tablet->tablet_path() << "/" << seg_filename;
+
[GitHub] [doris] Kikyou1997 opened a new pull request, #13271: [fix](regression) Remove unstable unstable regression test for nereids
Kikyou1997 opened a new pull request, #13271: URL: https://github.com/apache/doris/pull/13271 # Proposed changes Issue Number: noissue ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on pull request #13271: [fix](regression) Remove unstable unstable regression test for nereids
Kikyou1997 commented on PR #13271: URL: https://github.com/apache/doris/pull/13271#issuecomment-1274019075 @morrySnow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #13243: [Enhancement](nereids) Remove redundant log when fall back to stale parser
morrySnow merged PR #13243: URL: https://github.com/apache/doris/pull/13243 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (6ee150755a -> b5da751c2a)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 6ee150755a [refactor](datax)Refactoring doris writer code (#13226) add b5da751c2a [enhancement](Nereids) remove redundant log when fall back to legacy parser (#13243) No new revisions were added by this update. Summary of changes: fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow opened a new pull request, #13272: [regression](Nereids) temporary remove test case 'tpch q21' until we fix it
morrySnow opened a new pull request, #13272: URL: https://github.com/apache/doris/pull/13272 # Proposed changes Issue Number: close #xxx ## Problem summary NPE thrown when we run regression test tpc-h q21 on Nereids ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13263: [fix](new-scann) scanner may be marked close twice
github-actions[bot] commented on PR #13263: URL: https://github.com/apache/doris/pull/13263#issuecomment-1274038827 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jacktengg opened a new pull request, #13273: [improvement](schema change) avoid using column ptr swap
jacktengg opened a new pull request, #13273: URL: https://github.com/apache/doris/pull/13273 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] LemonLiTree opened a new pull request, #13274: [typo](docs)delete duplicate document and fix some problem
LemonLiTree opened a new pull request, #13274: URL: https://github.com/apache/doris/pull/13274 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13274: [typo](docs)delete duplicate document and fix some problem
github-actions[bot] commented on PR #13274: URL: https://github.com/apache/doris/pull/13274#issuecomment-1274054069 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #13271: [fix](regression) Remove unstable regression test for nereids
dataroaring merged PR #13271: URL: https://github.com/apache/doris/pull/13271 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (b5da751c2a -> 2b5db24d82)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from b5da751c2a [enhancement](Nereids) remove redundant log when fall back to legacy parser (#13243) add 2b5db24d82 remove unstable q21 (#13271) No new revisions were added by this update. Summary of changes: .../suites/tpch_sf1_p1/tpch_sf1/nereids/q21.groovy | 109 - 1 file changed, 109 deletions(-) delete mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q21.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] LemonLiTree commented on issue #13251: [Bug] BE日志告警UpdateDerivedVars is too busy!服务器卡住
LemonLiTree commented on issue #13251: URL: https://github.com/apache/doris/issues/13251#issuecomment-1274060622 Hi，看下是不是OOM问题，是否做了内存使用限制？ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13259: test benchmark pipeline
github-actions[bot] commented on PR #13259:
URL: https://github.com/apache/doris/pull/13259#issuecomment-1274066196
`sh-checker report`
To get the full details, please check in the
[job]("https://github.com/apache/doris/actions/runs/3224086064";) output.
shellcheck errors
```
'shellcheck ' returned error 1 finding the following syntactical issues:
--
In teamcity/doris/benchmark/clickbench/common/check_and_kill_deleted_proc.sh
line 11:
check=$(lsof -i:$port|awk '{print $2}'|wc -l)
^---^ SC2086 (info): Double quote to prevent globbing and
word splitting.
^---^ SC2250 (style): Prefer putting braces around variable
references even when not strictly required.
Did you mean:
check=$(lsof -i:"$port"|awk '{print $2}'|wc -l)
In teamcity/doris/benchmark/clickbench/common/check_and_kill_deleted_proc.sh
line 12:
if [ "check$check" != "check0" ];then
^---^ SC2292 (style): Prefer [[ ]] over [ ] for
tests in Bash/Ksh.
^^ SC2250 (style): Prefer putting braces around variable
references even when not strictly required.
Did you mean:
if [[ "check${check}" != "check0" ]];then
In teamcity/doris/benchmark/clickbench/common/check_and_kill_deleted_proc.sh
line 13:
if [ "check"${case_level} == "check" ];then
^-- SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
^---^ SC2086 (info): Double quote to prevent
globbing and word splitting.
Did you mean:
if [[ "check""${case_level}" == "check" ]];then
In teamcity/doris/benchmark/clickbench/common/check_and_kill_deleted_proc.sh
line 14:
check_res=$(pwdx `lsof -i:$port|grep -v PID|awk '{print $2}'`|grep
$env_name|grep deleted||wc -l)
^-- SC2046 (warning): Quote this to prevent word
splitting.
^-- SC2006 (style): Use $(...) notation instead of
legacy backticks `...`.
^---^ SC2086 (info): Double quote to
prevent globbing and word splitting.
^---^ SC2250 (style): Prefer putting
braces around variable references even when not strictly required.
^---^ SC2086 (info): Double quote to prevent globbing and word splitting.
^---^ SC2250 (style): Prefer putting braces around variable references even
when not strictly required.
Did you mean:
check_res=$(pwdx $(lsof -i:"$port"|grep -v PID|awk '{print
$2}')|grep $env_name|grep deleted||wc -l)
In teamcity/doris/benchmark/clickbench/common/check_and_kill_deleted_proc.sh
line 16:
check_res=$(pwdx `lsof -i:$port|grep -v PID|awk '{print $2}'`|grep
$env_name|grep $case_level|grep deleted||wc -l)
^-- SC2046 (warning): Quote this to prevent word
splitting.
^-- SC2006 (style): Use $(...) notation instead of
legacy backticks `...`.
^---^ SC2086 (info): Double quote to
prevent globbing and word splitting.
^---^ SC2250 (style): Prefer putting
braces around variable references even when not strictly required.
^---^ SC2086 (info): Double quote to prevent globbing and word splitting.
^---^ SC2250 (style): Prefer putting braces around variable references even
when not strictly required.
^-^ SC2086 (info): Double quote to prevent globbing and
word splitting.
^-^ SC2250 (style): Prefer putting braces around variable
references even when not strictly required.
Did you mean:
check_res=$(pwdx $(lsof -i:"$port"|grep -v PID|awk '{print
$2}')|grep $env_name|grep $case_level|grep deleted||wc -l)
In teamcity/doris/benchmark/clickbench/common/check_and_kill_deleted_proc.sh
line 18:
if [ "check$check_res" != "check0" ];then
^-- SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
^^ SC2250 (style): Prefer putting braces around
variable references even when not strictly required.
Did you mean:
if [[ "check${check_res}" != "check0" ]];then
In teamcity/doris/benchmark/clickbench/common/check_and_kill_deleted_proc.sh
line 19:
be_pid=(`lsof -i:$port|grep -v PID|awk '{print $2}'`)
^-- SC2207 (warning): Prefer mapfil
[GitHub] [doris] mrhhsg opened a new pull request, #13275: cherry-pick #11516 and #11822 from master
mrhhsg opened a new pull request, #13275: URL: https://github.com/apache/doris/pull/13275 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 opened a new pull request, #13276: [function](datev2) Complements for datev2 functions
Gabriel39 opened a new pull request, #13276: URL: https://github.com/apache/doris/pull/13276 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13274: [typo](docs)delete duplicate document and fix some problem
github-actions[bot] commented on PR #13274: URL: https://github.com/apache/doris/pull/13274#issuecomment-1274075809 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 closed pull request #13258: [config](datev2) set `enable_date_conversion = true` by default
Gabriel39 closed pull request #13258: [config](datev2) set `enable_date_conversion = true` by default URL: https://github.com/apache/doris/pull/13258 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] weizhengte commented on a diff in pull request #10170: [Enhancement](optimizer) Support select table sample
weizhengte commented on code in PR #10170:
URL: https://github.com/apache/doris/pull/10170#discussion_r991818187
##
fe/fe-core/src/main/java/org/apache/doris/analysis/TableSample.java:
##
@@ -0,0 +1,101 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.common.AnalysisException;
+
+/*
+ * To represent following stmt:
+ * TABLESAMPLE (10 PERCENT)
+ * TABLESAMPLE (100 ROWS)
+ * TABLESAMPLE (10 PERCENT) REPEATABLE (123)
+ * TABLESAMPLE (100 ROWS) REPEATABLE (123)R
+ *
+ * references:
+ *
https://simplebiinsights.com/sql-server-tablesample-retrieving-random-data-from-sql-server/
+ * https://sqlrambling.net/2018/01/24/tablesample-basic-examples/
+ */
+public class TableSample implements ParseNode {
+
+private final Long sampleValue;
+private final boolean isPercent;
+private final Long seek;
+
+public TableSample(boolean isPercent, Long sampleValue) {
+this.sampleValue = sampleValue;
+this.isPercent = isPercent;
+this.seek = -1L;
Review Comment:
> sry, I meant to give an initial value, which means `private Long seek =
-1;`, and remove `this.seek = -1L;`, It looks better. but it doesn't matter
much, you can ignore it~
Later I mean no `final`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #13277: [test][not-merge] test ci
morningman opened a new pull request, #13277: URL: https://github.com/apache/doris/pull/13277 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] weizhengte commented on a diff in pull request #10170: [Enhancement](optimizer) Support select table sample
weizhengte commented on code in PR #10170:
URL: https://github.com/apache/doris/pull/10170#discussion_r991818187
##
fe/fe-core/src/main/java/org/apache/doris/analysis/TableSample.java:
##
@@ -0,0 +1,101 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.common.AnalysisException;
+
+/*
+ * To represent following stmt:
+ * TABLESAMPLE (10 PERCENT)
+ * TABLESAMPLE (100 ROWS)
+ * TABLESAMPLE (10 PERCENT) REPEATABLE (123)
+ * TABLESAMPLE (100 ROWS) REPEATABLE (123)R
+ *
+ * references:
+ *
https://simplebiinsights.com/sql-server-tablesample-retrieving-random-data-from-sql-server/
+ * https://sqlrambling.net/2018/01/24/tablesample-basic-examples/
+ */
+public class TableSample implements ParseNode {
+
+private final Long sampleValue;
+private final boolean isPercent;
+private final Long seek;
+
+public TableSample(boolean isPercent, Long sampleValue) {
+this.sampleValue = sampleValue;
+this.isPercent = isPercent;
+this.seek = -1L;
Review Comment:
> sry, I meant to give an initial value, which means `private Long seek =
-1;`, and remove `this.seek = -1L;`, It looks better. but it doesn't matter
much, you can ignore it~
I mean no `final`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
