[GitHub] [doris] morningman commented on pull request #13145: [feature-wip](CN Node)Support Compute Only Node in doris

[GitBox]

morningman commented on PR #13145:
URL: https://github.com/apache/doris/pull/13145#issuecomment-1272249801

   link to the wrong issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] SaintBacchus commented on pull request #13145: [feature-wip](CN Node)Support Compute Only Node in doris

[GitBox]

SaintBacchus commented on PR #13145:
URL: https://github.com/apache/doris/pull/13145#issuecomment-1272250733

   > link to the wrong issue?
   
   yeah, fix now
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yuanyuan8983 opened a new pull request, #13169: [typo](docs) Fix the jump link 404 in basic usage.md

[GitBox]

yuanyuan8983 opened a new pull request, #13169:
URL: https://github.com/apache/doris/pull/13169

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] englefly commented on a diff in pull request #12987: [feature](nereids) refactor statistics framework and introduce StatsCalculatorV2

[GitBox]

englefly commented on code in PR #12987:
URL: https://github.com/apache/doris/pull/12987#discussion_r990604144


##
fe/fe-core/src/main/java/org/apache/doris/statistics/StatsDeriveResult.java:
##
@@ -146,4 +147,22 @@ public StatsDeriveResult merge(StatsDeriveResult other) {
 public StatsDeriveResult copy() {
 return new StatsDeriveResult(this);
 }
+
+public StatsDeriveResult updateRowCountOnCopy(double selectivity) {
+StatsDeriveResult copy = new StatsDeriveResult(this);
+copy.setRowCount(rowCount * selectivity);
+for (Entry entry : 
copy.slotToColumnStats.entrySet()) {
+entry.getValue().updateBySelectivity(selectivity, rowCount);

Review Comment:
   this logic is not solid. 
   select A , B from T where B=1.
   if the selectivity is 0.1, there is no reason to say that A.ndv reduced by 
10 times.



##
fe/fe-core/src/main/java/org/apache/doris/statistics/TableStats.java:
##
@@ -214,12 +187,126 @@ private PartitionStats getNotNullPartitionStats(String 
partitionName) {
  * @param columnName column name
  * @return @ColumnStats
  */
-private ColumnStats getNotNullColumnStats(String columnName) {
-ColumnStats columnStats = nameToColumnStats.get(columnName);
-if (columnStats == null) {
-columnStats = new ColumnStats();
-nameToColumnStats.put(columnName, columnStats);
+private ColumnStat getNotNullColumnStats(String columnName) {
+ColumnStat columnStat = nameToColumnStats.get(columnName);
+if (columnStat == null) {
+columnStat = new ColumnStat();
+nameToColumnStats.put(columnName, columnStat);
 }
-return columnStats;
+return columnStat;
+}
+
+public ColumnStat getColumnStats(String columnName) {
+ColumnStat columnStat = nameToColumnStats.get(columnName);
+if (columnStat == null) {
+columnStat = new ColumnStat();
+nameToColumnStats.put(columnName, columnStat);
+}
+return columnStat;
+}
+
+public ColumnStat getColumnStatCopy(String columnName) {
+ColumnStat columnStat = getColumnStats(columnName);
+return columnStat.copy();
+}
+
+public List getShowInfo() {
+List result = Lists.newArrayList();
+result.add(Double.toString(getRowCount()));
+result.add(Long.toString(getDataSize()));
+return result;
+}
+
+public List getShowInfo(String partitionName) {
+PartitionStats partitionStats = 
nameToPartitionStats.get(partitionName);
+return partitionStats.getShowInfo();
+}
+
+private Map getAggPartitionColStats() {
+Map aggColumnStats = new HashMap<>();
+for (PartitionStats partitionStats : nameToPartitionStats.values()) {
+partitionStats.getNameToColumnStats().forEach((colName, 
columnStats) -> {
+if (!aggColumnStats.containsKey(colName)) {
+aggColumnStats.put(colName, columnStats);
+} else {
+ColumnStat tblColStats = aggColumnStats.get(colName);
+mergePartitionColumnStats(tblColStats, columnStats);
+}
+});
+}
+
+return aggColumnStats;
+}
+
+private void mergePartitionColumnStats(ColumnStat leftStats, ColumnStat 
rightStats) {
+if (leftStats.getNdv() == -1) {
+if (rightStats.getNdv() != -1) {
+leftStats.setNdv(rightStats.getNdv());
+}
+} else {
+if (rightStats.getNdv() != -1) {
+double ndv = leftStats.getNdv() + rightStats.getNdv();
+leftStats.setNdv(ndv);
+}
+}
+
+if (leftStats.getAvgSizeByte() == -1) {
+if (rightStats.getAvgSizeByte() != -1) {
+leftStats.setAvgSizeByte(rightStats.getAvgSizeByte());
+}
+} else {
+if (rightStats.getAvgSizeByte() != -1) {
+double avgSize = (leftStats.getAvgSizeByte() + 
rightStats.getAvgSizeByte()) / 2;
+leftStats.setAvgSizeByte(avgSize);
+}
+}
+
+if (leftStats.getMaxSizeByte() == -1) {
+if (rightStats.getMaxSizeByte() != -1) {
+leftStats.setMaxSizeByte(rightStats.getMaxSizeByte());
+}
+} else {
+if (rightStats.getMaxSizeByte() != -1) {
+double maxSize = Math.max(leftStats.getMaxSizeByte(), 
rightStats.getMaxSizeByte());
+leftStats.setMaxSizeByte(maxSize);
+}
+}
+
+if (leftStats.getNumNulls() == -1) {
+if (rightStats.getNumNulls() != -1) {
+leftStats.setNumNulls(rightStats.getNumNulls());
+}
+} else {
+if (rightStats.getNumNulls() != -1) {
+double numNulls = leftStats.getNumNulls() + 
rightStats.getNumNulls();
+

[GitHub] [doris] englefly commented on a diff in pull request #12987: [feature](nereids) refactor statistics framework and introduce StatsCalculatorV2

[GitBox]

englefly commented on code in PR #12987:
URL: https://github.com/apache/doris/pull/12987#discussion_r990582926


##
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/ExpressionEstimation.java:
##
@@ -0,0 +1,210 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.stats;
+
+import org.apache.doris.nereids.trees.expressions.Add;
+import org.apache.doris.nereids.trees.expressions.BinaryArithmetic;
+import org.apache.doris.nereids.trees.expressions.Divide;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Multiply;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.Subtract;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Avg;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Count;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Max;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Min;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Sum;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.Substring;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.WeekOfYear;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.Year;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.util.Utils;
+import org.apache.doris.statistics.ColumnStat;
+import org.apache.doris.statistics.StatsDeriveResult;
+
+import com.google.common.base.Preconditions;
+
+/**
+ * Used to estimate for expressions that not producing boolean value.
+ */
+public class ExpressionEstimation extends ExpressionVisitor {
+
+private static ExpressionEstimation INSTANCE = new ExpressionEstimation();
+
+public static ColumnStat estimate(Expression expression, StatsDeriveResult 
stats) {
+return INSTANCE.visit(expression, stats);
+}
+
+@Override
+public ColumnStat visit(Expression expr, StatsDeriveResult context) {
+return expr.accept(this, context);
+}
+
+@Override
+public ColumnStat visitLiteral(Literal literal, StatsDeriveResult context) 
{
+if (literal.isStringLiteral()) {
+return ColumnStat.UNKNOWN;
+}
+double literalVal = Double.parseDouble(literal.getValue().toString());
+ColumnStat columnStat = new ColumnStat();
+columnStat.setMaxValue(literalVal);
+columnStat.setMinValue(literalVal);
+columnStat.setNdv(1);
+columnStat.setNumNulls(1);
+columnStat.setAvgSizeByte(1);
+return columnStat;
+}
+
+@Override
+public ColumnStat visitSlotReference(SlotReference slotReference, 
StatsDeriveResult context) {
+ColumnStat columnStat = context.getColumnStatsBySlot(slotReference);
+Preconditions.checkState(columnStat != null);
+return columnStat;
+}
+
+@Override
+public ColumnStat visitBinaryArithmetic(BinaryArithmetic binaryArithmetic, 
StatsDeriveResult context) {
+ColumnStat leftColStats = binaryArithmetic.left().accept(this, 
context);
+ColumnStat rightColStats = binaryArithmetic.right().accept(this, 
context);
+double leftNdv = leftColStats.getNdv();
+double rightNdv = rightColStats.getNdv();
+double ndv = Math.max(leftNdv, rightNdv);
+double leftNullCount = leftColStats.getNumNulls();
+double rightNullCount = rightColStats.getNumNulls();
+double rowCount = context.getRowCount();

Review Comment:
   this rowCount is for left, right or binary result?
   in L92, `leftNullCount / rowCount ` makes sense only if rowCount is the left 
row count.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
T

[GitHub] [doris] englefly commented on a diff in pull request #12987: [feature](nereids) refactor statistics framework and introduce StatsCalculatorV2

[GitBox]

englefly commented on code in PR #12987:
URL: https://github.com/apache/doris/pull/12987#discussion_r990580515


##
fe/fe-core/src/main/java/org/apache/doris/analysis/SlotDescriptor.java:
##
@@ -219,7 +219,7 @@ public ColumnStats getStats() {
 }
 }
 // FIXME(dhc): mock ndv
-stats.setNumDistinctValues(parent.getCardinality());
+stats.setNumDistinctValues((long) parent.getCardinality());

Review Comment:
   getCardinality() return type is already 'long'



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-website] DongLiang-0 opened a new pull request, #125: fix 404 links

[GitBox]

DongLiang-0 opened a new pull request, #125:
URL: https://github.com/apache/doris-website/pull/125

   fix 404 links


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-website] yuanyuan8983 opened a new pull request, #126: [typo](docs) Fix the jump link 404

[GitBox]

yuanyuan8983 opened a new pull request, #126:
URL: https://github.com/apache/doris-website/pull/126

   Fix the jump link 404 in basic usage.md


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-website] yuanyuan8983 opened a new pull request, #127: [typo](docs) Fix jump link 404

[GitBox]

yuanyuan8983 opened a new pull request, #127:
URL: https://github.com/apache/doris-website/pull/127

   Fix jump link 404 in kafka load.md


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-website] hf200012 merged pull request #124: fix link

[GitBox]

hf200012 merged PR #124:
URL: https://github.com/apache/doris-website/pull/124


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris-website] branch master updated: fix link (#124)

[jiafengzheng]

This is an automated email from the ASF dual-hosted git repository.

jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git


The following commit(s) were added to refs/heads/master by this push:
 new ef42cb667bb fix link (#124)
ef42cb667bb is described below

commit ef42cb667bb04d1f4bc386ae2787a17dcedd795d
Author: FreeOnePlus <54164178+freeonep...@users.noreply.github.com>
AuthorDate: Sat Oct 8 15:46:19 2022 +0800

fix link (#124)

fix link
---
 docs/releasenotes/release-1.1.0.md| 4 +---
 .../sql-reference/Data-Definition-Statements/Drop/DROP-DATABASE.md| 2 +-
 .../current/ecosystem/doris-manager/space-list.md | 2 +-
 .../current/releasenotes/release-1.1.0.md | 4 +---
 .../Data-Definition-Statements/Alter/ALTER-TABLE-PARTITION.md | 2 +-
 .../Data-Definition-Statements/Alter/ALTER-TABLE-REPLACE.md   | 2 +-
 .../Data-Definition-Statements/Alter/ALTER-TABLE-ROLLUP.md| 2 +-
 .../sql-reference/Data-Definition-Statements/Drop/DROP-DATABASE.md| 2 +-
 .../Show-Statements/SHOW-ALTER-TABLE-MATERIALIZED-VIEW.md | 2 +-
 9 files changed, 9 insertions(+), 13 deletions(-)

diff --git a/docs/releasenotes/release-1.1.0.md 
b/docs/releasenotes/release-1.1.0.md
index c58a1ff46ff..e9e517d7c00 100644
--- a/docs/releasenotes/release-1.1.0.md
+++ b/docs/releasenotes/release-1.1.0.md
@@ -42,9 +42,7 @@ BE binary file has been renamed from palo_be to doris_be . 
Please pay attention
 
 The storage format of earlier versions of Apache Doris was Segment V1. In 
version 0.12, we had implemented Segment V2 as a new storage format, which 
introduced Bitmap indexes, memory tables, page cache, dictionary compression, 
delayed materialization and many other features. Starting from version 0.13, 
the default storage format for newly created tables is Segment V2, while 
maintaining compatibility with the Segment V1 format.
 
-In order to ensure the maintainability of the code structure and reduce the 
additional learning and development costs caused by redundant historical codes, 
we have decided to no longer support the Segment v1 storage format from the 
next version. It is expected that this part of the code will be deleted in the 
Apache Doris 1.2 version, and all users who are still using the Segment V1 
storage format must complete the data format conversion in version 1.1. Please 
refer to the following link [...]
-
-[https://doris.apache.org/zh-CN/docs/1.0/administrator-guide/segment-v2-usage](https://doris.apache.org/zh-CN/docs/1.0/administrator-guide/segment-v2-usage)
+In order to ensure the maintainability of the code structure and reduce the 
additional learning and development costs caused by redundant historical codes, 
we have decided to no longer support the Segment v1 storage format from the 
next version. It is expected that this part of the code will be deleted in the 
Apache Doris 1.2 version.
 
 ### Normal Upgrade
 
diff --git 
a/docs/sql-manual/sql-reference/Data-Definition-Statements/Drop/DROP-DATABASE.md
 
b/docs/sql-manual/sql-reference/Data-Definition-Statements/Drop/DROP-DATABASE.md
index 05fdaed73ae..ef0ab3273d1 100644
--- 
a/docs/sql-manual/sql-reference/Data-Definition-Statements/Drop/DROP-DATABASE.md
+++ 
b/docs/sql-manual/sql-reference/Data-Definition-Statements/Drop/DROP-DATABASE.md
@@ -41,7 +41,7 @@ DROP DATABASE [IF EXISTS] db_name [FORCE];
 
 illustrate:
 
-- During the execution of DROP DATABASE, the deleted database can be recovered 
through the RECOVER statement. See the 
[RECOVER](../../Data-Definition-Statements/Backup-and-Restore/RECOVER) 
statement for details
+- During the execution of DROP DATABASE, the deleted database can be recovered 
through the RECOVER statement. See the 
[RECOVER](../../Database-Administration-Statements/RECOVER) statement for 
details
 - If you execute DROP DATABASE FORCE, the system will not check the database 
for unfinished transactions, the database will be deleted directly and cannot 
be recovered, this operation is generally not recommended
 
 ### Example
diff --git 
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/ecosystem/doris-manager/space-list.md
 
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/ecosystem/doris-manager/space-list.md
index 78abf4e980c..fdbb578f8e5 100644
--- 
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/ecosystem/doris-manager/space-list.md
+++ 
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/ecosystem/doris-manager/space-list.md
@@ -104,7 +104,7 @@ ssh agen...@xx.xxx.xx.xx
 
 1. 代码包路径
 
-   通过Doris Manager 进行集群部署时，需要提供已编译好的 Doris 安装包，您可以通过 Doris 
源码自行编译，或使用官方提供的[二进制版本](https://doris.apache.org/zh-CN/downloads/downloads.html)。
+   通过Doris Manager 进行集群部署时，需要提供已编译好的 Doris 安装包，您可以通过 Doris 
源码自行编译，或使用官方提供的[二进制版本](https://doris.apache.org/zh-CN/download)。
 
 `Doris Manager 将通过 http 方式拉取Doris安装包，若您需要自建 http 服务，请参考文档底部-自建http服务`。
 

[GitHub] [doris] yuanyuan8983 opened a new pull request, #13170: [typo](docs)Fix jump link 404 in jdbc load.md

[GitBox]

yuanyuan8983 opened a new pull request, #13170:
URL: https://github.com/apache/doris/pull/13170

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-website] yuanyuan8983 opened a new pull request, #128: [typo](docs)Fix jump link 404 in jdbc load.md

[GitBox]

yuanyuan8983 opened a new pull request, #128:
URL: https://github.com/apache/doris-website/pull/128

   Fix jump link 404 in jdbc load.md


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] caoliang-web opened a new pull request, #13171: [typo](docs)fix error url

[GitBox]

caoliang-web opened a new pull request, #13171:
URL: https://github.com/apache/doris/pull/13171

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   fix error url
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-website] caoliang-web opened a new pull request, #129: [typo](docs)fix error url

[GitBox]

caoliang-web opened a new pull request, #129:
URL: https://github.com/apache/doris-website/pull/129

   fix error url


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-website] yuanyuan8983 opened a new pull request, #130: [typo](docs)Fix jump link 404 in external storage load.md

[GitBox]

yuanyuan8983 opened a new pull request, #130:
URL: https://github.com/apache/doris-website/pull/130

   Fix jump link 404 in external storage load.md


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] SeekingYang opened a new pull request, #13172: Update outfile.md

[GitBox]

SeekingYang opened a new pull request, #13172:
URL: https://github.com/apache/doris/pull/13172

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] 01/03: (runtimefilter) shorter time prepare consumes (#13127)

[morningman]

This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch branch-1.1-lts
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 65247d03b081f1b2618efab18c6b262fc28850fe
Author: Yongqiang YANG <98214048+dataroar...@users.noreply.github.com>
AuthorDate: Thu Oct 6 10:12:29 2022 +0800

(runtimefilter) shorter time prepare consumes (#13127)

Now, every preare put a runtime filter controller, so it takes the
mutex lock on the controller map. Init of bloom filter takes some
time in allocate and memset. If we run p1 tests with -parallel=20
-suiteParallel=20 -actionParallel=20, then we get error message like
'send fragment timeout 5s'.

The patch fixes the problem in the following 2 ways:
1. Replace one mutex block with 128s.
2. If a plan fragment does not have a runtime filter, it does not need to 
take
the locks.
---
 be/src/runtime/runtime_filter_mgr.cpp | 32 
 be/src/runtime/runtime_filter_mgr.h   | 15 +++
 2 files changed, 31 insertions(+), 16 deletions(-)

diff --git a/be/src/runtime/runtime_filter_mgr.cpp 
b/be/src/runtime/runtime_filter_mgr.cpp
index b5302aeace..ffea62a155 100644
--- a/be/src/runtime/runtime_filter_mgr.cpp
+++ b/be/src/runtime/runtime_filter_mgr.cpp
@@ -272,48 +272,56 @@ Status RuntimeFilterMergeControllerEntity::merge(const 
PMergeFilterRequest* requ
 Status RuntimeFilterMergeController::add_entity(
 const TExecPlanFragmentParams& params,
 std::shared_ptr* handle) {
+if (!params.params.__isset.runtime_filter_params ||
+params.params.runtime_filter_params.rid_to_runtime_filter.size() == 0) 
{
+return Status::OK();
+}
+
 runtime_filter_merge_entity_closer entity_closer =
 std::bind(runtime_filter_merge_entity_close, this, 
std::placeholders::_1);
 
-std::lock_guard guard(_controller_mutex);
 UniqueId query_id(params.params.query_id);
 std::string query_id_str = query_id.to_string();
-auto iter = _filter_controller_map.find(query_id_str);
 UniqueId fragment_instance_id = 
UniqueId(params.params.fragment_instance_id);
+uint32_t shard = _get_controller_shard_idx(query_id);
+std::lock_guard guard(_controller_mutex[shard]);
+auto iter = _filter_controller_map[shard].find(query_id_str);
 
-if (iter == _filter_controller_map.end()) {
+if (iter == _filter_controller_map[shard].end()) {
 *handle = std::shared_ptr(
 new RuntimeFilterMergeControllerEntity(), entity_closer);
-_filter_controller_map[query_id_str] = *handle;
+_filter_controller_map[shard][query_id_str] = *handle;
 const TRuntimeFilterParams& filter_params = 
params.params.runtime_filter_params;
 if (params.params.__isset.runtime_filter_params) {
 RETURN_IF_ERROR(handle->get()->init(query_id, 
fragment_instance_id, filter_params, params.query_options));
 }
 } else {
-*handle = _filter_controller_map[query_id_str].lock();
+*handle = _filter_controller_map[shard][query_id_str].lock();
 }
 return Status::OK();
 }
 
 Status RuntimeFilterMergeController::acquire(
 UniqueId query_id, 
std::shared_ptr* handle) {
-std::lock_guard guard(_controller_mutex);
+uint32_t shard = _get_controller_shard_idx(query_id);
+std::lock_guard guard(_controller_mutex[shard]);
 std::string query_id_str = query_id.to_string();
-auto iter = _filter_controller_map.find(query_id_str);
-if (iter == _filter_controller_map.end()) {
+auto iter = _filter_controller_map[shard].find(query_id_str);
+if (iter == _filter_controller_map[shard].end()) {
 LOG(WARNING) << "not found entity, query-id:" << query_id_str;
 return Status::InvalidArgument("not found entity");
 }
-*handle = _filter_controller_map[query_id_str].lock();
+*handle = _filter_controller_map[shard][query_id_str].lock();
 if (*handle == nullptr) {
 return Status::InvalidArgument("entity is closed");
 }
 return Status::OK();
 }
 
-Status RuntimeFilterMergeController::remove_entity(UniqueId queryId) {
-std::lock_guard guard(_controller_mutex);
-_filter_controller_map.erase(queryId.to_string());
+Status RuntimeFilterMergeController::remove_entity(UniqueId query_id) {
+uint32_t shard = _get_controller_shard_idx(query_id);
+std::lock_guard guard(_controller_mutex[shard]);
+_filter_controller_map[shard].erase(query_id.to_string());
 return Status::OK();
 }
 
diff --git a/be/src/runtime/runtime_filter_mgr.h 
b/be/src/runtime/runtime_filter_mgr.h
index 653ce675b2..346a3cbab8 100644
--- a/be/src/runtime/runtime_filter_mgr.h
+++ b/be/src/runtime/runtime_filter_mgr.h
@@ -22,6 +22,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include "common/object_pool.h"
 #include "common/status.h"
@@ -163,16 +164,22 @@ public:
 // thread safe
 // remove a entity by quer

[doris] branch branch-1.1-lts updated (ed831aec90 -> 844cfceb76)

[morningman]

This is an automated email from the ASF dual-hosted git repository.

morningman pushed a change to branch branch-1.1-lts
in repository https://gitbox.apache.org/repos/asf/doris.git


from ed831aec90 Revert "[improvement](memory) set TCMALLOC_HEAP_LIMIT_MB to 
control memory consumption of tcmalloc (#12981)" (#13152)
 new 65247d03b0 (runtimefilter) shorter time prepare consumes (#13127)
 new ea7d9ec34f [fix](string) allocate memory according to actual size 
instead of max size (#13112)
 new 844cfceb76 [improvement](load) config flush_thread_num_per_store to be 
6 by default (#13076)

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 be/src/common/config.h|  2 +-
 be/src/olap/wrapper_field.cpp |  6 +-
 be/src/runtime/runtime_filter_mgr.cpp | 32 
 be/src/runtime/runtime_filter_mgr.h   | 15 +++
 4 files changed, 33 insertions(+), 22 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] 03/03: [improvement](load) config flush_thread_num_per_store to be 6 by default (#13076)

[morningman]

This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch branch-1.1-lts
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 844cfceb76ecd865454ba1d1600d55e69c22b316
Author: Yongqiang YANG <98214048+dataroar...@users.noreply.github.com>
AuthorDate: Sat Oct 8 09:16:22 2022 +0800

[improvement](load) config flush_thread_num_per_store to be 6 by default 
(#13076)

Flushing memtable is cpu bound, so 2 thread for a disk is tool small.
---
 be/src/common/config.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/be/src/common/config.h b/be/src/common/config.h
index 8e91db663a..dfba339af5 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -540,7 +540,7 @@ CONF_mInt32(storage_flood_stage_usage_percent, "90"); // 90%
 // The min bytes that should be left of a data dir
 CONF_mInt64(storage_flood_stage_left_capacity_bytes, "1073741824"); // 1GB
 // number of thread for flushing memtable per store
-CONF_Int32(flush_thread_num_per_store, "2");
+CONF_Int32(flush_thread_num_per_store, "6");
 // number of thread for flushing memtable per store, for high priority load 
task
 CONF_Int32(high_priority_flush_thread_num_per_store, "1");
 


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] 02/03: [fix](string) allocate memory according to actual size instead of max size (#13112)

[morningman]

This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch branch-1.1-lts
in repository https://gitbox.apache.org/repos/asf/doris.git

commit ea7d9ec34f9ff611e0e0c0019bf7ac250316bbc6
Author: Yongqiang YANG <98214048+dataroar...@users.noreply.github.com>
AuthorDate: Thu Oct 6 09:56:22 2022 +0800

[fix](string) allocate memory according to actual size instead of max size 
(#13112)

String column lengh is 2GB, if we allocate memory according to column 
length,
string would consume a lot of memory. It also misleads memory tracker.
---
 be/src/olap/wrapper_field.cpp | 6 +-
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/be/src/olap/wrapper_field.cpp b/be/src/olap/wrapper_field.cpp
index 887b93b259..ffb4516193 100644
--- a/be/src/olap/wrapper_field.cpp
+++ b/be/src/olap/wrapper_field.cpp
@@ -52,11 +52,7 @@ WrapperField* WrapperField::create(const TabletColumn& 
column, uint32_t len) {
 variable_len =
 std::max(len, static_cast(column.length() - 
sizeof(VarcharLengthType)));
 } else if (column.type() == OLAP_FIELD_TYPE_STRING) {
-// column.length is the serialized varchar length
-// the first sizeof(StringLengthType) bytes is the length of varchar
-// variable_len is the real length of varchar
-variable_len =
-std::max(len, static_cast(column.length() - 
sizeof(StringLengthType)));
+variable_len = len;
 } else {
 variable_len = column.length();
 }


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] yuanyuan8983 opened a new pull request, #13173: [typo](docs)Fix jump link 404 in external storage load.md

[GitBox]

yuanyuan8983 opened a new pull request, #13173:
URL: https://github.com/apache/doris/pull/13173

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] hf200012 commented on a diff in pull request #13172: Update outfile.md

[GitBox]

hf200012 commented on code in PR #13172:
URL: https://github.com/apache/doris/pull/13172#discussion_r990611468


##
docs/en/docs/admin-manual/config/be-config.md:
##
@@ -450,7 +450,7 @@ Cgroups assigned to doris
 ### `doris_max_scan_key_num`
 
 * Type: int
-* Description: Used to limit the maximum number of scan keys that a scan node 
can split in a query request. When a conditional query request reaches the scan 
node, the scan node will try to split the conditions related to the key column 
in the query condition into multiple scan key ranges. After that, these scan 
key ranges will be assigned to multiple scanner threads for data scanning. A 
larger value usually means that more scanner threads can be used to increase 
the parallelism of the scanning operation. However, in high concurrency 
scenarios, too many threads may bring greater scheduling overhead and system 
load, and will slow down the query response speed. An empirical value is 50. 
This configuration can be configured separately at the session level. For 
details, please refer to the description of `max_scan_key_num` in 
[Variables](../../advanced/variables.md).
+* Description: Used to limit the maximum number of scan keys that a scan node 
can split in a query request. When a conditional query request reaches the scan 
node, the scan node will try to split the conditions related to the key column 
in the query condition into multiple scan key ranges. After that, these scan 
key ranges will be assigned to multiple scanner threads for data scanning. A 
larger value usually means that more scanner threads can be used to increase 
the parallelism of the scanning operation. However, in high concurrency 
scenarios, too many threads may bring greater scheduling overhead and system 
load, and will slow down the query response speed. An empirical value is 50. 
This configuration can be configured separately at the session level. For 
details, please refer to the description of `max_scan_key_num` in 
[Variables](../../../advanced/variables.md).

Review Comment:
   ```suggestion
   * Description: Used to limit the maximum number of scan keys that a scan 
node can split in a query request. When a conditional query request reaches the 
scan node, the scan node will try to split the conditions related to the key 
column in the query condition into multiple scan key ranges. After that, these 
scan key ranges will be assigned to multiple scanner threads for data scanning. 
A larger value usually means that more scanner threads can be used to increase 
the parallelism of the scanning operation. However, in high concurrency 
scenarios, too many threads may bring greater scheduling overhead and system 
load, and will slow down the query response speed. An empirical value is 50. 
This configuration can be configured separately at the session level. For 
details, please refer to the description of `max_scan_key_num` in 
[Variables](../../../advanced/variables).
   ```



##
docs/en/docs/data-operate/export/outfile.md:
##
@@ -106,7 +106,7 @@ Planning example for concurrent export:
 
 ## Usage example
 
-For details, please refer to [OUTFILE 
Document](../sql-reference/sql-statements/Data%20Manipulation/OUTFILE.md).
+For details, please refer to [OUTFILE 
Document](../../sql-manual/sql-reference/Data-Manipulation-Statements/OUTFILE.md).

Review Comment:
   ```suggestion
   For details, please refer to [OUTFILE 
Document](../../sql-manual/sql-reference/Data-Manipulation-Statements/OUTFILE).
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] hf200012 commented on a diff in pull request #13171: [typo](docs)fix error url

[GitBox]

hf200012 commented on code in PR #13171:
URL: https://github.com/apache/doris/pull/13171#discussion_r990611888


##
docs/en/docs/data-table/basic-usage.md:
##
@@ -142,7 +142,7 @@ mysql> USE example_db;
 Database changed
 ```
 
-Doris supports [composite partition and single 
partition](data-partition.html#composite partition and single partition)  two 
table building methods. The following takes the aggregation model as an example 
to demonstrate how to create two partitioned data tables.
+Doris supports [composite partition and single partition](data-partition)  two 
table building methods. The following takes the aggregation model as an example 
to demonstrate how to create two partitioned data tables.

Review Comment:
   ```suggestion
   Doris supports [composite partition and single partition](./data-partition)  
two table building methods. The following takes the aggregation model as an 
example to demonstrate how to create two partitioned data tables.
   ```



##
docs/en/docs/data-table/basic-usage.md:
##
@@ -406,7 +406,7 @@ MySQL> SELECT SUM(pv) FROM table2 WHERE siteid IN (SELECT 
siteid FROM table1 WHE
 
 ## Table Structure Change
 
-Use the [ALTER TABLE COLUMN](. 
/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-COLUMN.md)
 command to modify the table Schema, including the following changes.
+Use the [ALTER TABLE 
COLUMN](../sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-COLUMN.md)
 command to modify the table Schema, including the following changes.

Review Comment:
   ```suggestion
   Use the [ALTER TABLE 
COLUMN](../sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-COLUMN)
 command to modify the table Schema, including the following changes.
   ```



##
docs/zh-CN/docs/data-operate/import/import-scenes/jdbc-load.md:
##
@@ -160,4 +160,4 @@ public class DorisJDBCDemo {
 
前面提到，我们建议在使用 INSERT 导入数据时，采用 ”批“ 的方式进行导入，而不是单条插入。
 
-   同时，我们可以为每次 INSERT 操作设置一个 Label。通过 [Label 
机制](./load-atomicity.html#label-机制) 可以保证操作的幂等性和原子性，最终做到数据的不丢不重。关于 INSERT 中 
Label 的具体用法，可以参阅 
[INSERT](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/INSERT.md)
 文档。
+   同时，我们可以为每次 INSERT 操作设置一个 Label。通过 [Label 机制](./load-atomicity) 
可以保证操作的幂等性和原子性，最终做到数据的不丢不重。关于 INSERT 中 Label 的具体用法，可以参阅 
[INSERT](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/INSERT.md)
 文档。

Review Comment:
   ```suggestion
  同时，我们可以为每次 INSERT 操作设置一个 Label。通过 [Label 机制](./load-atomicity) 
可以保证操作的幂等性和原子性，最终做到数据的不丢不重。关于 INSERT 中 Label 的具体用法，可以参阅 
[INSERT](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/INSERT)
 文档。
   ```



##
docs/en/docs/data-operate/import/import-scenes/jdbc-load.md:
##
@@ -160,4 +160,4 @@ Please note the following:
 
As mentioned earlier, we recommend that when using INSERT to import data, 
use the "batch" method to import, rather than a single insert.
 
-   At the same time, we can set a Label for each INSERT operation. Through the 
[Label mechanism](./load-atomicity.html#label-mechanism), the idempotency and 
atomicity of operations can be guaranteed, and the data will not be lost or 
heavy in the end. For the specific usage of Label in INSERT, you can refer to 
the 
[INSERT](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/INSERT.md)
 document.
+   At the same time, we can set a Label for each INSERT operation. Through the 
[Label mechanism](./load-atomicity), the idempotency and atomicity of 
operations can be guaranteed, and the data will not be lost or heavy in the 
end. For the specific usage of Label in INSERT, you can refer to the 
[INSERT](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/INSERT.md)
 document.

Review Comment:
   ```suggestion
  At the same time, we can set a Label for each INSERT operation. Through 
the [Label mechanism](./load-atomicity), the idempotency and atomicity of 
operations can be guaranteed, and the data will not be lost or heavy in the 
end. For the specific usage of Label in INSERT, you can refer to the 
[INSERT](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/INSERT)
 document.
   ```



##
docs/en/docs/data-table/basic-usage.md:
##
@@ -470,7 +470,7 @@ For more help, see ``HELP ALTER TABLE``.
 
 Rollup can be understood as a materialized index structure for a Table. 
**Materialized** because its data is physically stored independently, and 
**Indexed** in the sense that Rollup can reorder columns to increase the hit 
rate of prefix indexes, and can reduce key columns to increase the aggregation 
of data.
 
-Use [ALTER TABLE ROLLUP](... 
/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-ROLLUP.md)
 to perform various changes to Rollup.
+Use [ALTER TABLE 
ROLLUP](../sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-ROLLUP.md)
 to perform 

[GitHub] [doris] hf200012 commented on a diff in pull request #13169: [typo](docs) Fix the jump link 404 in basic usage.md

[GitBox]

hf200012 commented on code in PR #13169:
URL: https://github.com/apache/doris/pull/13169#discussion_r990612161


##
docs/en/docs/data-table/basic-usage.md:
##
@@ -107,7 +107,7 @@ CREATE DATABASE example_db;
 >
 > If you don't know the full name of the command, you can use "help command a 
 > field" for fuzzy query. If you type `HELP CREATE`, you can match commands 
 > like `CREATE DATABASE', `CREATE TABLE', `CREATE USER', etc.
 
-After the database is created, you can view the database information through 
[SHOW 
DATABASES](../sql-manual/sql-reference/Show-Statements/SHOW-DATABASES.html#show-databases).
+After the database is created, you can view the database information through 
[SHOW 
DATABASES](../sql-manual/sql-reference/Show-Statements/SHOW-DATABASES.md#show-databases).

Review Comment:
   ```suggestion
   After the database is created, you can view the database information through 
[SHOW DATABASES](../sql-manual/sql-reference/Show-Statements/SHOW-DATABASES).
   ```



##
docs/zh-CN/docs/data-table/basic-usage.md:
##
@@ -130,7 +130,7 @@ CREATE DATABASE example_db;
 >SHOW CREATE ROUTINE LOAD
 > ```
 
-数据库创建完成之后，可以通过 [SHOW 
DATABASES](../sql-manual/sql-reference/Show-Statements/SHOW-DATABASES.html#show-databases)
 查看数据库信息。
+数据库创建完成之后，可以通过 [SHOW 
DATABASES](../sql-manual/sql-reference/Show-Statements/SHOW-DATABASES.md#show-databases)
 查看数据库信息。

Review Comment:
   ```suggestion
   数据库创建完成之后，可以通过 [SHOW 
DATABASES](../sql-manual/sql-reference/Show-Statements/SHOW-DATABASES) 查看数据库信息。
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] MrsZHui opened a new issue, #13174: [Bug] Doris创建物化视图导致be集群全部宕机

[GitBox]

MrsZHui opened a new issue, #13174:
URL: https://github.com/apache/doris/issues/13174

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Version
   
   1.1.1
   
   ### What's Wrong?
   
   当我创建一个物化视图时(对应建表和物化视图可以复现)，be集群全部宕机。
   最后发现是因为指标列是TEXT类型导致。
   
   ### What You Expected?
   
   期望使用不合理的创建物化视图不合理时候，机器可以不宕机，这个问题很严重
   
   ### How to Reproduce?
   
   复现流程 1、建表 2、导入数据 3、创建物化视图，以下是对应的语句
   
CREATE TABLE `test` (
 `dt` date NOT NULL COMMENT "分区日期",
 `new_cid` varchar(100) NULL COMMENT "new_cid",
 `pv` text NULL COMMENT "浏览pv",
 `duration` double NULL COMMENT "",
 `start_cnt` text NULL COMMENT ""
   ) ENGINE=OLAP
   DUPLICATE KEY(`dt`, `new_cid`)
   COMMENT "OLAP"
   PARTITION BY RANGE(`dt`)
   (
   PARTITION p20221006 VALUES [('2022-10-06'), ('2022-10-07')))
   DISTRIBUTED BY HASH(`new_cid`) BUCKETS 8
   PROPERTIES (
   "in_memory" = "false",
   "storage_format" = "V2"
   );
   
   create materialized view test_mv as select dt
   ,new_cid
   ,sum(cast(pv as int))   as pv 
   ,sum(duration)  as duration
   ,sum(cast(start_cnt as int))as start_cnt
   from 
   test
   group by dt,new_cid;
   
   ### Anything Else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] FreeOnePlus closed pull request #13160: [typo](docs)Fix Docs Error Url

[GitBox]

FreeOnePlus closed pull request #13160: [typo](docs)Fix Docs Error Url 
URL: https://github.com/apache/doris/pull/13160


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] FreeOnePlus closed pull request #13159: [typo](docs)Fix Docs Error Url

[GitBox]

FreeOnePlus closed pull request #13159: [typo](docs)Fix Docs Error Url
URL: https://github.com/apache/doris/pull/13159


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] FreeOnePlus closed pull request #13158: [typo](docs)Fix Docs Error Url

[GitBox]

FreeOnePlus closed pull request #13158: [typo](docs)Fix Docs Error Url
URL: https://github.com/apache/doris/pull/13158


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] FreeOnePlus closed pull request #13155: [typo](docs)Fix Doc Error Url

[GitBox]

FreeOnePlus closed pull request #13155: [typo](docs)Fix Doc Error Url
URL: https://github.com/apache/doris/pull/13155


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] FreeOnePlus closed pull request #13154: [typo](docs)Fix Doc Error Url

[GitBox]

FreeOnePlus closed pull request #13154: [typo](docs)Fix Doc Error Url
URL: https://github.com/apache/doris/pull/13154


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] FreeOnePlus closed pull request #13153: [typo](docs)Fix Docs Error Urls

[GitBox]

FreeOnePlus closed pull request #13153: [typo](docs)Fix Docs Error Urls
URL: https://github.com/apache/doris/pull/13153


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] hf200012 commented on a diff in pull request #13156: [typo](docs)Fix the jump link 404 in delete recover.md

[GitBox]

hf200012 commented on code in PR #13156:
URL: https://github.com/apache/doris/pull/13156#discussion_r990613199


##
docs/zh-CN/docs/admin-manual/data-admin/delete-recover.md:
##
@@ -50,4 +50,4 @@ RECOVER PARTITION p1 FROM example_tbl;
 
 ## 更多帮助
 
-关于 RECOVER 使用的更多详细语法及最佳实践，请参阅 
[RECOVER](../../sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/RECOVER.md)
 命令手册，你也可以在 MySql 客户端命令行下输入 `HELP RECOVER` 获取更多帮助信息。
+关于 RECOVER 使用的更多详细语法及最佳实践，请参阅 
[RECOVER](../../sql-manual/sql-reference/Database-Administration-Statements/RECOVER.md)
 命令手册，你也可以在 MySql 客户端命令行下输入 `HELP RECOVER` 获取更多帮助信息。

Review Comment:
   ```suggestion
   关于 RECOVER 使用的更多详细语法及最佳实践，请参阅 
[RECOVER](../../sql-manual/sql-reference/Database-Administration-Statements/RECOVER)
 命令手册，你也可以在 MySql 客户端命令行下输入 `HELP RECOVER` 获取更多帮助信息。
   ```



##
docs/en/docs/admin-manual/data-admin/delete-recover.md:
##
@@ -50,4 +50,4 @@ RECOVER PARTITION p1 FROM example_tbl;
 
 ## More Help
 
-For more detailed syntax and best practices used by RECOVER, please refer to 
the 
[RECOVER](../../sql-manual/sql-reference/Data-Definition-Statements/Backup-and-Restore/RECOVER.md)
 command manual, You can also type `HELP RECOVER` on the MySql client command 
line for more help.
+For more detailed syntax and best practices used by RECOVER, please refer to 
the 
[RECOVER](../../sql-manual/sql-reference/Database-Administration-Statements/RECOVER.md)
 command manual, You can also type `HELP RECOVER` on the MySql client command 
line for more help.

Review Comment:
   ```suggestion
   For more detailed syntax and best practices used by RECOVER, please refer to 
the 
[RECOVER](../../sql-manual/sql-reference/Database-Administration-Statements/RECOVER)
 command manual, You can also type `HELP RECOVER` on the MySql client command 
line for more help.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-website] catpineapple opened a new pull request, #131: doc_fix_404_221008

[GitBox]

catpineapple opened a new pull request, #131:
URL: https://github.com/apache/doris-website/pull/131

   fix doc 404 page


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #13157: [typo](docs)fix docs 404 url

[GitBox]

github-actions[bot] commented on PR #13157:
URL: https://github.com/apache/doris/pull/13157#issuecomment-1272269626

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #13157: [typo](docs)fix docs 404 url

[GitBox]

github-actions[bot] commented on PR #13157:
URL: https://github.com/apache/doris/pull/13157#issuecomment-1272269640

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] catpineapple opened a new pull request, #13175: [typo](docs)Fix Docs 404 Url

[GitBox]

catpineapple opened a new pull request, #13175:
URL: https://github.com/apache/doris/pull/13175

   base URL :
   
1:https://doris.apache.org/docs/dev/sql-manual/sql-reference/Data-Manipulation-Statements/Load/BROKER-LOAD
   
2:https://doris.apache.org/docs/dev/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-ROLLUP
   
3:https://doris.apache.org/docs/dev/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-PARTITION
   4:https://doris.apache.org/docs/dev/install/install-deploy
   5:https://doris.apache.org/docs/dev/get-starting/
   
   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] weizhengte commented on pull request #13136: [Enhancement](statistics) optimize the default configuration related to statistics, etc.

[GitBox]

weizhengte commented on PR #13136:
URL: https://github.com/apache/doris/pull/13136#issuecomment-1272270155

   @morrySnow review plz


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] FreeOnePlus opened a new pull request, #13176: [typo](docs)Fix Docs Error Urls

[GitBox]

FreeOnePlus opened a new pull request, #13176:
URL: https://github.com/apache/doris/pull/13176

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   BaseUrl-01:
   
https://doris.apache.org/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-ROLLUP
   
   ErrorUrl-01:
   
https://doris.apache.org/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.html
   
   FixUrl-01:
   
https://doris.apache.org/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE
   
   BaseUrl-02:
   
https://doris.apache.org/zh-CN/docs/dev/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-REPLACE
   
   ErrorUrl-02:
   
https://doris.apache.org/zh-CN/docs/dev/sql-manual/sql-reference/Data-Definition-Statements/partition/table-tmp-partition.md
   
   FixUrl-02:
   
https://doris.apache.org/zh-CN/docs/dev/advanced/partition/table-temp-partition
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-website] hf200012 merged pull request #122: fix docs 404 url

[GitBox]

hf200012 merged PR #122:
URL: https://github.com/apache/doris-website/pull/122


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris-website] branch master updated: fix docs 404 url (#122)

[jiafengzheng]

This is an automated email from the ASF dual-hosted git repository.

jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git


The following commit(s) were added to refs/heads/master by this push:
 new 96324cdb5ab fix docs 404 url (#122)
96324cdb5ab is described below

commit 96324cdb5abc3dbc13cc2d83a0c66d0c48681a00
Author: zy-kkk 
AuthorDate: Sat Oct 8 17:12:11 2022 +0800

fix docs 404 url (#122)

Update SHOW-ALTER-TABLE-MATERIALIZED-VIEW.md
---
 docs/ecosystem/logstash.md |   4 +-
 docs/ecosystem/udf/contribute-udf.md   |  10 +-
 docs/faq/install-faq.md|   1 -
 .../Load/CREATE-SYNC-JOB.md|   2 +-
 .../SHOW-ALTER-TABLE-MATERIALIZED-VIEW.md  | 120 +
 .../sql-reference/Show-Statements/SHOW-STATUS.md   |  86 +--
 .../current/ecosystem/logstash.md  |   4 +-
 .../Alter/ALTER-TABLE-ROLLUP.md|   2 +-
 .../Load/CREATE-SYNC-JOB.md|   2 +-
 9 files changed, 136 insertions(+), 95 deletions(-)

diff --git a/docs/ecosystem/logstash.md b/docs/ecosystem/logstash.md
index dbf7901625e..c6ab151d1f4 100644
--- a/docs/ecosystem/logstash.md
+++ b/docs/ecosystem/logstash.md
@@ -28,9 +28,9 @@ under the License.
 
 This plugin is used to output data to Doris for logstash, use the HTTP 
protocol to interact with the Doris FE Http interface, and import data through 
Doris's stream load.
 
-[Learn more about Doris Stream Load 
](../../data-operate/import/import-way/stream-load-manual)
+[Learn more about Doris Stream Load 
](../data-operate/import/import-way/stream-load-manual)
 
-[Learn more about Doris](../)
+[Learn more about Doris](/)
 
 
 ## Install and compile
diff --git a/docs/ecosystem/udf/contribute-udf.md 
b/docs/ecosystem/udf/contribute-udf.md
index 1db7a8b9642..0492c7e0139 100644
--- a/docs/ecosystem/udf/contribute-udf.md
+++ b/docs/ecosystem/udf/contribute-udf.md
@@ -14,7 +14,7 @@ to you under the Apache License, Version 2.0 (the
 "License"); you may not use this file except in compliance
 with the License. You may obtain a copy of the License at
 
-  http://www.apache.org/licenses/LICENSE-2.0
+  http://www.apache.org/licenses/LICENSE-2.0
 
 Unless required by applicable law or agreed to in writing,
 software distributed under the License is distributed on an
@@ -73,8 +73,8 @@ The user manual needs to include: UDF function definition 
description, applicabl
 
 ```
 ├── docs
-│   └── zh-CN
-│   └──extending-doris
+│ └── zh-CN
+│ └──extending-doris
 │  └──udf
 │└──contrib
 │  ├── udf-simple-manual.md
@@ -83,8 +83,8 @@ The user manual needs to include: UDF function definition 
description, applicabl
 
 ```
 ├── docs
-│   └── en
-│   └──extending-doris
+│ └── en
+│ └──extending-doris
 │  └──udf
 │└──contrib
 │  ├── udf-simple-manual.md
diff --git a/docs/faq/install-faq.md b/docs/faq/install-faq.md
index d877c6adc0e..e945ceb6c79 100644
--- a/docs/faq/install-faq.md
+++ b/docs/faq/install-faq.md
@@ -155,7 +155,6 @@ In many cases, we need to troubleshoot problems through 
logs. The format and vie
 
   Logs starting with F are Fatal logs. For example, F0916 , indicating the 
Fatal log on September 16th. Fatal logs usually indicate a program assertion 
error, and an assertion error will directly cause the process to exit 
(indicating a bug in the program). Welcome to the WeChat group, github 
discussion or dev mail group for help.
 
-
 2. FE
 
FE is a java process, and the robustness is due to the C/C++ program. 
Usually the reason for FE to hang up may be OOM (Out-of-Memory) or metadata 
write failure. These errors usually have an error stack in fe.log or fe.out. 
Further investigation is required based on the error stack information.
diff --git 
a/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/CREATE-SYNC-JOB.md
 
b/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/CREATE-SYNC-JOB.md
index 09302c36e4e..99b592c7619 100644
--- 
a/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/CREATE-SYNC-JOB.md
+++ 
b/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/CREATE-SYNC-JOB.md
@@ -36,7 +36,7 @@ The data synchronization (Sync Job) function supports users 
to submit a resident
 
 Currently, the data synchronization job only supports connecting to Canal, 
obtaining the parsed Binlog data from the Canal Server and importing it into 
Doris.
 
-Users can view the data synchronization job status through [SHOW SYNC 
JOB](../../Show-Statements/SHOW-SYNC-JOB).
+Users can view the data synchronization job status through [SHOW SYNC 
JOB](../../../../sql-manual/sql-reference/Show-Statements/SHOW-SYNC-JOB).
 
 grammar:
 
diff --git 
a/docs/sql-

[GitHub] [doris-website] SeekingYang opened a new pull request, #132: 404 repair

[GitBox]

SeekingYang opened a new pull request, #132:
URL: https://github.com/apache/doris-website/pull/132

   127-130


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-flink-connector] dinggege1024 opened a new issue, #70: [Enhancement] ADD RowSerializer for doris flink connector

[GitBox]

dinggege1024 opened a new issue, #70:
URL: https://github.com/apache/doris-flink-connector/issues/70

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### Description
   
   It's better to add RowSerializer which is more fitable for flink datastream
   
   ### Solution
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-flink-connector] dinggege1024 opened a new pull request, #71: [Enhancement] ADD RowSerializer for doris flink connector

[GitBox]

dinggege1024 opened a new pull request, #71:
URL: https://github.com/apache/doris-flink-connector/pull/71

   # Proposed changes
   
   Issue Number: #70 
   
   ## Problem Summary:
   
   ADD RowSerializer AND ut
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (No)
   2. Has unit tests been added: (Yes)
   3. Has document been added or modified: (No,but will)
   4. Does it need to update dependencies: (No)
   5. Are there any changes that cannot be rolled back: (No)
   
   ## Further comments
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-flink-connector] dinggege1024 commented on pull request #71: [Enhancement] ADD RowSerializer for doris flink connector

[GitBox]

dinggege1024 commented on PR #71:
URL: 
https://github.com/apache/doris-flink-connector/pull/71#issuecomment-1272276351

   @JNSimba  Hi Simba, can you review this , thx~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] catpineapple opened a new pull request, #13177: [typo](docs)Fix Docs Install Deploy

[GitBox]

catpineapple opened a new pull request, #13177:
URL: https://github.com/apache/doris/pull/13177

   
   base URL :
   https://doris.apache.org/docs/dev/install/install-deploy
   
   fix:  
   1:error url:https://doris.apache.org/docs/dev/advanced/variables.html
   2:delete doc 'Elastic scaling' , These documents do not belong here.
   
   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] hf200012 opened a new pull request, #13178: [typo](docs)SQL Server 2017 version ODBC usage instructions

[GitBox]

hf200012 opened a new pull request, #13178:
URL: https://github.com/apache/doris/pull/13178

   sql server 2017 version ODBC usage instructions
   
   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] dataalive commented on issue #13045: [Bug] 测试部署机器有一台一直报错 No memory available to store statement

[GitBox]

dataalive commented on issue #13045:
URL: https://github.com/apache/doris/issues/13045#issuecomment-1272280342

   有么有更完整的栈？或者复现的步骤，这个看着是odbc里报的。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #13178: [typo](docs)SQL Server 2017 version ODBC usage instructions

[GitBox]

github-actions[bot] commented on PR #13178:
URL: https://github.com/apache/doris/pull/13178#issuecomment-1272280442

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #13178: [typo](docs)SQL Server 2017 version ODBC usage instructions

[GitBox]

github-actions[bot] commented on PR #13178:
URL: https://github.com/apache/doris/pull/13178#issuecomment-1272280445

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #13165: [Bug](libjvm) reorder initialization of JNI

[GitBox]

github-actions[bot] commented on PR #13165:
URL: https://github.com/apache/doris/pull/13165#issuecomment-1272280952

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #13165: [Bug](libjvm) reorder initialization of JNI

[GitBox]

github-actions[bot] commented on PR #13165:
URL: https://github.com/apache/doris/pull/13165#issuecomment-1272280942

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #13176: [typo](docs)Fix Docs Error Urls

[GitBox]

github-actions[bot] commented on PR #13176:
URL: https://github.com/apache/doris/pull/13176#issuecomment-1272281617

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #13176: [typo](docs)Fix Docs Error Urls

[GitBox]

github-actions[bot] commented on PR #13176:
URL: https://github.com/apache/doris/pull/13176#issuecomment-1272281610

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #13175: [typo](docs)Fix Docs 404 Url

[GitBox]

github-actions[bot] commented on PR #13175:
URL: https://github.com/apache/doris/pull/13175#issuecomment-1272282169

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #13175: [typo](docs)Fix Docs 404 Url

[GitBox]

github-actions[bot] commented on PR #13175:
URL: https://github.com/apache/doris/pull/13175#issuecomment-1272282179

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #13170: [typo](docs)Fix jump link 404 in jdbc load.md

[GitBox]

github-actions[bot] commented on PR #13170:
URL: https://github.com/apache/doris/pull/13170#issuecomment-1272282689

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #13170: [typo](docs)Fix jump link 404 in jdbc load.md

[GitBox]

github-actions[bot] commented on PR #13170:
URL: https://github.com/apache/doris/pull/13170#issuecomment-1272282695

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morningman merged pull request #13072: [feature-wip](parquet-reader) prepare for BE benchmark

[GitBox]

morningman merged PR #13072:
URL: https://github.com/apache/doris/pull/13072


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [fix](parquet-reader) skip data/datatime column predicate filter to avoid coredump (#13072)

[morningman]

This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 5214e898d9 [fix](parquet-reader) skip data/datatime column predicate 
filter to avoid coredump (#13072)
5214e898d9 is described below

commit 5214e898d9655d10d3fc75c9e1f008c1246ab89f
Author: slothever <18522955+w...@users.noreply.github.com>
AuthorDate: Sat Oct 8 18:02:35 2022 +0800

[fix](parquet-reader) skip data/datatime column predicate filter to avoid 
coredump (#13072)

Will be fixed later
Co-authored-by: jinzhe 
---
 be/src/vec/exec/format/parquet/parquet_pred_cmp.h  | 50 --
 .../exec/format/parquet/vparquet_column_reader.cpp |  1 +
 2 files changed, 1 insertion(+), 50 deletions(-)

diff --git a/be/src/vec/exec/format/parquet/parquet_pred_cmp.h 
b/be/src/vec/exec/format/parquet/parquet_pred_cmp.h
index 1b5d78bb12..517bab5d61 100644
--- a/be/src/vec/exec/format/parquet/parquet_pred_cmp.h
+++ b/be/src/vec/exec/format/parquet/parquet_pred_cmp.h
@@ -145,16 +145,6 @@ static bool _eval_eq(PrimitiveType conjunct_type, void* 
value, const char* min_b
 _FILTER_GROUP_BY_EQ_PRED(conjunct_value, min, max)
 break;
 }
-case TYPE_DOUBLE: {
-_PLAIN_DECODE(double, value, min_bytes, max_bytes, conjunct_value, 
min, max)
-_FILTER_GROUP_BY_EQ_PRED(conjunct_value, min, max)
-break;
-}
-case TYPE_FLOAT: {
-_PLAIN_DECODE(float, value, min_bytes, max_bytes, conjunct_value, min, 
max)
-_FILTER_GROUP_BY_EQ_PRED(conjunct_value, min, max)
-break;
-}
 case TYPE_STRING:
 case TYPE_VARCHAR:
 case TYPE_CHAR:
@@ -194,16 +184,6 @@ static bool _eval_gt(PrimitiveType conjunct_type, void* 
value, const char* max_b
 _FILTER_GROUP_BY_GT_PRED(conjunct_value, max)
 break;
 }
-case TYPE_DOUBLE: {
-_PLAIN_DECODE_SINGLE(double, value, max_bytes, conjunct_value, max)
-_FILTER_GROUP_BY_GT_PRED(conjunct_value, max)
-break;
-}
-case TYPE_FLOAT: {
-_PLAIN_DECODE_SINGLE(float, value, max_bytes, conjunct_value, max)
-_FILTER_GROUP_BY_GT_PRED(conjunct_value, max)
-break;
-}
 case TYPE_STRING:
 case TYPE_VARCHAR:
 case TYPE_CHAR:
@@ -244,16 +224,6 @@ static bool _eval_ge(PrimitiveType conjunct_type, void* 
value, const char* max_b
 _FILTER_GROUP_BY_GE_PRED(conjunct_value, max)
 break;
 }
-case TYPE_DOUBLE: {
-_PLAIN_DECODE_SINGLE(double, value, max_bytes, conjunct_value, max)
-_FILTER_GROUP_BY_GE_PRED(conjunct_value, max)
-break;
-}
-case TYPE_FLOAT: {
-_PLAIN_DECODE_SINGLE(float, value, max_bytes, conjunct_value, max)
-_FILTER_GROUP_BY_GE_PRED(conjunct_value, max)
-break;
-}
 case TYPE_STRING:
 case TYPE_VARCHAR:
 case TYPE_CHAR:
@@ -294,16 +264,6 @@ static bool _eval_lt(PrimitiveType conjunct_type, void* 
value, const char* min_b
 _FILTER_GROUP_BY_LT_PRED(conjunct_value, min)
 break;
 }
-case TYPE_DOUBLE: {
-_PLAIN_DECODE_SINGLE(double, value, min_bytes, conjunct_value, min)
-_FILTER_GROUP_BY_LT_PRED(conjunct_value, min)
-break;
-}
-case TYPE_FLOAT: {
-_PLAIN_DECODE_SINGLE(float, value, min_bytes, conjunct_value, min)
-_FILTER_GROUP_BY_LT_PRED(conjunct_value, min)
-break;
-}
 case TYPE_STRING:
 case TYPE_VARCHAR:
 case TYPE_CHAR:
@@ -344,16 +304,6 @@ static bool _eval_le(PrimitiveType conjunct_type, void* 
value, const char* min_b
 _FILTER_GROUP_BY_LE_PRED(conjunct_value, min)
 break;
 }
-case TYPE_DOUBLE: {
-_PLAIN_DECODE_SINGLE(double, value, min_bytes, conjunct_value, min)
-_FILTER_GROUP_BY_LE_PRED(conjunct_value, min)
-break;
-}
-case TYPE_FLOAT: {
-_PLAIN_DECODE_SINGLE(float, value, min_bytes, conjunct_value, min)
-_FILTER_GROUP_BY_LE_PRED(conjunct_value, min)
-break;
-}
 case TYPE_STRING:
 case TYPE_VARCHAR:
 case TYPE_CHAR:
diff --git a/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp 
b/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp
index 1bcb640865..5338c169f7 100644
--- a/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp
+++ b/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp
@@ -65,6 +65,7 @@ void ParquetColumnReader::_generate_read_ranges(int64_t 
start_index, int64_t end
 std::list& 
read_ranges) {
 if (_row_ranges.size() == 0) {
 read_ranges.emplace_back(start_index, end_index);
+return;
 }
 int index = _row_range_index;
 while (index < _row_ranges.size()) {


-
To unsubscribe, e-mail: commits-unsubs

[GitHub] [doris] github-actions[bot] commented on pull request #13072: [feature-wip](parquet-reader) prepare for BE benchmark

[GitBox]

github-actions[bot] commented on PR #13072:
URL: https://github.com/apache/doris/pull/13072#issuecomment-1272282742

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #13072: [feature-wip](parquet-reader) prepare for BE benchmark

[GitBox]

github-actions[bot] commented on PR #13072:
URL: https://github.com/apache/doris/pull/13072#issuecomment-1272282752

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] morningman merged pull request #13122: [feature-wip](parquet-reader) optimize the performance of column conversion

[GitBox]

morningman merged PR #13122:
URL: https://github.com/apache/doris/pull/13122


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [feature-wip](parquet-reader) optimize the performance of column conversion (#13122)

[morningman]

This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new b81a8789c3 [feature-wip](parquet-reader) optimize the performance of 
column conversion (#13122)
b81a8789c3 is described below

commit b81a8789c3f6d47df6fee85c7f0fd470f2d1ca16
Author: Ashin Gau 
AuthorDate: Sat Oct 8 18:03:10 2022 +0800

[feature-wip](parquet-reader) optimize the performance of column conversion 
(#13122)

Convert Parquet column into doris column via batch method.
In the previous implementation, only numeric types can be converted in 
batches,
and other types can only be inserted one by one.
This process will generate repeated virtual function calls and container 
expansion.
---
 be/src/vec/exec/format/parquet/parquet_common.cpp  | 44 
 be/src/vec/exec/format/parquet/parquet_common.h| 61 +++---
 be/src/vec/exec/format/parquet/parquet_pred_cmp.h  | 28 +++---
 .../parquet/vparquet_column_chunk_reader.cpp   |  4 +-
 .../exec/format/parquet/vparquet_column_reader.cpp | 30 ---
 5 files changed, 93 insertions(+), 74 deletions(-)

diff --git a/be/src/vec/exec/format/parquet/parquet_common.cpp 
b/be/src/vec/exec/format/parquet/parquet_common.cpp
index fd8d5d7428..83c702f800 100644
--- a/be/src/vec/exec/format/parquet/parquet_common.cpp
+++ b/be/src/vec/exec/format/parquet/parquet_common.cpp
@@ -40,6 +40,12 @@ inline uint64_t ParquetInt96::to_timestamp_micros() const {
 M(TypeIndex::Float32, Float32)   \
 M(TypeIndex::Float64, Float64)
 
+#define FOR_SHORT_INT_TYPES(M) \
+M(TypeIndex::Int8, Int8)   \
+M(TypeIndex::UInt8, UInt8) \
+M(TypeIndex::Int16, Int16) \
+M(TypeIndex::UInt16, UInt16)
+
 Status Decoder::get_decoder(tparquet::Type::type type, 
tparquet::Encoding::type encoding,
 std::unique_ptr& decoder) {
 switch (encoding) {
@@ -155,19 +161,6 @@ Status FixLengthDecoder::skip_values(size_t num_values) {
 return Status::OK();
 }
 
-Status FixLengthDecoder::_decode_short_int(MutableColumnPtr& doris_column, 
size_t num_values,
-   size_t real_length) {
-if (UNLIKELY(_physical_type != tparquet::Type::INT32)) {
-return Status::InternalError("Short int can only be decoded from 
INT32");
-}
-for (int i = 0; i < num_values; ++i) {
-char* buf_start = _FIXED_GET_DATA_OFFSET(i);
-doris_column->insert_data(buf_start, real_length);
-_FIXED_SHIFT_DATA_OFFSET();
-}
-return Status::OK();
-}
-
 Status FixLengthDecoder::decode_values(MutableColumnPtr& doris_column, 
DataTypePtr& data_type,
size_t num_values) {
 if (_has_dict) {
@@ -178,12 +171,11 @@ Status FixLengthDecoder::decode_values(MutableColumnPtr& 
doris_column, DataTypeP
 }
 TypeIndex logical_type = remove_nullable(data_type)->get_type_id();
 switch (logical_type) {
-case TypeIndex::Int8:
-case TypeIndex::UInt8:
-return _decode_short_int(doris_column, num_values, 1);
-case TypeIndex::Int16:
-case TypeIndex::UInt16:
-return _decode_short_int(doris_column, num_values, 2);
+#define DISPATCH(SHORT_INT_TYPE, CPP_SHORT_INT_TYPE) \
+case SHORT_INT_TYPE: \
+return _decode_short_int(doris_column, num_values);
+FOR_SHORT_INT_TYPES(DISPATCH)
+#undef DISPATCH
 #define DISPATCH(NUMERIC_TYPE, CPP_NUMERIC_TYPE) \
 case NUMERIC_TYPE:   \
 return _decode_numeric(doris_column, num_values);
@@ -329,13 +321,15 @@ Status ByteArrayDecoder::decode_values(MutableColumnPtr& 
doris_column, DataTypeP
 TypeIndex logical_type = remove_nullable(data_type)->get_type_id();
 switch (logical_type) {
 case TypeIndex::String:
-case TypeIndex::FixedString:
+case TypeIndex::FixedString: {
+std::vector string_values;
+string_values.reserve(num_values);
 for (int i = 0; i < num_values; ++i) {
 if (_has_dict) {
 uint32_t idx = _indexes[i];
 uint32_t idx_cursor = _dict_offsets[idx];
 char* buff_start = reinterpret_cast(_dict.get() + 
idx_cursor);
-doris_column->insert_data(buff_start, _dict_offsets[idx + 1] - 
idx_cursor - 4);
+string_values.emplace_back(buff_start, _dict_offsets[idx + 1] 
- idx_cursor - 4);
 } else {
 if (UNLIKELY(_offset + 4 > _data->size)) {
 return Status::IOError("Can't read byte array length from 
plain decoder");
@@ -346,11 +340,13 @@ Status ByteArrayDecoder::decode_values(MutableColumnPtr& 
doris_column, DataTypeP
 if (UNLIKELY(_offset + length) > _data->size) {
 return Status::IOError("Can't read enough bytes in

[GitHub] [doris] pengxiangyu commented on a diff in pull request #12897: [feature](remote) support local cache GC by disk usage

[GitBox]

pengxiangyu commented on code in PR #12897:
URL: https://github.com/apache/doris/pull/12897#discussion_r990580003


##
be/src/io/cache/file_cache_manager.cpp:
##
@@ -18,15 +18,39 @@
 #include "io/cache/file_cache_manager.h"
 
 #include "gutil/strings/util.h"
+#include "io/cache/dummy_file_cache.h"
 #include "io/cache/sub_file_cache.h"
 #include "io/cache/whole_file_cache.h"
 #include "io/fs/local_file_system.h"
+#include "olap/storage_engine.h"
 #include "util/file_utils.h"
 #include "util/string_util.h"
 
 namespace doris {
 namespace io {
 
+void GCContextPerDisk::init(const std::string& path, int64_t max_size) {
+_disk_path = path;
+_conf_max_size = max_size;
+_used_size = 0;
+}

Review Comment:
   Need an empty line after a function



##
be/src/io/cache/dummy_file_cache.cpp:
##
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "io/cache/dummy_file_cache.h"
+
+#include "gutil/strings/util.h"
+#include "io/fs/local_file_system.h"
+#include "util/file_utils.h"
+#include "util/string_util.h"
+
+namespace doris {
+namespace io {
+
+const static std::string WHOLE_FILE_CACHE_NAME = "WHOLE_FILE_CACHE";

Review Comment:
   WHOLE_FILE_CACHE_NAME is not used.



##
be/src/io/cache/file_cache_manager.cpp:
##
@@ -56,88 +80,82 @@ void FileCacheManager::remove_file_cache(const std::string& 
cache_path) {
 }
 }
 
-void FileCacheManager::clean_timeout_caches() {
-std::shared_lock rdlock(_cache_map_lock);
-for (std::map::const_iterator iter = 
_file_cache_map.cbegin();
- iter != _file_cache_map.cend(); ++iter) {
-if (iter->second == nullptr) {
-continue;
+void 
FileCacheManager::_add_file_cache_for_gc_by_disk(std::vector& 
contexts,
+  FileCachePtr file_cache) 
{
+// sort file cache by last match time
+if (config::file_cache_max_size_per_disk > 0) {
+auto file_size = file_cache->cache_file_size();
+if (file_size <= 0) {
+return;
+}
+for (size_t i = 0; i < contexts.size(); ++i) {
+if (contexts[i].try_add_file_cache(file_cache, file_size)) {
+break;
+}
 }
-iter->second->clean_timeout_cache();
 }
 }
+void FileCacheManager::gc_file_caches() {
+int64_t gc_conf_size = config::file_cache_max_size_per_disk;
+std::vector contexts;
+// init for GC by disk size
+if (gc_conf_size > 0) {
+std::vector data_dirs = 
doris::StorageEngine::instance()->get_stores();
+contexts.resize(data_dirs.size());
+for (size_t i = 0; i < contexts.size(); ++i) {
+contexts[i].init(data_dirs[i]->path(), gc_conf_size);
+}
+}
 
-void FileCacheManager::clean_timeout_file_not_in_mem(const std::string& 
cache_path) {
-time_t now = time(nullptr);
+// process unused file caches
 std::shared_lock rdlock(_cache_map_lock);
-// Deal with caches not in _file_cache_map
-if (_file_cache_map.find(cache_path) == _file_cache_map.end()) {
-std::vector cache_file_names;
-if (io::global_local_filesystem()->list(cache_path, 
&cache_file_names).ok()) {
-std::map cache_names;
-std::list done_names;
-for (Path cache_file_name : cache_file_names) {
-std::string filename = cache_file_name.native();
-if (!ends_with(filename, CACHE_DONE_FILE_SUFFIX)) {
-cache_names[filename] = true;
-continue;
-}
-done_names.push_back(filename);
-std::stringstream done_file_ss;
-done_file_ss << cache_path << "/" << filename;
-std::string done_file_path = done_file_ss.str();
-time_t m_time;
-if (!FileUtils::mtime(done_file_path, &m_time).ok()) {
-continue;
-}
-if (now - m_time < config::file_cache_alive_time_sec) {
-continue;
-}
-std::string cache_file_path =
-StringReplace(done_file_path, CACHE_DONE_FILE_SUFFIX, 
""

[GitHub] [doris] luozenglin opened a new pull request, #13179: [fix](docs) fix trim function docs error

[GitBox]

luozenglin opened a new pull request, #13179:
URL: https://github.com/apache/doris/pull/13179

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] siriume commented on issue #13174: [Bug] Doris创建物化视图导致be集群全部宕机

[GitBox]

siriume commented on issue #13174:
URL: https://github.com/apache/doris/issues/13174#issuecomment-1272286306

   [Materialized 
view](https://doris.apache.org/zh-CN/docs/advanced/materialized-view/) don't 
support `cast` function. Please check the document above.
   Use Materialized view like this.
   ```sql
   create materialized view test_mv as select dt
   ,new_cid
   ,sum(pv) as pv
   ,sum(duration) as duration
   ,sum(start_cnt) as start_cnt
   from
   test
   group by dt,new_cid;
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #13179: [fix](docs) fix trim function docs error

[GitBox]

github-actions[bot] commented on PR #13179:
URL: https://github.com/apache/doris/pull/13179#issuecomment-1272289510

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #13179: [fix](docs) fix trim function docs error

[GitBox]

github-actions[bot] commented on PR #13179:
URL: https://github.com/apache/doris/pull/13179#issuecomment-1272289518

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] smallhibiscus opened a new pull request, #13180: [typo](docs)fix the bad link of docs

[GitBox]

smallhibiscus opened a new pull request, #13180:
URL: https://github.com/apache/doris/pull/13180

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   fix the bad link of docs.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] MrsZHui commented on issue #13174: [Bug] Doris创建物化视图导致be集群全部宕机

[GitBox]

MrsZHui commented on issue #13174:
URL: https://github.com/apache/doris/issues/13174#issuecomment-1272290996

   明白了，但是不能说我使用cast function来创建物化视图导致进程崩溃，这个问题本身就不合理。不能保证用户肯定会按照规范来使用
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   At 2022-10-08 18:18:51, "siriume" ***@***.***> wrote:
   
   Materialized view don't support cast function. Please check the document 
above.
   Use Materialized view like this.
   
   create materialized view test_mv asselect dt
   ,new_cid
   ,sum(pv) as pv
   ,sum(duration) as duration
   ,sum(start_cnt) as start_cnt
   from
   test
   group by dt,new_cid;
   
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you authored the thread.Message ID: 
***@***.***>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] caoliang-web opened a new pull request, #13181: [typo](docs)fix community module error url

[GitBox]

caoliang-web opened a new pull request, #13181:
URL: https://github.com/apache/doris/pull/13181

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   fix community module error url
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris-website] caoliang-web opened a new pull request, #134: [typo](docs)fix error url in version 0.15

[GitBox]

caoliang-web opened a new pull request, #134:
URL: https://github.com/apache/doris-website/pull/134

   fix error url in version 0.15


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] BiteTheDDDDt commented on issue #8721: [Enhancement] Remove some unused include

[GitBox]

BiteThet commented on issue #8721:
URL: https://github.com/apache/doris/issues/8721#issuecomment-1272291634

   There are some method more specifictly (just a advice):
   1. Firstly you should build doris 
https://doris.apache.org/zh-CN/docs/dev/install/source-install/compilation-with-ldb-toolchain
   2. config clangd well 
https://doris.apache.org/community/developer-guide/cpp-diagnostic-code/
   3. find some unused include and remove it.
   https://user-images.githubusercontent.com/7939630/194703705-77eb84a9-99bb-44c8-96cf-fd1e8a05fb72.png";>
   4. start a pull request and waiting for review 
https://doris.apache.org/community/how-to-contribute/pull-request/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #13180: [typo](docs)fix the bad link of docs

[GitBox]

github-actions[bot] commented on PR #13180:
URL: https://github.com/apache/doris/pull/13180#issuecomment-1272291673

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #13180: [typo](docs)fix the bad link of docs

[GitBox]

github-actions[bot] commented on PR #13180:
URL: https://github.com/apache/doris/pull/13180#issuecomment-1272291681

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] Gabriel39 closed issue #13164: [Bug] BE crash with when JVM exists

[GitBox]

Gabriel39 closed issue #13164: [Bug] BE crash with when JVM exists
URL: https://github.com/apache/doris/issues/13164


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated (b81a8789c3 -> c5f802b93c)

[gabriellee]

This is an automated email from the ASF dual-hosted git repository.

gabriellee pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


from b81a8789c3 [feature-wip](parquet-reader) optimize the performance of 
column conversion (#13122)
 add c5f802b93c [Bug](libjvm) reorder initialization of JNI (#13165)

No new revisions were added by this update.

Summary of changes:
 be/src/service/doris_main.cpp | 23 ---
 1 file changed, 12 insertions(+), 11 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] Gabriel39 merged pull request #13165: [Bug](libjvm) reorder initialization of JNI

[GitBox]

Gabriel39 merged PR #13165:
URL: https://github.com/apache/doris/pull/13165


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] JNSimba commented on a diff in pull request #13181: [typo](docs)fix community module error url

[GitBox]

JNSimba commented on code in PR #13181:
URL: https://github.com/apache/doris/pull/13181#discussion_r990626603


##
docs/en/docs/data-table/basic-usage.md:
##
@@ -142,7 +142,7 @@ mysql> USE example_db;
 Database changed
 ```
 
-Doris supports [composite partition and single 
partition](data-partition.html#composite partition and single partition)  two 
table building methods. The following takes the aggregation model as an example 
to demonstrate how to create two partitioned data tables.
+Doris supports [composite partition and single partition](data-partition)  two 
table building methods. The following takes the aggregation model as an example 
to demonstrate how to create two partitioned data tables.

Review Comment:
   may be `./data-partition` ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] caoliang-web closed pull request #13181: [typo](docs)fix community module error url

[GitBox]

caoliang-web closed pull request #13181: [typo](docs)fix community module error 
url
URL: https://github.com/apache/doris/pull/13181


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #13019: [Improvement](outfile) Support ORC format in outfile

[GitBox]

github-actions[bot] commented on PR #13019:
URL: https://github.com/apache/doris/pull/13019#issuecomment-1272293553

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #13019: [Improvement](outfile) Support ORC format in outfile

[GitBox]

github-actions[bot] commented on PR #13019:
URL: https://github.com/apache/doris/pull/13019#issuecomment-1272293555

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] caoliang-web opened a new pull request, #13182: [typo](docs)fix community module error url

[GitBox]

caoliang-web opened a new pull request, #13182:
URL: https://github.com/apache/doris/pull/13182

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   fix community module error url
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] jacktengg opened a new pull request, #13183: [schema change](fix) fix coredump of schema change

[GitBox]

jacktengg opened a new pull request, #13183:
URL: https://github.com/apache/doris/pull/13183

   When schema change and compaction is executing simutaneously, both nullable 
and not nullable data can be read for the same column, need to reset _nullmap 
for each Block when converting Block data, or else Column case will be wrong.
   
   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] cambyzju commented on a diff in pull request #13088: [refractor](vectorized) refractor some insert_xxx functions

[GitBox]

cambyzju commented on code in PR #13088:
URL: https://github.com/apache/doris/pull/13088#discussion_r990630411


##
be/src/vec/columns/predicate_column.h:
##
@@ -134,17 +134,15 @@ class PredicateColumnType final : public 
COWHelper

[GitHub] [doris] wsjz opened a new pull request, #13184: [feature-wip](parquet-reader) fix string test

[GitBox]

wsjz opened a new pull request, #13184:
URL: https://github.com/apache/doris/pull/13184

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [ ] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [ ] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [ ] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] cambyzju commented on a diff in pull request #12897: [feature](remote) support local cache GC by disk usage

[GitBox]

cambyzju commented on code in PR #12897:
URL: https://github.com/apache/doris/pull/12897#discussion_r990632913


##
be/src/io/cache/dummy_file_cache.cpp:
##
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "io/cache/dummy_file_cache.h"
+
+#include "gutil/strings/util.h"
+#include "io/fs/local_file_system.h"
+#include "util/file_utils.h"
+#include "util/string_util.h"
+
+namespace doris {
+namespace io {
+
+const static std::string WHOLE_FILE_CACHE_NAME = "WHOLE_FILE_CACHE";

Review Comment:
   done



##
be/src/io/cache/file_cache_manager.cpp:
##
@@ -18,15 +18,39 @@
 #include "io/cache/file_cache_manager.h"
 
 #include "gutil/strings/util.h"
+#include "io/cache/dummy_file_cache.h"
 #include "io/cache/sub_file_cache.h"
 #include "io/cache/whole_file_cache.h"
 #include "io/fs/local_file_system.h"
+#include "olap/storage_engine.h"
 #include "util/file_utils.h"
 #include "util/string_util.h"
 
 namespace doris {
 namespace io {
 
+void GCContextPerDisk::init(const std::string& path, int64_t max_size) {
+_disk_path = path;
+_conf_max_size = max_size;
+_used_size = 0;
+}

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #13173: [typo](docs)Fix jump link 404 in external storage load.md

[GitBox]

github-actions[bot] commented on PR #13173:
URL: https://github.com/apache/doris/pull/13173#issuecomment-1272302669

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] github-actions[bot] commented on pull request #13173: [typo](docs)Fix jump link 404 in external storage load.md

[GitBox]

github-actions[bot] commented on PR #13173:
URL: https://github.com/apache/doris/pull/13173#issuecomment-1272302677

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] dataroaring merged pull request #13173: [typo](docs)Fix jump link 404 in external storage load.md

[GitBox]

dataroaring merged PR #13173:
URL: https://github.com/apache/doris/pull/13173


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated (c5f802b93c -> 6b0410450b)

[dataroaring]

This is an automated email from the ASF dual-hosted git repository.

dataroaring pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


from c5f802b93c [Bug](libjvm) reorder initialization of JNI (#13165)
 add 6b0410450b [typo](docs)Fix jump link 404 in external storage load.md 
(#13173)

No new revisions were added by this update.

Summary of changes:
 docs/en/docs/data-operate/import/import-scenes/external-storage-load.md | 2 +-
 .../docs/data-operate/import/import-scenes/external-storage-load.md | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] dataroaring merged pull request #13178: [typo](docs)SQL Server 2017 version ODBC usage instructions

[GitBox]

dataroaring merged PR #13178:
URL: https://github.com/apache/doris/pull/13178


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated (6b0410450b -> 4386f41442)

[dataroaring]

This is an automated email from the ASF dual-hosted git repository.

dataroaring pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


from 6b0410450b [typo](docs)Fix jump link 404 in external storage load.md 
(#13173)
 add 4386f41442 sql server 2017 version ODBC usage instructions (#13178)

No new revisions were added by this update.

Summary of changes:
 docs/en/docs/ecosystem/external-table/odbc-of-doris.md| 5 -
 docs/zh-CN/docs/ecosystem/external-table/odbc-of-doris.md | 4 +++-
 2 files changed, 7 insertions(+), 2 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] dataroaring merged pull request #13172: Update outfile.md

[GitBox]

dataroaring merged PR #13172:
URL: https://github.com/apache/doris/pull/13172


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: Update outfile.md (#13172)

[dataroaring]

This is an automated email from the ASF dual-hosted git repository.

dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 86e47650cf Update outfile.md (#13172)
86e47650cf is described below

commit 86e47650cfc2188662939e4c8b5d02379c5c2157
Author: SeekingYang <100211303+seekingy...@users.noreply.github.com>
AuthorDate: Sat Oct 8 20:01:20 2022 +0800

Update outfile.md (#13172)
---
 docs/en/docs/admin-manual/cluster-management/elastic-expansion.md | 4 ++--
 docs/en/docs/admin-manual/config/be-config.md | 2 +-
 docs/en/docs/admin-manual/maint-monitor/metadata-operation.md | 2 +-
 docs/en/docs/advanced/alter-table/schema-change.md| 2 +-
 docs/en/docs/data-operate/export/outfile.md   | 4 ++--
 5 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/docs/en/docs/admin-manual/cluster-management/elastic-expansion.md 
b/docs/en/docs/admin-manual/cluster-management/elastic-expansion.md
index b5738f6a2d..7cd849da58 100644
--- a/docs/en/docs/admin-manual/cluster-management/elastic-expansion.md
+++ b/docs/en/docs/admin-manual/cluster-management/elastic-expansion.md
@@ -106,7 +106,7 @@ You can also view the BE node through the front-end page 
connection: ``http://fe
 
 All of the above methods require Doris's root user rights.
 
-The expansion and scaling process of BE nodes does not affect the current 
system operation and the tasks being performed, and does not affect the 
performance of the current system. Data balancing is done automatically. 
Depending on the amount of data available in the cluster, the cluster will be 
restored to load balancing in a few hours to a day. For cluster load, see the 
[Tablet Load Balancing Document](../maint-monitor/tablet-meta-tool.md).
+The expansion and scaling process of BE nodes does not affect the current 
system operation and the tasks being performed, and does not affect the 
performance of the current system. Data balancing is done automatically. 
Depending on the amount of data available in the cluster, the cluster will be 
restored to load balancing in a few hours to a day. For cluster load, see the 
[Tablet Load Balancing Document](../../maint-monitor/tablet-meta-tool).
 
 ### Add BE nodes
 
@@ -140,7 +140,7 @@ DECOMMISSION clause:
 >  ```CANCEL ALTER SYSTEM DECOMMISSION BACKEND 
 > "be_host:be_heartbeat_service_port";```
 >  The order was cancelled. When cancelled, the data on the BE will 
 > maintain the current amount of data remaining. Follow-up Doris re-load 
 > balancing
 
-**For expansion and scaling of BE nodes in multi-tenant deployment 
environments, please refer to the [Multi-tenant Design 
Document](../multi-tenant.md).**
+**For expansion and scaling of BE nodes in multi-tenant deployment 
environments, please refer to the [Multi-tenant Design 
Document](../../maint-monitor/multi-tenant).**
 
 ## Broker Expansion and Shrinkage
 
diff --git a/docs/en/docs/admin-manual/config/be-config.md 
b/docs/en/docs/admin-manual/config/be-config.md
index ad1c5af365..fdc6c2 100644
--- a/docs/en/docs/admin-manual/config/be-config.md
+++ b/docs/en/docs/admin-manual/config/be-config.md
@@ -450,7 +450,7 @@ Cgroups assigned to doris
 ### `doris_max_scan_key_num`
 
 * Type: int
-* Description: Used to limit the maximum number of scan keys that a scan node 
can split in a query request. When a conditional query request reaches the scan 
node, the scan node will try to split the conditions related to the key column 
in the query condition into multiple scan key ranges. After that, these scan 
key ranges will be assigned to multiple scanner threads for data scanning. A 
larger value usually means that more scanner threads can be used to increase 
the parallelism of the s [...]
+* Description: Used to limit the maximum number of scan keys that a scan node 
can split in a query request. When a conditional query request reaches the scan 
node, the scan node will try to split the conditions related to the key column 
in the query condition into multiple scan key ranges. After that, these scan 
key ranges will be assigned to multiple scanner threads for data scanning. A 
larger value usually means that more scanner threads can be used to increase 
the parallelism of the s [...]
 * Default value: 1024
 
 When the concurrency cannot be improved in high concurrency scenarios, try to 
reduce this value and observe the impact.
diff --git a/docs/en/docs/admin-manual/maint-monitor/metadata-operation.md 
b/docs/en/docs/admin-manual/maint-monitor/metadata-operation.md
index bc2439ff58..8871c05512 100644
--- a/docs/en/docs/admin-manual/maint-monitor/metadata-operation.md
+++ b/docs/en/docs/admin-manual/maint-monitor/metadata-operation.md
@@ -32,7 +32,7 @@ For the time being, read the [Doris metadata design 
document](/community/design/
 
 ## Important tips
 
-* Current metadata de

[GitHub] [doris] dataroaring merged pull request #13170: [typo](docs)Fix jump link 404 in jdbc load.md

[GitBox]

dataroaring merged PR #13170:
URL: https://github.com/apache/doris/pull/13170


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[doris] branch master updated: [typo](docs)Fix jump link 404 in jdbc load.md (#13170)

[dataroaring]

This is an automated email from the ASF dual-hosted git repository.

dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 344377beb7 [typo](docs)Fix jump link 404 in jdbc load.md (#13170)
344377beb7 is described below

commit 344377beb778e47fa444228b33c699b3b7e02697
Author: yuanyuan8983 <99315889+yuanyuan8...@users.noreply.github.com>
AuthorDate: Sat Oct 8 20:01:52 2022 +0800

[typo](docs)Fix jump link 404 in jdbc load.md (#13170)
---
 docs/en/docs/data-operate/import/import-scenes/jdbc-load.md| 2 +-
 docs/zh-CN/docs/data-operate/import/import-scenes/jdbc-load.md | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/en/docs/data-operate/import/import-scenes/jdbc-load.md 
b/docs/en/docs/data-operate/import/import-scenes/jdbc-load.md
index 5dc6a2fd5c..a74f1c9a6b 100644
--- a/docs/en/docs/data-operate/import/import-scenes/jdbc-load.md
+++ b/docs/en/docs/data-operate/import/import-scenes/jdbc-load.md
@@ -160,4 +160,4 @@ Please note the following:
 
As mentioned earlier, we recommend that when using INSERT to import data, 
use the "batch" method to import, rather than a single insert.
 
-   At the same time, we can set a Label for each INSERT operation. Through the 
[Label mechanism](./load-atomicity.html#label-mechanism), the idempotency and 
atomicity of operations can be guaranteed, and the data will not be lost or 
heavy in the end. For the specific usage of Label in INSERT, you can refer to 
the 
[INSERT](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/INSERT.md)
 document.
+   At the same time, we can set a Label for each INSERT operation. Through the 
[Label mechanism](./load-atomicity.md#label-mechanism), the idempotency and 
atomicity of operations can be guaranteed, and the data will not be lost or 
heavy in the end. For the specific usage of Label in INSERT, you can refer to 
the 
[INSERT](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/INSERT.md)
 document.
diff --git a/docs/zh-CN/docs/data-operate/import/import-scenes/jdbc-load.md 
b/docs/zh-CN/docs/data-operate/import/import-scenes/jdbc-load.md
index c0012300fd..ebc8492fbf 100644
--- a/docs/zh-CN/docs/data-operate/import/import-scenes/jdbc-load.md
+++ b/docs/zh-CN/docs/data-operate/import/import-scenes/jdbc-load.md
@@ -160,4 +160,4 @@ public class DorisJDBCDemo {
 
前面提到，我们建议在使用 INSERT 导入数据时，采用 ”批“ 的方式进行导入，而不是单条插入。
 
-   同时，我们可以为每次 INSERT 操作设置一个 Label。通过 [Label 
机制](./load-atomicity.html#label-机制) 可以保证操作的幂等性和原子性，最终做到数据的不丢不重。关于 INSERT 中 
Label 的具体用法，可以参阅 
[INSERT](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/INSERT.md)
 文档。
+   同时，我们可以为每次 INSERT 操作设置一个 Label。通过 [Label 机制](./load-atomicity.md#label-机制) 
可以保证操作的幂等性和原子性，最终做到数据的不丢不重。关于 INSERT 中 Label 的具体用法，可以参阅 
[INSERT](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/INSERT.md)
 文档。


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] ByteYue opened a new pull request, #13185: [enhancement](regression-test) add sync for tests containing stream load

[GitBox]

ByteYue opened a new pull request, #13185:
URL: https://github.com/apache/doris/pull/13185

   # Proposed changes
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   The former code in cases like `test_stream_load` didn't consider the data 
sync, which might cause the qt_sql right after the stream load turning out to 
fail.
   By simply add the `sql "sync"` before the qt_sql, the problems might be 
resolved.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
   - [ ] Yes
   - [x] No
   - [ ] I don't know
   2. Has unit tests been added:
   - [ ] Yes
   - [x] No
   - [ ] No Need
   3. Has document been added or modified:
   - [ ] Yes
   - [ ] No
   - [x] No Need
   4. Does it need to update dependencies:
   - [ ] Yes
   - [x] No
   5. Are there any changes that cannot be rolled back:
   - [ ] Yes (If Yes, please explain WHY)
   - [x] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at 
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you 
chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [doris] cambyzju commented on a diff in pull request #12897: [feature](remote) support local cache GC by disk usage

[GitBox]

cambyzju commented on code in PR #12897:
URL: https://github.com/apache/doris/pull/12897#discussion_r990634022


##
be/src/io/cache/file_cache_manager.cpp:
##
@@ -56,88 +80,82 @@ void FileCacheManager::remove_file_cache(const std::string& 
cache_path) {
 }
 }
 
-void FileCacheManager::clean_timeout_caches() {
-std::shared_lock rdlock(_cache_map_lock);
-for (std::map::const_iterator iter = 
_file_cache_map.cbegin();
- iter != _file_cache_map.cend(); ++iter) {
-if (iter->second == nullptr) {
-continue;
+void 
FileCacheManager::_add_file_cache_for_gc_by_disk(std::vector& 
contexts,
+  FileCachePtr file_cache) 
{
+// sort file cache by last match time
+if (config::file_cache_max_size_per_disk > 0) {
+auto file_size = file_cache->cache_file_size();
+if (file_size <= 0) {
+return;
+}
+for (size_t i = 0; i < contexts.size(); ++i) {
+if (contexts[i].try_add_file_cache(file_cache, file_size)) {
+break;
+}
 }
-iter->second->clean_timeout_cache();
 }
 }
+void FileCacheManager::gc_file_caches() {
+int64_t gc_conf_size = config::file_cache_max_size_per_disk;
+std::vector contexts;
+// init for GC by disk size
+if (gc_conf_size > 0) {
+std::vector data_dirs = 
doris::StorageEngine::instance()->get_stores();
+contexts.resize(data_dirs.size());
+for (size_t i = 0; i < contexts.size(); ++i) {
+contexts[i].init(data_dirs[i]->path(), gc_conf_size);
+}
+}
 
-void FileCacheManager::clean_timeout_file_not_in_mem(const std::string& 
cache_path) {
-time_t now = time(nullptr);
+// process unused file caches
 std::shared_lock rdlock(_cache_map_lock);
-// Deal with caches not in _file_cache_map
-if (_file_cache_map.find(cache_path) == _file_cache_map.end()) {
-std::vector cache_file_names;
-if (io::global_local_filesystem()->list(cache_path, 
&cache_file_names).ok()) {
-std::map cache_names;
-std::list done_names;
-for (Path cache_file_name : cache_file_names) {
-std::string filename = cache_file_name.native();
-if (!ends_with(filename, CACHE_DONE_FILE_SUFFIX)) {
-cache_names[filename] = true;
-continue;
-}
-done_names.push_back(filename);
-std::stringstream done_file_ss;
-done_file_ss << cache_path << "/" << filename;
-std::string done_file_path = done_file_ss.str();
-time_t m_time;
-if (!FileUtils::mtime(done_file_path, &m_time).ok()) {
-continue;
-}
-if (now - m_time < config::file_cache_alive_time_sec) {
-continue;
-}
-std::string cache_file_path =
-StringReplace(done_file_path, CACHE_DONE_FILE_SUFFIX, 
"", true);
-LOG(INFO) << "Delete timeout done_cache_path: " << 
done_file_path
-  << ", cache_file_path: " << cache_file_path << ", 
m_time: " << m_time;
-if 
(!io::global_local_filesystem()->delete_file(done_file_path).ok()) {
-LOG(ERROR) << "delete_file failed: " << done_file_path;
+std::vector tablets =
+StorageEngine::instance()->tablet_manager()->get_all_tablet();
+for (const auto& tablet : tablets) {
+std::vector seg_file_paths;
+if (io::global_local_filesystem()->list(tablet->tablet_path(), 
&seg_file_paths).ok()) {
+for (Path seg_file : seg_file_paths) {
+std::string seg_filename = seg_file.native();
+// check if it is a dir name
+if (ends_with(seg_filename, ".dat")) {
 continue;
 }
-if 
(!io::global_local_filesystem()->delete_file(cache_file_path).ok()) {
-LOG(ERROR) << "delete_file failed: " << cache_file_path;
+// skip file cache already in memory
+std::stringstream ss;
+ss << tablet->tablet_path() << "/" << seg_filename;
+std::string cache_path = ss.str();
+if (_file_cache_map.find(cache_path) != _file_cache_map.end()) 
{
 continue;
 }
+
+auto file_cache = std::make_shared(
+cache_path, config::file_cache_alive_time_sec);
+// load cache meta from disk and clean unfinished cache files
+file_cache->load_and_clean();
+// policy1: GC file cache by timeout
+file_cache->clean_timeout_cache();
+// sort file cache by last match time
+_add_file_cache_for_gc_by_disk(contexts, file_cac