[PR] [chore](third) Upgrade curl 7.79.0 -> 8.2.1 [doris]
w41ter opened a new pull request, #37893: URL: https://github.com/apache/doris/pull/37893 ref #23414. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](third) Upgrade curl 7.79.0 -> 8.2.1 [doris]
doris-robot commented on PR #37893: URL: https://github.com/apache/doris/pull/37893#issuecomment-2230165008 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) Refactor character conversion to use Java 8's String.valueOf() [doris]
doris-robot commented on PR #37773: URL: https://github.com/apache/doris/pull/37773#issuecomment-2230165108 TPC-DS: Total hot run time: 173929 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 20246ba9728261cf743a878ac2d75fb71cc6e8a0, data reload: false query1 933 368 358 358 query2 6450184218081808 query3 6633205 215 205 query4 27832 17512 17505 17505 query5 3660505 476 476 query6 257 176 172 172 query7 4577283 285 283 query8 235 194 192 192 query9 8659240223932393 query10 431 299 274 274 query11 10728 10162 10083 10083 query12 117 88 82 82 query13 1645378 375 375 query14 10196 833475377537 query15 239 167 170 167 query16 7823329 309 309 query17 1800566 544 544 query18 1976280 283 280 query19 198 162 159 159 query20 99 82 82 82 query21 212 137 127 127 query22 4370425742024202 query23 33858 33851 33684 33684 query24 11090 292129092909 query25 626 422 407 407 query26 917 152 157 152 query27 2206279 275 275 query28 6153207720882077 query29 913 640 654 640 query30 253 157 161 157 query31 994 775 793 775 query32 97 54 60 54 query33 849 295 285 285 query34 1056492 491 491 query35 687 602 616 602 query36 1115981 999 981 query37 147 81 85 81 query38 3043284228092809 query39 923 832 828 828 query40 212 123 123 123 query41 47 45 46 45 query42 122 103 105 103 query43 503 461 474 461 query44 1187721 725 721 query45 189 156 164 156 query46 1096723 737 723 query47 1880180018081800 query48 352 290 294 290 query49 829 407 410 407 query50 772 391 384 384 query51 6858683767066706 query52 99 95 89 89 query53 356 279 295 279 query54 900 455 438 438 query55 73 71 75 71 query56 284 282 262 262 query57 1141105410671054 query58 235 246 262 246 query59 2776268025892589 query60 307 275 286 275 query61 96 91 95 91 query62 787 641 657 641 query63 314 278 295 278 query64 9272219316521652 query65 3178312931133113 query66 694 322 331 322 query67 15833 15158 14956 14956 query68 4545585 556 556 query69 616 380 339 339 query70 1109116810371037 query71 424 278 278 278 query72 8606554064145540 query73 742 332 329 329 query74 6146566657045666 query75 3760268326712671 query76 3318992 892 892 query77 691 299 303 299 query78 9620973290039003 query79 5857518 524 518 query80 3031481 473 473 query81 596 221 220 220 query82 1436139 140 139 query83 308 220 166 166 query84 274 83 84 83 query85 2019314 295 295 query86 426 283 303 283 query87 3309312831923128 query88 4679235823882358 query89 486 387 368 368 query90 1987195 189 189 query91 133 101 101 101 query92 61 49 49 49 query93 5431519 506 506 query94 1295252 216 216 query95 391 312 309 309 query96 601 276 276 276 query97 3172310430883088 query98 218 200 193 193 query99 1526126812661266 Total cold run time: 291202 ms Total hot run time: 173929 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th
Re: [PR] [fix](function) the bucket number arg of width_bucket should be a positive integer value [doris]
github-actions[bot] commented on PR #37892: URL: https://github.com/apache/doris/pull/37892#issuecomment-2230165141 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](third) Upgrade curl 7.79.0 -> 8.2.1 [doris]
w41ter commented on PR #37893: URL: https://github.com/apache/doris/pull/37893#issuecomment-2230166021 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](mtmv) use isManagedTable instead of check table type (#34287) (#37822)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 2c80259f668 [fix](mtmv) use isManagedTable instead of check table type
(#34287) (#37822)
2c80259f668 is described below
commit 2c80259f668738c3d75b965aff8e77e9c5c580ab
Author: zhangdong <493738...@qq.com>
AuthorDate: Tue Jul 16 15:01:28 2024 +0800
[fix](mtmv) use isManagedTable instead of check table type (#34287) (#37822)
pick: https://github.com/apache/doris/pull/34287
---
.../java/org/apache/doris/analysis/Analyzer.java | 2 +-
.../java/org/apache/doris/analysis/SelectStmt.java | 3 +-
.../apache/doris/catalog/CatalogRecycleBin.java| 33 +---
.../java/org/apache/doris/catalog/Database.java| 8 +-
.../main/java/org/apache/doris/catalog/Env.java| 10 +--
.../apache/doris/common/proc/IndexInfoProcDir.java | 5 +-
.../common/proc/RemoteIndexSchemaProcDir.java | 3 +-
.../doris/common/proc/StatisticProcNode.java | 3 +-
.../doris/consistency/ConsistencyChecker.java | 3 +-
.../apache/doris/datasource/InternalCatalog.java | 9 ++-
.../org/apache/doris/httpv2/rest/ShowAction.java | 4 +-
.../doris/httpv2/rest/StorageTypeCheckAction.java | 3 +-
.../main/java/org/apache/doris/load/ExportJob.java | 2 +-
.../org/apache/doris/load/ExportTaskExecutor.java | 3 +-
.../trees/plans/commands/info/DropMTMVInfo.java| 2 +-
.../apache/doris/statistics/query/IndexStats.java | 3 +-
.../apache/doris/statistics/query/QueryStats.java | 15 ++--
.../apache/doris/statistics/query/TableStats.java | 3 +-
.../suites/mtmv_p0/test_recover_mtmv.groovy| 93 ++
19 files changed, 152 insertions(+), 55 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/Analyzer.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/Analyzer.java
index 86ce4143569..861dc517a04 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/Analyzer.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/Analyzer.java
@@ -841,7 +841,7 @@ public class Analyzer {
.getDbOrAnalysisException(tableName.getDb());
TableIf table =
database.getTableOrAnalysisException(tableName.getTbl());
-if (table.getType() == TableType.OLAP && (((OlapTable)
table).getState() == OlapTableState.RESTORE
+if (table.isManagedTable() && (((OlapTable) table).getState() ==
OlapTableState.RESTORE
|| ((OlapTable) table).getState() ==
OlapTableState.RESTORE_WITH_LOAD)) {
Boolean isNotRestoring = ((OlapTable)
table).getPartitions().stream()
.filter(partition -> partition.getState() ==
PartitionState.RESTORE).collect(Collectors.toList())
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
index ee2d33b96b1..97b929762ed 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
@@ -30,7 +30,6 @@ import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.PrimitiveType;
import org.apache.doris.catalog.Table;
import org.apache.doris.catalog.TableIf;
-import org.apache.doris.catalog.TableIf.TableType;
import org.apache.doris.catalog.Type;
import org.apache.doris.catalog.View;
import org.apache.doris.common.AnalysisException;
@@ -1087,7 +1086,7 @@ public class SelectStmt extends QueryStmt {
break;
}
long rowCount = 0;
-if (tblRef.getTable().getType() == TableType.OLAP) {
+if (tblRef.getTable().isManagedTable()) {
rowCount = ((OlapTable) (tblRef.getTable())).getRowCount();
if (LOG.isDebugEnabled()) {
LOG.debug("tableName={} rowCount={}", tblRef.getAlias(),
rowCount);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java
index 46a5ad26ed0..7d31a8c6bb3 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java
@@ -112,7 +112,7 @@ public class CatalogRecycleBin extends MasterDaemon
implements Writable {
}
private void addRecycledTabletsForTable(Set recycledTabletSet, Table
table) {
-if (table.getType() == TableType.OLAP) {
+if (table.isManagedTable()) {
OlapTable olapTable = (OlapTable) table;
Collection allPartitions = olapTable.getAllPartitions();
for (Partition partition : allPartitions) {
@@ -321,7 +321,7 @@ public class CatalogRecycleBin extends MasterDaemon
implements Writable {
Re: [PR] [Improve]Schema change parses ddl sql using jsqlparser framework [doris-flink-connector]
JNSimba commented on code in PR #422:
URL:
https://github.com/apache/doris-flink-connector/pull/422#discussion_r1678862412
##
flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/serializer/JsonDebeziumSchemaSerializer.java:
##
@@ -218,6 +239,14 @@ public JsonDebeziumSchemaSerializer.Builder
setNewSchemaChange(boolean newSchema
return this;
}
+public JsonDebeziumSchemaSerializer.Builder setSchemaChangeMode(String
schemaChangeMode) {
+if
(org.apache.commons.lang3.StringUtils.isEmpty(schemaChangeMode)) {
+return this;
+}
Review Comment:
Which schemachangemode is used when it is empty?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](mtmv) use isManagedTable instead of check table type (#34287) [doris]
yiguolei merged PR #37822: URL: https://github.com/apache/doris/pull/37822 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](sql function) memory overflow to the left of string address when do_money_format has small negative value #36226 [doris]
yiguolei merged PR #37870: URL: https://github.com/apache/doris/pull/37870 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [Fix](sql function) memory overflow to the left of string address when do_money_format has small negative value #36226 (#37870)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 02716598d48 [Fix](sql function) memory overflow to the left of string
address when do_money_format has small negative value #36226 (#37870)
02716598d48 is described below
commit 02716598d48a83855519c88045b20d9185bf06f5
Author: zhiqiang
AuthorDate: Tue Jul 16 15:04:42 2024 +0800
[Fix](sql function) memory overflow to the left of string address when
do_money_format has small negative value #36226 (#37870)
cherry pick from #36226
Co-authored-by: sparrow <38098988+biohazard4...@users.noreply.github.com>
---
be/src/vec/functions/function_string.h | 5 -
be/test/vec/function/function_math_test.cpp | 3 ++-
2 files changed, 6 insertions(+), 2 deletions(-)
diff --git a/be/src/vec/functions/function_string.h
b/be/src/vec/functions/function_string.h
index 4096bcca6d3..df15bc600ec 100644
--- a/be/src/vec/functions/function_string.h
+++ b/be/src/vec/functions/function_string.h
@@ -3073,7 +3073,7 @@ static StringRef do_money_format(FunctionContext*
context, const string& value)
if (!is_positive) {
*result_data = '-';
}
-for (int i = value.size() - 4, j = result_len - 4; i >= 0; i = i - 3, j =
j - 4) {
+for (int i = value.size() - 4, j = result_len - 4; i >= 0; i = i - 3) {
*(result_data + j) = *(value.data() + i);
if (i - 1 < 0) {
break;
@@ -3085,6 +3085,9 @@ static StringRef do_money_format(FunctionContext*
context, const string& value)
*(result_data + j - 2) = *(value.data() + i - 2);
if (j - 3 > 1 || (j - 3 == 1 && is_positive)) {
*(result_data + j - 3) = ',';
+j -= 4;
+} else {
+j -= 3;
}
}
memcpy(result_data + result_len - 3, value.data() + value.size() - 3, 3);
diff --git a/be/test/vec/function/function_math_test.cpp
b/be/test/vec/function/function_math_test.cpp
index c93c6ca324f..00d0770935b 100644
--- a/be/test/vec/function/function_math_test.cpp
+++ b/be/test/vec/function/function_math_test.cpp
@@ -511,7 +511,8 @@ TEST(MathFunctionTest, money_format_test) {
InputTypeSet input_types = {TypeIndex::Float64};
DataSet data_set = {{{Null()}, Null()},
{{DOUBLE(17014116.67)}, VARCHAR("17,014,116.67")},
-{{DOUBLE(-17014116.67)},
VARCHAR("-17,014,116.67")}};
+{{DOUBLE(-17014116.67)},
VARCHAR("-17,014,116.67")},
+{{DOUBLE(-123.45)}, VARCHAR("-123.45")}};
static_cast(check_function(func_name,
input_types, data_set));
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) Refactor character conversion to use Java 8's String.valueOf() [doris]
doris-robot commented on PR #37773: URL: https://github.com/apache/doris/pull/37773#issuecomment-2230174014 ClickBench: Total hot run time: 31.54 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 20246ba9728261cf743a878ac2d75fb71cc6e8a0, data reload: false query1 0.040.040.03 query2 0.080.040.04 query3 0.220.060.06 query4 1.660.090.10 query5 0.500.500.48 query6 1.140.740.72 query7 0.020.010.01 query8 0.050.040.04 query9 0.540.480.50 query10 0.550.540.55 query11 0.150.110.11 query12 0.150.130.12 query13 0.580.590.58 query14 0.780.760.81 query15 0.860.820.81 query16 0.370.380.37 query17 1.051.001.06 query18 0.220.210.21 query19 1.891.761.85 query20 0.020.010.01 query21 15.45 0.750.67 query22 4.386.172.68 query23 18.32 1.331.35 query24 2.190.220.23 query25 0.160.090.08 query26 0.320.210.20 query27 0.450.230.22 query28 13.22 1.021.00 query29 12.60 3.343.26 query30 0.250.050.06 query31 2.870.390.40 query32 3.260.470.47 query33 2.912.882.93 query34 16.93 4.414.41 query35 4.464.414.42 query36 0.650.470.47 query37 0.180.160.16 query38 0.150.140.14 query39 0.050.040.04 query40 0.150.120.12 query41 0.080.040.04 query42 0.050.040.05 query43 0.040.040.04 Total cold run time: 109.99 s Total hot run time: 31.54 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cloud) provide a conf to enable/disable streamload commit on be [doris]
freemandealer commented on PR #37855: URL: https://github.com/apache/doris/pull/37855#issuecomment-2230176035 run external -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](nereids)subquery unnesting get wrong result if correlated conjuncts is not slot_a = slot_b (#37683)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 80ea98b371a [fix](nereids)subquery unnesting get wrong result if
correlated conjuncts is not slot_a = slot_b (#37683)
80ea98b371a is described below
commit 80ea98b371a5aa55efa049f501ca305ba136f7a7
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Tue Jul 16 15:06:40 2024 +0800
[fix](nereids)subquery unnesting get wrong result if correlated conjuncts
is not slot_a = slot_b (#37683)
pick from master https://github.com/apache/doris/pull/37644
## Proposed changes
Issue Number: close #xxx
---
.../doris/nereids/jobs/executor/Rewriter.java | 6 +-
.../nereids/rules/analysis/SubExprAnalyzer.java| 10 +++
.../rewrite/UnCorrelatedApplyAggregateFilter.java | 18 +-
.../trees/plans/logical/LogicalProject.java| 4 ++
.../java/org/apache/doris/nereids/util/Utils.java | 56 ++---
.../nereids_syntax_p0/test_subquery_conjunct.out | 54
.../sub_query_diff_old_optimize.groovy | 2 +-
.../test_subquery_conjunct.groovy | 71 ++
8 files changed, 206 insertions(+), 15 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
index a13e784c7a8..9df26fce31f 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
@@ -200,7 +200,11 @@ public class Rewriter extends AbstractBatchJobExecutor {
* TODO: group these rules to make sure the
result plan is what we expected.
*/
new CorrelateApplyToUnCorrelateApply(),
-new ApplyToJoin()
+new ApplyToJoin(),
+// UnCorrelatedApplyAggregateFilter rule will
create new aggregate outputs,
+// The later rule CheckPrivileges which
inherent from ColumnPruning only works
+// if the aggregation node is normalized, so
we need call NormalizeAggregate here
+new NormalizeAggregate()
)
),
// before `Subquery unnesting` topic, some correlate slots
should have appeared at LogicalApply.left,
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubExprAnalyzer.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubExprAnalyzer.java
index d52fc96ff91..7bfd5256f6a 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubExprAnalyzer.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubExprAnalyzer.java
@@ -75,6 +75,11 @@ class SubExprAnalyzer extends
DefaultExpressionRewriter {
@Override
public Expression visitExistsSubquery(Exists exists, T context) {
+LogicalPlan queryPlan = exists.getQueryPlan();
+// distinct is useless, remove it
+if (queryPlan instanceof LogicalProject && ((LogicalProject)
queryPlan).isDistinct()) {
+exists = exists.withSubquery(((LogicalProject)
queryPlan).withDistinct(false));
+}
AnalyzedResult analyzedResult = analyzeSubquery(exists);
if (analyzedResult.rootIsLimitZero()) {
return BooleanLiteral.of(exists.isNot());
@@ -89,6 +94,11 @@ class SubExprAnalyzer extends
DefaultExpressionRewriter {
@Override
public Expression visitInSubquery(InSubquery expr, T context) {
+LogicalPlan queryPlan = expr.getQueryPlan();
+// distinct is useless, remove it
+if (queryPlan instanceof LogicalProject && ((LogicalProject)
queryPlan).isDistinct()) {
+expr = expr.withSubquery(((LogicalProject)
queryPlan).withDistinct(false));
+}
AnalyzedResult analyzedResult = analyzeSubquery(expr);
checkOutputColumn(analyzedResult.getLogicalPlan());
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/UnCorrelatedApplyAggregateFilter.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/UnCorrelatedApplyAggregateFilter.java
index 211e76710c5..f4acd847652 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/UnCorrelatedApplyAggregateFilter.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/UnCorrelatedApplyAggregateFilter.java
@@ -19,8 +19,10 @@ package org.apache.doris.nereids.rules.rewrite;
import org.apache.doris.nereids.rules.Rule;
i
Re: [PR] [fix](nereids)subquery unnesting get wrong result if correlated conjuncts is not slot_a = slot_b [doris]
yiguolei merged PR #37683: URL: https://github.com/apache/doris/pull/37683 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve]Schema change parses ddl sql using jsqlparser framework [doris-flink-connector]
JNSimba commented on code in PR #422:
URL:
https://github.com/apache/doris-flink-connector/pull/422#discussion_r1678868754
##
flink-doris-connector/src/test/java/org/apache/doris/flink/tools/cdc/CdcMysqlSyncDatabaseCase.java:
##
@@ -90,6 +92,7 @@ public static void main(String[] args) throws Exception {
.setTableConfig(tableConfig)
.setCreateTableOnly(false)
.setNewSchemaChange(useNewSchemaChange)
+.setSchemaChangeMode(schemaChangeMode)
Review Comment:
What will happen if this is not set?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](regression test) fix test_schema_change_fail due to set force_olap_table_replication_num #34343 [doris]
yiguolei commented on PR #37860: URL: https://github.com/apache/doris/pull/37860#issuecomment-2230177176 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](backup) Fix the db name of the restored view [doris]
doris-robot commented on PR #37412: URL: https://github.com/apache/doris/pull/37412#issuecomment-2230176561 TPC-H: Total hot run time: 40202 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit f90c857298d66d3f3e25166911a2362961889c43, data reload: false -- Round 1 -- q1 17858 438143114311 q2 2002193 186 186 q3 10451 121711071107 q4 10186 878 840 840 q5 8636269426642664 q6 217 137 135 135 q7 958 598 604 598 q8 9229206620972066 q9 9118668966866686 q10 9028396640353966 q11 484 250 242 242 q12 403 230 222 222 q13 17952 296329872963 q14 276 238 253 238 q15 524 493 494 493 q16 516 395 383 383 q17 994 683 670 670 q18 8587741273507350 q19 7970131314841313 q20 672 331 306 306 q21 4808324331783178 q22 350 294 285 285 Total cold run time: 121219 ms Total hot run time: 40202 ms - Round 2, with runtime_filter_mode=off - q1 4379425142154215 q2 373 265 263 263 q3 3155295728732873 q4 2000176817021702 q5 5559549855915498 q6 224 137 134 134 q7 2234183818551838 q8 3285343833773377 q9 8778896387498749 q10 4020384439003844 q11 599 525 501 501 q12 807 660 634 634 q13 16293 312231373122 q14 305 296 285 285 q15 523 488 487 487 q16 483 424 420 420 q17 1791154515091509 q18 8097795678967896 q19 1722144517091445 q20 2526187319071873 q21 4937466348754663 q22 602 499 499 499 Total cold run time: 72692 ms Total hot run time: 55827 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](MoW) fix MoW & segcompaction conflict on cache of temp segment [doris]
dataroaring merged PR #37760: URL: https://github.com/apache/doris/pull/37760 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](MoW) fix MoW & segcompaction conflict on cache of temp segment (#37760)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new ba2b37157a2 [fix](MoW) fix MoW & segcompaction conflict on cache of temp segment (#37760) ba2b37157a2 is described below commit ba2b37157a23a91b2ef1223f9befbb81dac20d11 Author: zhengyu AuthorDate: Tue Jul 16 15:07:50 2024 +0800 [fix](MoW) fix MoW & segcompaction conflict on cache of temp segment (#37760) MoW will update delete bitmap during load, and the page cache could be modified by segcompaction. Disable page cache touchs when doing segcompaction could solve this problem. ## Proposed changes Issue Number: close #xxx Signed-off-by: freemandealer --- be/src/olap/rowset/segcompaction.cpp | 1 + 1 file changed, 1 insertion(+) diff --git a/be/src/olap/rowset/segcompaction.cpp b/be/src/olap/rowset/segcompaction.cpp index 95f2a945134..f3e8d9f085c 100644 --- a/be/src/olap/rowset/segcompaction.cpp +++ b/be/src/olap/rowset/segcompaction.cpp @@ -101,6 +101,7 @@ Status SegcompactionWorker::_get_segcompaction_reader( reader_params.tablet = tablet; reader_params.return_columns = return_columns; reader_params.is_key_column_group = is_key; +reader_params.use_page_cache = false; return (*reader)->init(reader_params, nullptr); } - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](arrow-flight-sql) Fix get Schema failed when `enable_parallel_result_sink` is false [doris]
xinyiZzz merged PR #37779: URL: https://github.com/apache/doris/pull/37779 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (ba2b37157a2 -> a82aed8ab12)
This is an automated email from the ASF dual-hosted git repository. zouxinyi pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from ba2b37157a2 [fix](MoW) fix MoW & segcompaction conflict on cache of temp segment (#37760) add a82aed8ab12 [fix](arrow-flight-sql) Fix get Schema failed when `enable_parallel_result_sink` is false (#37779) No new revisions were added by this update. Summary of changes: be/src/pipeline/exec/result_sink_operator.cpp | 7 ++- be/src/service/arrow_flight/arrow_flight_batch_reader.cpp | 2 +- be/src/service/internal_service.cpp | 4 ++-- .../doris/service/arrowflight/FlightSqlConnectProcessor.java | 8 +++- 4 files changed, 16 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve]Schema change parses ddl sql using jsqlparser framework [doris-flink-connector]
DongLiang-0 commented on code in PR #422:
URL:
https://github.com/apache/doris-flink-connector/pull/422#discussion_r1678869750
##
flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/serializer/JsonDebeziumSchemaSerializer.java:
##
@@ -218,6 +239,14 @@ public JsonDebeziumSchemaSerializer.Builder
setNewSchemaChange(boolean newSchema
return this;
}
+public JsonDebeziumSchemaSerializer.Builder setSchemaChangeMode(String
schemaChangeMode) {
+if
(org.apache.commons.lang3.StringUtils.isEmpty(schemaChangeMode)) {
+return this;
+}
Review Comment:
JsonDebeziumSchemaChangeImplV2 is created when empty.
https://github.com/user-attachments/assets/7ceaa96b-16d2-423f-b4a7-c88393948bf2";>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve]Schema change parses ddl sql using jsqlparser framework [doris-flink-connector]
JNSimba commented on code in PR #422:
URL:
https://github.com/apache/doris-flink-connector/pull/422#discussion_r1678870394
##
flink-doris-connector/src/main/java/org/apache/doris/flink/sink/schema/SQLParserManager.java:
##
@@ -0,0 +1,197 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.flink.sink.schema;
+
+import net.sf.jsqlparser.JSQLParserException;
+import net.sf.jsqlparser.parser.CCJSqlParserUtil;
+import net.sf.jsqlparser.statement.Statement;
+import net.sf.jsqlparser.statement.alter.Alter;
+import net.sf.jsqlparser.statement.alter.AlterExpression;
+import net.sf.jsqlparser.statement.alter.AlterExpression.ColumnDataType;
+import net.sf.jsqlparser.statement.alter.AlterOperation;
+import net.sf.jsqlparser.statement.create.table.ColDataType;
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.doris.flink.catalog.doris.FieldSchema;
+import
org.apache.doris.flink.sink.writer.serializer.jsondebezium.JsonDebeziumChangeUtils;
+import org.apache.doris.flink.tools.cdc.SourceConnector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/** Use {@link net.sf.jsqlparser.parser.CCJSqlParserUtil} to parse SQL
statements. */
+public class SQLParserManager implements Serializable {
+private static final Logger LOG =
LoggerFactory.getLogger(SQLParserManager.class);
+
+/**
+ * Doris' schema change only supports ADD, DROP, and RENAME operations.
This method is only used
+ * to parse the above schema change operations.
+ */
+public List parserAlterDDLs(
Review Comment:
add ut for this method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cloud) provide a conf to enable/disable streamload commit on be [doris]
github-actions[bot] commented on PR #37855: URL: https://github.com/apache/doris/pull/37855#issuecomment-2230179884 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve]Schema change parses ddl sql using jsqlparser framework [doris-flink-connector]
DongLiang-0 commented on code in PR #422:
URL:
https://github.com/apache/doris-flink-connector/pull/422#discussion_r1678871101
##
flink-doris-connector/src/test/java/org/apache/doris/flink/tools/cdc/CdcMysqlSyncDatabaseCase.java:
##
@@ -90,6 +92,7 @@ public static void main(String[] args) throws Exception {
.setTableConfig(tableConfig)
.setCreateTableOnly(false)
.setNewSchemaChange(useNewSchemaChange)
+.setSchemaChangeMode(schemaChangeMode)
Review Comment:
Same as above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](Recycler) Parallelize s3 delete operations and recycle_tablet [doris]
platoneko commented on code in PR #37630:
URL: https://github.com/apache/doris/pull/37630#discussion_r1678802435
##
cloud/src/recycler/s3_accessor.h:
##
@@ -29,11 +29,14 @@ class S3Client;
namespace doris::cloud {
class ObjectStoreInfoPB;
+class SimpleThreadPool;
enum class S3RateLimitType;
extern int reset_s3_rate_limiter(S3RateLimitType type, size_t max_speed,
size_t max_burst,
size_t limit);
+extern std::shared_ptr worker_pool;
Review Comment:
No need to extern this symbol
##
cloud/src/recycler/s3_accessor.cpp:
##
@@ -249,7 +250,15 @@ int S3Accessor::create(S3Conf conf,
std::shared_ptr* accessor) {
return (*accessor)->init();
}
+std::shared_ptr worker_pool;
Review Comment:
static std::shared_ptr worker_pool;
##
cloud/src/recycler/recycler.h:
##
@@ -83,11 +92,13 @@ class Recycler {
WhiteBlackList instance_filter_;
std::unique_ptr checker_;
+std::unique_ptr _thread_pool_group;
Review Comment:
use `std::shared_ptr`
##
cloud/src/recycler/util.h:
##
@@ -17,15 +17,112 @@
#pragma once
+#include
#include
#include
+#include
+#include
#include
+#include "common/simple_thread_pool.h"
+
namespace doris::cloud {
class TxnKv;
+template
+class SyncExecutor {
+public:
+SyncExecutor(
+SimpleThreadPool* pool, std::string name_tag,
+std::function cancel = [](const T& /**/) { return
false; })
+: _pool(pool), _cancel(std::move(cancel)),
_name_tag(std::move(name_tag)) {}
+auto add(std::function callback) -> SyncExecutor& {
+auto task = std::make_unique(std::move(callback), _cancel,
_count);
+_count.add_count();
+// The actual task logic would be wrapped by one promise and passed to
the threadpool.
+// The result would be returned by the future once the task is
finished.
+// Or the task would be invalid if the whole task is cancelled.
+int r = _pool->submit([this, t = task.get()]() { (*t)(_stop_token); });
+CHECK(r == 0);
+_res.emplace_back(std::move(task));
+return *this;
+}
+std::vector when_all(bool* finished) {
+timespec current_time;
+auto current_time_second = time(nullptr);
+current_time.tv_sec = current_time_second + 300;
+current_time.tv_nsec = 0;
+auto msg = fmt::format("{} has already taken 5 min", _name_tag);
+while (0 != _count.timed_wait(current_time)) {
+current_time.tv_sec += 300;
+LOG(WARNING) << msg;
+}
+*finished = !_stop_token;
+std::vector res;
+res.reserve(_res.size());
+for (auto& task : _res) {
+if (!task->valid()) {
+*finished = false;
+return res;
+}
+res.emplace_back((*task).get());
+}
+return res;
+}
+void reset() {
+_res.clear();
+_stop_token = false;
+}
+
+private:
+class Task {
+public:
+Task(std::function callback, std::function cancel,
+ bthread::CountdownEvent& count)
+: _callback(std::move(callback)),
+ _cancel(std::move(cancel)),
+ _count(count),
+ _fut(_pro.get_future()) {}
+void operator()(std::atomic_bool& stop_token) {
+std::unique_ptr> defer((int*)0x01,
+ [&](int*) {
_count.signal(); });
+if (stop_token) {
+_valid = false;
+return;
+}
+T t = _callback();
+// We'll return this task result to user even if this task return
error
+// So we don't set _valid to false here
+if (_cancel(t)) {
+stop_token = true;
+}
+_pro.set_value(std::move(t));
+}
+bool valid() { return _valid; }
+T get() { return _fut.get(); }
+
+private:
+// It's guarantted that the valid function can only be called inside
SyncExecutor's `when_all()` function
+// and only be called when the _count.timed_wait function returned. So
there would be no data race for
+// _valid then it doesn't need to be one atomic bool.
+bool _valid = true;
+std::function _callback;
+std::function _cancel;
+std::promise _pro;
+bthread::CountdownEvent& _count;
+std::future _fut;
+};
+std::vector> _res;
+// use CountdownEvent to do periodically log using
CountdownEvent::time_wait()
+bthread::CountdownEvent _count {0};
+std::atomic_bool _stop_token {false};
+SimpleThreadPool* _pool;
Review Comment:
use `shared_ptr`
##
cloud/src/recycler/obj_storage_client.h:
##
@@ -51,6 +51,12 @@ class ObjectListIterator {
virtual std::optional next() = 0;
};
+cla
Re: [PR] [Improve]Schema change parses ddl sql using jsqlparser framework [doris-flink-connector]
DongLiang-0 commented on code in PR #422:
URL:
https://github.com/apache/doris-flink-connector/pull/422#discussion_r1678869750
##
flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/serializer/JsonDebeziumSchemaSerializer.java:
##
@@ -218,6 +239,14 @@ public JsonDebeziumSchemaSerializer.Builder
setNewSchemaChange(boolean newSchema
return this;
}
+public JsonDebeziumSchemaSerializer.Builder setSchemaChangeMode(String
schemaChangeMode) {
+if
(org.apache.commons.lang3.StringUtils.isEmpty(schemaChangeMode)) {
+return this;
+}
Review Comment:
`JsonDebeziumSchemaChangeImplV2` is created when empty.
https://github.com/user-attachments/assets/7ceaa96b-16d2-423f-b4a7-c88393948bf2";>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) Pick "[Fix](bloom filter) Fix bloom filter memory leak (#34871)" [doris]
Yukang-Lian closed pull request #37824: [cherry-pick](branch-2.0) Pick "[Fix](bloom filter) Fix bloom filter memory leak (#34871)" URL: https://github.com/apache/doris/pull/37824 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](multi-catalog)fix paimon meta properties convert [doris]
github-actions[bot] commented on PR #37249: URL: https://github.com/apache/doris/pull/37249#issuecomment-2230184968 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](regression)Fix test_agg_complex_type fail case. Improve analyze_stats time consumption. [doris]
Jibing-Li opened a new pull request, #37895: URL: https://github.com/apache/doris/pull/37895 Reduce time consumption of analyze_stats case. Fix cloud version failed case test_agg_complex_type, set force_sample_analyze to false. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](regression)Fix test_agg_complex_type fail case. Improve analyze_stats time consumption. [doris]
doris-robot commented on PR #37895: URL: https://github.com/apache/doris/pull/37895#issuecomment-2230185412 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [opt](column nullable) Optimization for ColumnNullable serialization [doris]
zhiqiang- opened a new pull request, #37896: URL: https://github.com/apache/doris/pull/37896 about 25% improvement -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](regression)Fix test_agg_complex_type fail case. Improve analyze_stats time consumption. [doris]
Jibing-Li commented on PR #37895: URL: https://github.com/apache/doris/pull/37895#issuecomment-2230186018 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](column nullable) Optimization for ColumnNullable serialization [doris]
doris-robot commented on PR #37896: URL: https://github.com/apache/doris/pull/37896#issuecomment-2230186065 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](stacktrace) Add conf `enable_stacktrace` [doris]
github-actions[bot] commented on PR #37713: URL: https://github.com/apache/doris/pull/37713#issuecomment-2230186055 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Feature] Separate sql parser to a independent module [doris]
codegit10001 commented on issue #9671: URL: https://github.com/apache/doris/issues/9671#issuecomment-2230186222 啥时候能支持，很期待这个功能 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](function) the bucket number arg of width_bucket should be a positive integer value [doris]
doris-robot commented on PR #37892: URL: https://github.com/apache/doris/pull/37892#issuecomment-2230190217 TPC-H: Total hot run time: 39995 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 9349284e1f6dc60e97953a132ea0576afc5f1b0c, data reload: false -- Round 1 -- q1 18143 443843394339 q2 2798209 188 188 q3 11144 123011151115 q4 10731 772 820 772 q5 8197276727012701 q6 229 134 137 134 q7 973 596 616 596 q8 9223205120722051 q9 8779658965606560 q10 8824377037933770 q11 452 236 239 236 q12 396 214 217 214 q13 18776 297129962971 q14 278 237 248 237 q15 524 477 496 477 q16 490 387 383 383 q17 962 748 736 736 q18 8132746173837383 q19 6929136014311360 q20 699 328 310 310 q21 4927318233393182 q22 353 291 280 280 Total cold run time: 121959 ms Total hot run time: 39995 ms - Round 2, with runtime_filter_mode=off - q1 4387423742274227 q2 368 271 273 271 q3 3007278827612761 q4 1872168316301630 q5 5290529452655265 q6 222 130 127 127 q7 2140175417131713 q8 3207330533033303 q9 8363835682968296 q10 3858369237303692 q11 585 498 483 483 q12 754 588 598 588 q13 17491 297930012979 q14 306 274 285 274 q15 519 473 473 473 q16 474 412 429 412 q17 1773147414901474 q18 7688754274237423 q19 1700153416431534 q20 2012177517711771 q21 4814478946604660 q22 578 508 500 500 Total cold run time: 71408 ms Total hot run time: 53856 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve]Schema change parses ddl sql using jsqlparser framework [doris-flink-connector]
JNSimba commented on code in PR #422:
URL:
https://github.com/apache/doris-flink-connector/pull/422#discussion_r1678878401
##
flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/serializer/jsondebezium/JsonDebeziumSchemaChange.java:
##
@@ -120,6 +137,57 @@ protected JsonNode extractHistoryRecord(JsonNode record)
throws JsonProcessingEx
return record;
}
+/** Parse event type. */
+protected EventType extractEventType(JsonNode record) throws
JsonProcessingException {
+JsonNode tableChange = extractTableChange(record);
+if (tableChange == null || tableChange.get("type") == null) {
+return null;
+}
+String type = tableChange.get("type").asText();
+if (EventType.ALTER.toString().equalsIgnoreCase(type)) {
+return EventType.ALTER;
+} else if (EventType.CREATE.toString().equalsIgnoreCase(type)) {
+return EventType.CREATE;
+}
+return null;
+}
+
+protected JsonNode extractTableChange(JsonNode record) throws
JsonProcessingException {
+JsonNode historyRecord = extractHistoryRecord(record);
+JsonNode tableChanges = historyRecord.get("tableChanges");
+if (Objects.nonNull(tableChanges)) {
+return tableChanges.get(0);
+}
+LOG.warn("Failed to extract tableChanges. record={}", record);
+return null;
+}
+
+protected boolean executeAlterDDLs(
+List ddlSqlList,
+JsonNode recordRoot,
+Tuple2 dorisTableTuple,
+boolean status)
+throws IOException, IllegalArgumentException {
+if (CollectionUtils.isEmpty(ddlSqlList)) {
+LOG.info("The recordRoot cannot extract ddl sql. recordRoot={}",
recordRoot);
+return false;
+}
+
+for (String ddlSql : ddlSqlList) {
+status = schemaChangeManager.execute(ddlSql, dorisTableTuple.f0);
+LOG.info("schema change status:{}, ddl: {}", status, ddlSql);
+}
+return status;
+}
+
+protected void extractSourceConnector(JsonNode record) {
+if (Objects.isNull(sourceConnector)) {
+sourceConnector =
Review Comment:
Will sourceConnector be null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](multi-catalog)fix paimon meta properties convert [doris]
morningman merged PR #37249: URL: https://github.com/apache/doris/pull/37249 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve]Schema change parses ddl sql using jsqlparser framework [doris-flink-connector]
JNSimba commented on code in PR #422:
URL:
https://github.com/apache/doris-flink-connector/pull/422#discussion_r1678879907
##
flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/serializer/jsondebezium/JsonDebeziumSchemaChangeImplV2.java:
##
@@ -137,50 +134,33 @@ public boolean schemaChange(JsonNode recordRoot) {
LOG.info("create table ddl status: {}", status);
}
} else if (eventType.equals(EventType.ALTER)) {
-// db,table
-Tuple2 tuple = getDorisTableTuple(recordRoot);
-if (tuple == null) {
+Tuple2 dorisTableTuple =
getDorisTableTuple(recordRoot);
+if (dorisTableTuple == null) {
+LOG.warn("Failed to get doris table tuple. record={}",
recordRoot);
return false;
}
List ddlSqlList = extractDDLList(recordRoot);
-if (CollectionUtils.isEmpty(ddlSqlList)) {
-LOG.info("ddl can not do schema change:{}", recordRoot);
-return false;
-}
-List ddlSchemas =
SchemaChangeHelper.getDdlSchemas();
-for (int i = 0; i < ddlSqlList.size(); i++) {
-DDLSchema ddlSchema = ddlSchemas.get(i);
-String ddlSql = ddlSqlList.get(i);
-boolean doSchemaChange = checkSchemaChange(tuple.f0,
tuple.f1, ddlSchema);
-status = doSchemaChange &&
schemaChangeManager.execute(ddlSql, tuple.f0);
-LOG.info("schema change status:{}, ddl:{}", status,
ddlSql);
-}
-} else {
-LOG.info("Unsupported event type {}", eventType);
Review Comment:
It seems that this branch is missing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (a82aed8ab12 -> 358765c8e36)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from a82aed8ab12 [fix](arrow-flight-sql) Fix get Schema failed when `enable_parallel_result_sink` is false (#37779) add 358765c8e36 [fix](multi-catalog)fix paimon meta properties convert (#37249) No new revisions were added by this update. Summary of changes: .../datasource/paimon/PaimonExternalCatalog.java | 7 +- .../paimon/PaimonFileExternalCatalog.java | 4 +- .../paimon/PaimonHMSExternalCatalog.java | 4 +- .../datasource/property/PropertyConverter.java | 8 +- .../datasource/property/S3ClientBEProperties.java | 3 +- .../property/constants/S3Properties.java | 12 ++- .../datasource/property/PropertyConverterTest.java | 101 + .../external_table_p0/paimon/test_paimon_minio.out | 12 +++ .../paimon/test_paimon_minio.groovy| 66 ++ 9 files changed, 204 insertions(+), 13 deletions(-) create mode 100644 regression-test/data/external_table_p0/paimon/test_paimon_minio.out create mode 100644 regression-test/suites/external_table_p0/paimon/test_paimon_minio.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](stacktrace) Add conf `enable_stacktrace` [doris]
jacktengg commented on code in PR #37713:
URL: https://github.com/apache/doris/pull/37713#discussion_r1678880940
##
be/src/vec/common/allocator.cpp:
##
@@ -64,9 +64,9 @@ void Allocator::sys_memory_check(size_t
[[maybe_unused]] auto stack_trace_st =
doris::Status::Error(
injection_err_msg);
-#ifndef ENABLE_STACKTRACE
-LOG(INFO) << stack_trace_st.to_string();
-#endif
+if (!doris::config::enable_stacktrace) {
Review Comment:
should be `if (doris::config::enable_stacktrace)` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](backup) Fix the db name of the restored view [doris]
doris-robot commented on PR #37412: URL: https://github.com/apache/doris/pull/37412#issuecomment-2230194695 TPC-DS: Total hot run time: 174076 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit f90c857298d66d3f3e25166911a2362961889c43, data reload: false query1 918 365 360 360 query2 6388192218451845 query3 6644210 218 210 query4 27967 17422 17517 17422 query5 3705473 491 473 query6 274 177 170 170 query7 4586295 290 290 query8 254 199 192 192 query9 8627247024572457 query10 424 271 261 261 query11 10647 10075 10143 10075 query12 111 85 83 83 query13 1640352 356 352 query14 9263755474877487 query15 225 173 169 169 query16 7815313 318 313 query17 1660524 525 524 query18 1947274 269 269 query19 192 148 144 144 query20 85 81 82 81 query21 205 133 121 121 query22 4368431540424042 query23 33967 33717 34112 33717 query24 9960294329022902 query25 583 390 382 382 query26 704 159 157 157 query27 2190281 282 281 query28 6159209720852085 query29 884 649 643 643 query30 258 149 151 149 query31 992 771 761 761 query32 103 55 53 53 query33 651 296 282 282 query34 889 487 511 487 query35 672 578 571 571 query36 1134990 996 990 query37 151 86 84 84 query38 3020288028552855 query39 895 869 843 843 query40 219 127 123 123 query41 50 46 49 46 query42 112 104 103 103 query43 517 483 478 478 query44 1090743 725 725 query45 202 168 160 160 query46 1089750 713 713 query47 1866179818021798 query48 388 291 295 291 query49 843 416 417 416 query50 781 393 402 393 query51 6867684267676767 query52 100 96 93 93 query53 357 285 290 285 query54 893 450 450 450 query55 74 75 73 73 query56 301 278 277 277 query57 1120107710301030 query58 260 264 264 264 query59 2883276227982762 query60 321 292 296 292 query61 115 112 111 111 query62 772 660 635 635 query63 318 282 288 282 query64 9198228517611761 query65 3203311231023102 query66 698 330 339 330 query67 15527 14975 14884 14884 query68 6024541 544 541 query69 730 443 356 356 query70 1211112411601124 query71 445 279 274 274 query72 8671607152485248 query73 790 324 327 324 query74 6175571256295629 query75 4362265326972653 query76 3587956 965 956 query77 755 304 296 296 query78 11369 12084 93219321 query79 9487531 519 519 query80 1559470 467 467 query81 592 220 217 217 query82 294 129 130 129 query83 343 163 157 157 query84 276 84 87 84 query85 729 309 289 289 query86 429 303 271 271 query87 3429313931063106 query88 5660252924752475 query89 458 380 379 379 query90 2457187 185 185 query91 125 100 99 99 query92 62 49 48 48 query93 1491498 491 491 query94 1495210 206 206 query95 400 308 309 308 query96 607 283 275 275 query97 3192299529902990 query98 217 201 200 200 query99 1525127012331233 Total cold run time: 290267 ms Total hot run time: 174076 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th
Re: [PR] [fix](multicast) should not ignore Status of block::merge #35886 [doris]
doris-robot commented on PR #37869: URL: https://github.com/apache/doris/pull/37869#issuecomment-2230195035 TeamCity be ut coverage result: Function Coverage: 36.45% (9240/25351) Line Coverage: 28.00% (75546/269798) Region Coverage: 26.83% (38841/144784) Branch Coverage: 23.57% (19720/83680) Coverage Report: http://coverage.selectdb-in.cc/coverage/d31be80432cb302f28fba59951375c5f4684ec95_d31be80432cb302f28fba59951375c5f4684ec95/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](column nullable) Optimization for ColumnNullable serialization [doris]
github-actions[bot] commented on PR #37896: URL: https://github.com/apache/doris/pull/37896#issuecomment-2230195303 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](stacktrace) Add conf `enable_stacktrace` [doris]
yiguolei commented on code in PR #37713:
URL: https://github.com/apache/doris/pull/37713#discussion_r1678882813
##
be/src/vec/common/allocator.cpp:
##
@@ -64,9 +64,9 @@ void Allocator::sys_memory_check(size_t
[[maybe_unused]] auto stack_trace_st =
doris::Status::Error(
injection_err_msg);
-#ifndef ENABLE_STACKTRACE
-LOG(INFO) << stack_trace_st.to_string();
-#endif
+if (!doris::config::enable_stacktrace) {
Review Comment:
YES
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve]Schema change parses ddl sql using jsqlparser framework [doris-flink-connector]
JNSimba commented on code in PR #422:
URL:
https://github.com/apache/doris-flink-connector/pull/422#discussion_r1678883170
##
flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/serializer/jsondebezium/SQLParserService.java:
##
@@ -0,0 +1,93 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.flink.sink.writer.serializer.jsondebezium;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.util.StringUtils;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.doris.flink.sink.schema.SQLParserManager;
+import org.apache.doris.flink.sink.schema.SchemaChangeManager;
+import org.apache.doris.flink.sink.writer.EventType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+public class SQLParserService extends JsonDebeziumSchemaChange {
+private static final Logger LOG =
LoggerFactory.getLogger(SQLParserService.class);
+private final SQLParserManager sqlParserManager;
+
+public SQLParserService(JsonDebeziumChangeContext changeContext) {
+this.changeContext = changeContext;
+this.dorisOptions = changeContext.getDorisOptions();
+this.schemaChangeManager = new SchemaChangeManager(dorisOptions);
+this.sqlParserManager = new SQLParserManager();
+this.tableMapping = changeContext.getTableMapping();
+this.objectMapper = changeContext.getObjectMapper();
+}
+
+@Override
+public void init(JsonNode recordRoot, String dorisTableName) {
+// do nothing
+}
+
+@Override
+public boolean schemaChange(JsonNode recordRoot) {
+boolean status = false;
+try {
+if (!StringUtils.isNullOrWhitespaceOnly(sourceTableName) &&
!checkTable(recordRoot)) {
+return false;
+}
+
+EventType eventType = extractEventType(recordRoot);
+if (eventType == null) {
+LOG.warn("Failed to parse eventType. recordRoot={}",
recordRoot);
+return false;
+}
+
+if (eventType.equals(EventType.CREATE)) {
+// TODO support auto create table
+LOG.warn("Not auto support create table. recordRoot={}",
recordRoot);
+} else if (eventType.equals(EventType.ALTER)) {
+Tuple2 dorisTableTuple =
getDorisTableTuple(recordRoot);
+if (dorisTableTuple == null) {
+LOG.warn("Failed to get doris table tuple. record={}",
recordRoot);
+return false;
+}
+List ddlList = tryParserAlterDDLs(recordRoot);
+status = executeAlterDDLs(ddlList, recordRoot,
dorisTableTuple, status);
+}
Review Comment:
Missing an else?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](Nereids) support no-key hint parameter [doris]
morrySnow commented on PR #37720: URL: https://github.com/apache/doris/pull/37720#issuecomment-2230199270 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](build index)Remove index_meta in tablet schema when the index is dropped. (#37646) [doris]
qidaye opened a new pull request, #37897: URL: https://github.com/apache/doris/pull/37897 ## Proposed changes backport #37646 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve]Schema change parses ddl sql using jsqlparser framework [doris-flink-connector]
JNSimba commented on code in PR #422:
URL:
https://github.com/apache/doris-flink-connector/pull/422#discussion_r1678887621
##
flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/serializer/jsondebezium/SQLParserService.java:
##
@@ -0,0 +1,93 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.flink.sink.writer.serializer.jsondebezium;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.util.StringUtils;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.doris.flink.sink.schema.SQLParserManager;
+import org.apache.doris.flink.sink.schema.SchemaChangeManager;
+import org.apache.doris.flink.sink.writer.EventType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+public class SQLParserService extends JsonDebeziumSchemaChange {
Review Comment:
It is best to use a unified naming style, similar to SQLParserSchemaChange
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) tablet prune wrong when decimal value scale is nagtive [doris]
doris-robot commented on PR #37889: URL: https://github.com/apache/doris/pull/37889#issuecomment-2230201739 TPC-H: Total hot run time: 39744 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 0cd67e7afca414ca0628570e28d5c7f97f1be72c, data reload: false -- Round 1 -- q1 17637 429242934292 q2 2022194 186 186 q3 10438 121411051105 q4 10183 781 832 781 q5 7549268026442644 q6 224 140 140 140 q7 958 602 596 596 q8 9214207020942070 q9 8791655465316531 q10 8762380238003800 q11 454 234 234 234 q12 453 229 232 229 q13 1 299729692969 q14 279 232 233 232 q15 520 492 497 492 q16 487 397 376 376 q17 975 703 673 673 q18 7952745573067306 q19 7099139613001300 q20 707 331 332 331 q21 4884317532643175 q22 349 282 288 282 Total cold run time: 117714 ms Total hot run time: 39744 ms - Round 2, with runtime_filter_mode=off - q1 4363423842294229 q2 372 277 274 274 q3 2998283829252838 q4 2029170217501702 q5 5614553154925492 q6 227 138 129 129 q7 2192186118211821 q8 3244340234303402 q9 8706882188518821 q10 4153381438213814 q11 607 508 502 502 q12 801 626 613 613 q13 15915 314731783147 q14 311 274 305 274 q15 529 484 481 481 q16 483 432 428 428 q17 1801151315211513 q18 8191789978397839 q19 1797151615101510 q20 2118185818921858 q21 5130503447164716 q22 582 547 519 519 Total cold run time: 72163 ms Total hot run time: 55922 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](build index)Remove index_meta in tablet schema when the index is dropped. (#37646) [doris]
doris-robot commented on PR #37897: URL: https://github.com/apache/doris/pull/37897#issuecomment-2230202391 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](build index)Remove index_meta in tablet schema when the index is dropped. (#37646) [doris]
qidaye commented on PR #37897: URL: https://github.com/apache/doris/pull/37897#issuecomment-2230202991 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](stacktrace) Add conf `enable_stacktrace` [doris]
xinyiZzz commented on code in PR #37713:
URL: https://github.com/apache/doris/pull/37713#discussion_r167776
##
be/src/vec/common/allocator.cpp:
##
@@ -64,9 +64,9 @@ void Allocator::sys_memory_check(size_t
[[maybe_unused]] auto stack_trace_st =
doris::Status::Error(
injection_err_msg);
-#ifndef ENABLE_STACKTRACE
-LOG(INFO) << stack_trace_st.to_string();
-#endif
+if (!doris::config::enable_stacktrace) {
Review Comment:
No, it is `if (!doris::config::enable_stacktrace)`
u can see before it was `#ifndef` `ENABLE_STACKTRACE`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](backup) Fix the db name of the restored view [doris]
doris-robot commented on PR #37412: URL: https://github.com/apache/doris/pull/37412#issuecomment-2230203771 ClickBench: Total hot run time: 30.77 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit f90c857298d66d3f3e25166911a2362961889c43, data reload: false query1 0.040.040.03 query2 0.080.040.04 query3 0.220.050.05 query4 1.680.080.08 query5 0.490.480.49 query6 1.130.730.72 query7 0.020.020.02 query8 0.060.040.04 query9 0.540.490.49 query10 0.540.530.54 query11 0.150.110.10 query12 0.150.120.13 query13 0.590.580.59 query14 0.750.780.77 query15 0.840.810.81 query16 0.350.360.35 query17 0.971.031.00 query18 0.230.220.21 query19 1.921.791.80 query20 0.020.010.01 query21 15.39 0.750.66 query22 4.366.591.91 query23 18.31 1.401.29 query24 2.040.250.24 query25 0.140.080.09 query26 0.290.200.20 query27 0.450.230.22 query28 13.23 1.010.99 query29 12.57 3.383.38 query30 0.250.060.06 query31 2.860.380.40 query32 3.270.470.48 query33 2.952.932.94 query34 17.17 4.404.31 query35 4.504.404.42 query36 0.650.470.46 query37 0.180.160.15 query38 0.160.150.15 query39 0.040.040.03 query40 0.140.110.12 query41 0.090.050.05 query42 0.060.050.05 query43 0.040.030.04 Total cold run time: 109.91 s Total hot run time: 30.77 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve]Schema change parses ddl sql using jsqlparser framework [doris-flink-connector]
DongLiang-0 commented on code in PR #422:
URL:
https://github.com/apache/doris-flink-connector/pull/422#discussion_r1678889671
##
flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/serializer/jsondebezium/JsonDebeziumSchemaChangeImplV2.java:
##
@@ -137,50 +134,33 @@ public boolean schemaChange(JsonNode recordRoot) {
LOG.info("create table ddl status: {}", status);
}
} else if (eventType.equals(EventType.ALTER)) {
-// db,table
-Tuple2 tuple = getDorisTableTuple(recordRoot);
-if (tuple == null) {
+Tuple2 dorisTableTuple =
getDorisTableTuple(recordRoot);
+if (dorisTableTuple == null) {
+LOG.warn("Failed to get doris table tuple. record={}",
recordRoot);
return false;
}
List ddlSqlList = extractDDLList(recordRoot);
-if (CollectionUtils.isEmpty(ddlSqlList)) {
-LOG.info("ddl can not do schema change:{}", recordRoot);
-return false;
-}
-List ddlSchemas =
SchemaChangeHelper.getDdlSchemas();
-for (int i = 0; i < ddlSqlList.size(); i++) {
-DDLSchema ddlSchema = ddlSchemas.get(i);
-String ddlSql = ddlSqlList.get(i);
-boolean doSchemaChange = checkSchemaChange(tuple.f0,
tuple.f1, ddlSchema);
-status = doSchemaChange &&
schemaChangeManager.execute(ddlSql, tuple.f0);
-LOG.info("schema change status:{}, ddl:{}", status,
ddlSql);
-}
-} else {
-LOG.info("Unsupported event type {}", eventType);
Review Comment:
`extractEventType(JsonNode record)`only returns three types of events,
namely `ALTER`, `CREATE`, and `null`. where `null` has been handled here
https://github.com/user-attachments/assets/088569cb-8d01-40b4-a255-ae05c30a2b25";>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](tools) remove redudant copies of tpch queries in tools [doris]
englefly commented on PR #37887: URL: https://github.com/apache/doris/pull/37887#issuecomment-2230204992 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [case](iceberg)move rest cases from p2 to p0 [doris]
doris-robot commented on PR #37898: URL: https://github.com/apache/doris/pull/37898#issuecomment-2230207017 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (358765c8e36 -> cd6f5684588)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 358765c8e36 [fix](multi-catalog)fix paimon meta properties convert (#37249) add cd6f5684588 [fix](delete) Incorrect precision detection for the decimal type in condition. (#37801) No new revisions were added by this update. Summary of changes: be/src/olap/utils.cpp | 13 - regression-test/data/delete_p0/test_delete.out | 3 ++- regression-test/suites/delete_p0/test_delete.groovy | 16 +--- 3 files changed, 23 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](Vault) Maintain map in FE master [doris]
doris-robot commented on PR #37899: URL: https://github.com/apache/doris/pull/37899#issuecomment-2230208599 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [case](iceberg)move rest cases from p2 to p0 [doris]
wuwenchi commented on PR #37898: URL: https://github.com/apache/doris/pull/37898#issuecomment-2230208157 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](delete) Incorrect precision detection for the decimal type in condition. [doris]
yiguolei merged PR #37801: URL: https://github.com/apache/doris/pull/37801 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [enhance](Vault) Maintain map in FE master [doris]
ByteYue opened a new pull request, #37899: URL: https://github.com/apache/doris/pull/37899 ## Proposed changes All the vault related sql stmt is processed in FE master, so FE master could serve as one cache for vault related map. For the create table stmt with vault name, it can try to get the vault id from the map first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](function) the bucket number arg of width_bucket should be a positive integer value [doris]
doris-robot commented on PR #37892: URL: https://github.com/apache/doris/pull/37892#issuecomment-2230208403 TPC-DS: Total hot run time: 173521 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 9349284e1f6dc60e97953a132ea0576afc5f1b0c, data reload: false query1 910 367 368 367 query2 6473187418301830 query3 206 224 206 query4 22052 17742 17446 17446 query5 4190485 466 466 query6 260 169 165 165 query7 4583285 285 285 query8 246 198 200 198 query9 8588248124662466 query10 434 286 272 272 query11 10643 10114 10150 10114 query12 137 84 83 83 query13 1646371 371 371 query14 10350 776076447644 query15 225 168 171 168 query16 7824327 332 327 query17 1782574 549 549 query18 1976285 287 285 query19 202 156 154 154 query20 92 82 82 82 query21 211 130 127 127 query22 4571419041824182 query23 33898 33184 33296 33184 query24 11077 282828752828 query25 671 401 396 396 query26 1447153 151 151 query27 2937277 274 274 query28 7282205320412041 query29 987 637 624 624 query30 276 151 152 151 query31 969 759 763 759 query32 97 55 56 55 query33 786 317 311 311 query34 912 498 490 490 query35 695 602 598 598 query36 1121936 947 936 query37 151 83 84 83 query38 2856279427872787 query39 865 842 846 842 query40 276 124 122 122 query41 47 46 46 46 query42 125 100 103 100 query43 536 473 477 473 query44 1185745 727 727 query45 194 165 167 165 query46 1109752 725 725 query47 1847179417921792 query48 367 294 308 294 query49 1107504 405 405 query50 786 396 394 394 query51 7029679667176717 query52 104 92 95 92 query53 357 283 288 283 query54 1064442 450 442 query55 75 75 76 75 query56 277 272 271 271 query57 1162104510851045 query58 275 234 255 234 query59 2861266225252525 query60 296 270 276 270 query61 94 91 101 91 query62 856 655 675 655 query63 319 288 289 288 query64 10634 220616701670 query65 3209314631013101 query66 1351353 353 353 query67 15805 15022 15122 15022 query68 5380545 537 537 query69 679 430 343 343 query70 1160110810861086 query71 464 283 276 276 query72 8183530351535153 query73 782 333 322 322 query74 6157567556685668 query75 4282271526922692 query76 3859913 963 913 query77 697 301 291 291 query78 12732 964291699169 query79 8127526 526 526 query80 981 485 479 479 query81 577 221 220 220 query82 494 141 132 132 query83 340 171 169 169 query84 271 85 91 85 query85 706 310 341 310 query86 483 306 317 306 query87 3373318431713171 query88 4791250725032503 query89 488 389 373 373 query90 1857191 196 191 query91 127 100 103 100 query92 68 51 50 50 query93 3246497 495 495 query94 1123221 219 219 query95 414 322 317 317 query96 607 294 279 279 query97 3239301630343016 query98 225 197 191 191 query99 1541128112651265 Total cold run time: 291888 ms Total hot run time: 173521 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th
Re: [PR] [chore](third) Upgrade curl 7.79.0 -> 8.2.1 [doris]
doris-robot commented on PR #37893: URL: https://github.com/apache/doris/pull/37893#issuecomment-2230209403 TPC-H: Total hot run time: 5 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 41ad3872e678835090d95927c0f6509b0dae5caf, data reload: false -- Round 1 -- q1 17690 441843434343 q2 2082170 150 150 q3 10318 193619201920 q4 10105 122413151224 q5 8524395039723950 q6 229 152 124 124 q7 2050162516221622 q8 9528275427142714 q9 14260 10406 10631 10406 q10 8620352735213521 q11 405 251 252 251 q12 476 301 303 301 q13 18357 397340393973 q14 355 324 341 324 q15 514 458 461 458 q16 658 576 578 576 q17 1136946 924 924 q18 7294684068956840 q19 1748162615941594 q20 537 308 320 308 q21 4405413040294029 q22 523 454 448 448 Total cold run time: 119814 ms Total hot run time: 5 ms - Round 2, with runtime_filter_mode=off - q1 4312432743054305 q2 324 223 219 219 q3 4202418641374137 q4 2760272427582724 q5 7308714871127112 q6 239 121 123 121 q7 3293284728772847 q8 4405449644924492 q9 17451 17205 17039 17039 q10 4273429042334233 q11 769 695 692 692 q12 1019843 869 843 q13 6875374937463746 q14 442 421 413 413 q15 524 459 464 459 q16 758 685 669 669 q17 3881383938653839 q18 8818876189338761 q19 1784169516731673 q20 2398210221242102 q21 8487858384678467 q22 1056972 1026972 Total cold run time: 85378 ms Total hot run time: 79865 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](vault) Support alter storage vault stmt in FE [doris]
platoneko commented on code in PR #37606:
URL: https://github.com/apache/doris/pull/37606#discussion_r1678894191
##
cloud/src/meta-service/meta_service_resource.cpp:
##
@@ -509,56 +510,182 @@ static void set_default_vault_log_helper(const
InstanceInfoPB& instance,
LOG(INFO) << vault_msg;
}
-void MetaServiceImpl::alter_obj_store_info(google::protobuf::RpcController*
controller,
- const AlterObjStoreInfoRequest*
request,
- AlterObjStoreInfoResponse* response,
- ::google::protobuf::Closure* done) {
+static int alter_s3_storage_vault(InstanceInfoPB& instance,
std::unique_ptr txn,
+ const StorageVaultPB& vault,
MetaServiceCode& code,
+ std::string& msg) {
+if (!vault.has_obj_info()) {
+code = MetaServiceCode::INVALID_ARGUMENT;
+std::stringstream ss;
+ss << "Only s3 vault can be altered";
+msg = ss.str();
+return -1;
+}
+const auto& obj_info = vault.obj_info();
+if (obj_info.has_bucket() || obj_info.has_endpoint() ||
obj_info.has_prefix() ||
+obj_info.has_provider()) {
+code = MetaServiceCode::INVALID_ARGUMENT;
+std::stringstream ss;
+ss << "Only ak, sk can be altered";
+msg = ss.str();
+return -1;
+}
+const auto& name = vault.name();
+auto name_itr = std::find_if(instance.storage_vault_names().begin(),
+ instance.storage_vault_names().end(),
+ [&](const auto& vault_name) { return name ==
vault_name; });
+if (name_itr == instance.storage_vault_names().end()) {
+code = MetaServiceCode::INVALID_ARGUMENT;
+std::stringstream ss;
+ss << "invalid storage vault name, name =" << name;
+msg = ss.str();
+return -1;
+}
+auto pos = name_itr - instance.storage_vault_names().begin();
+auto id_itr = instance.resource_ids().begin() + pos;
+auto vault_key = storage_vault_key({instance.instance_id(), *id_itr});
+std::string val;
+
+auto err = txn->get(vault_key, &val);
+LOG(INFO) << "get instance_key=" << hex(vault_key);
+
+if (err != TxnErrorCode::TXN_OK) {
+code = cast_as(err);
+std::stringstream ss;
+ss << "failed to get storage vault, vault_id=" << *name_itr << ",
vault_name="
+ << "" << name << " err=" << err;
+msg = ss.str();
+return -1;
+}
+StorageVaultPB alter;
+alter.ParseFromString(val);
+if (obj_info.has_ak()) {
+alter.mutable_obj_info()->set_ak(obj_info.ak());
+}
+if (obj_info.has_sk()) {
+alter.mutable_obj_info()->set_ak(obj_info.sk());
+}
+
+val = alter.SerializeAsString();
+if (val.empty()) {
+msg = "failed to serialize";
+code = MetaServiceCode::PROTOBUF_SERIALIZE_ERR;
+return -1;
+}
+
+txn->put(vault_key, val);
+LOG(INFO) << "put vault_id=" << *id_itr << " instance_key=" <<
hex(vault_key);
+err = txn->commit();
+if (err != TxnErrorCode::TXN_OK) {
+code = cast_as(err);
+msg = fmt::format("failed to commit kv txn, err={}", err);
+LOG(WARNING) << msg;
+}
+
+return 0;
+}
+
+static int extract_object_storage_info(
+const AlterObjStoreInfoRequest* request, MetaServiceCode& code,
std::string& msg,
+std::tuple
[PR] [Refactor](ShortCircuit) Conjuncts outof key columns's order should b… [doris]
eldenmoon opened a new pull request, #37900: URL: https://github.com/apache/doris/pull/37900 …e handled ## Proposed changes Issue Number: close #xxx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Refactor](ShortCircuit) Conjuncts outof key columns's order should b… [doris]
doris-robot commented on PR #37900: URL: https://github.com/apache/doris/pull/37900#issuecomment-2230211528 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Refactor](ShortCircuit) Conjuncts outof key columns's order should b… [doris]
eldenmoon commented on PR #37900: URL: https://github.com/apache/doris/pull/37900#issuecomment-2230211759 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](build index)Remove index_meta in tablet schema when the index is dropped. (#37646) [doris]
github-actions[bot] commented on PR #37897: URL: https://github.com/apache/doris/pull/37897#issuecomment-2230211815 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](third) Upgrade curl 7.79.0 -> 8.2.1 [doris]
doris-robot commented on PR #37893: URL: https://github.com/apache/doris/pull/37893#issuecomment-2230214154 TeamCity be ut coverage result: Function Coverage: 37.81% (8117/21466) Line Coverage: 29.49% (66541/225670) Region Coverage: 28.95% (34289/118428) Branch Coverage: 24.83% (17617/70940) Coverage Report: http://coverage.selectdb-in.cc/coverage/41ad3872e678835090d95927c0f6509b0dae5caf_41ad3872e678835090d95927c0f6509b0dae5caf/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](test)fix unstable test case [doris]
starocean999 opened a new pull request, #37901: URL: https://github.com/apache/doris/pull/37901 ## Proposed changes Issue Number: close #xxx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [spill](regression) add regression test cases for spill [doris]
doris-robot commented on PR #37319: URL: https://github.com/apache/doris/pull/37319#issuecomment-2230215798 TPC-H: Total hot run time: 40330 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 982fad2f60ed4b640245377e15284b147b2b6b34, data reload: false -- Round 1 -- q1 17820 460044684468 q2 2348193 190 190 q3 10448 124911401140 q4 10192 778 897 778 q5 7600275326672667 q6 221 138 139 138 q7 960 600 592 592 q8 9228211721342117 q9 8991664566286628 q10 8751379038013790 q11 457 248 243 243 q12 400 224 227 224 q13 18824 298630382986 q14 282 245 231 231 q15 538 499 501 499 q16 490 392 388 388 q17 989 792 680 680 q18 8313752174537453 q19 7812146313641364 q20 682 325 333 325 q21 4969314333043143 q22 365 290 286 286 Total cold run time: 120680 ms Total hot run time: 40330 ms - Round 2, with runtime_filter_mode=off - q1 4455433242904290 q2 375 283 276 276 q3 3018275728022757 q4 1916164316531643 q5 5354535854005358 q6 223 133 134 133 q7 2158174617501746 q8 3227340633533353 q9 8451843484038403 q10 3998375636603660 q11 580 501 515 501 q12 785 597 610 597 q13 16255 297829892978 q14 323 273 293 273 q15 520 489 471 471 q16 478 428 436 428 q17 1807152714821482 q18 7701759073737373 q19 2816167015981598 q20 2029181917691769 q21 4898481447524752 q22 614 502 502 502 Total cold run time: 71981 ms Total hot run time: 54343 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](stacktrace) Add conf `enable_stacktrace` [doris]
yiguolei merged PR #37713: URL: https://github.com/apache/doris/pull/37713 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (cd6f5684588 -> a741cd308b2)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from cd6f5684588 [fix](delete) Incorrect precision detection for the decimal type in condition. (#37801) add a741cd308b2 [fix](stacktrace) Add conf `enable_stacktrace` (#37713) No new revisions were added by this update. Summary of changes: be/CMakeLists.txt | 4 be/src/common/config.cpp | 2 ++ be/src/common/config.h | 3 +++ be/src/common/status.h | 28 ++-- be/src/util/stack_util.cpp | 6 +++--- be/src/vec/common/allocator.cpp| 6 +++--- be/test/testutil/run_all_tests.cpp | 5 + build.sh | 5 - 8 files changed, 26 insertions(+), 33 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](audit) Optimize the sql length in audit logs [doris]
gavinchou commented on PR #37894: URL: https://github.com/apache/doris/pull/37894#issuecomment-2230216247 Add example what the `insert into values` will look like after this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](test)fix unstable test case [doris]
doris-robot commented on PR #37901: URL: https://github.com/apache/doris/pull/37901#issuecomment-2230217043 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](function) the bucket number arg of width_bucket should be a positive integer value [doris]
doris-robot commented on PR #37892: URL: https://github.com/apache/doris/pull/37892#issuecomment-2230217509 ClickBench: Total hot run time: 30.36 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 9349284e1f6dc60e97953a132ea0576afc5f1b0c, data reload: false query1 0.040.040.04 query2 0.080.040.04 query3 0.230.060.05 query4 1.680.080.08 query5 0.510.500.48 query6 1.140.730.73 query7 0.010.010.01 query8 0.050.050.04 query9 0.540.490.50 query10 0.560.550.55 query11 0.140.110.12 query12 0.150.120.13 query13 0.590.590.58 query14 0.750.780.77 query15 0.860.810.80 query16 0.370.370.36 query17 0.961.031.02 query18 0.220.220.21 query19 1.861.781.73 query20 0.010.010.01 query21 15.40 0.770.67 query22 4.687.771.46 query23 18.27 1.351.32 query24 2.170.220.22 query25 0.160.090.08 query26 0.300.220.21 query27 0.460.240.22 query28 13.26 1.021.02 query29 12.97 3.333.29 query30 0.260.060.06 query31 2.860.380.37 query32 3.280.480.47 query33 2.962.932.93 query34 17.10 4.424.37 query35 4.394.414.47 query36 0.640.480.47 query37 0.180.160.15 query38 0.150.140.15 query39 0.040.040.03 query40 0.160.120.13 query41 0.100.040.05 query42 0.050.050.05 query43 0.050.040.04 Total cold run time: 110.64 s Total hot run time: 30.36 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](build index)Remove index_meta in tablet schema when the index is dropped. (#37646) [doris]
qidaye opened a new pull request, #37902: URL: https://github.com/apache/doris/pull/37902 ## Proposed changes branch-2.0 dose not have the bug. Only back port the test. backport #37646 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](build index)Remove index_meta in tablet schema when the index is dropped. (#37646) [doris]
doris-robot commented on PR #37902: URL: https://github.com/apache/doris/pull/37902#issuecomment-2230218753 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](build index)Remove index_meta in tablet schema when the index is dropped. (#37646) [doris]
qidaye commented on PR #37902: URL: https://github.com/apache/doris/pull/37902#issuecomment-2230219679 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) tablet prune wrong when decimal value scale is nagtive [doris]
doris-robot commented on PR #37889: URL: https://github.com/apache/doris/pull/37889#issuecomment-2230219863 TPC-DS: Total hot run time: 172727 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 0cd67e7afca414ca0628570e28d5c7f97f1be72c, data reload: false query1 913 357 365 357 query2 6798187218311831 query3 6637210 222 210 query4 28319 17438 17325 17325 query5 3648475 465 465 query6 270 170 168 168 query7 4582298 288 288 query8 252 192 207 192 query9 8771247424502450 query10 439 297 281 281 query11 10566 10003 99809980 query12 124 82 84 82 query13 1650371 365 365 query14 9301767673157315 query15 239 166 167 166 query16 7770316 305 305 query17 1817536 541 536 query18 1999284 272 272 query19 190 148 150 148 query20 88 80 77 77 query21 199 123 127 123 query22 4250422339193919 query23 34231 33405 33697 33405 query24 11040 290528722872 query25 601 401 385 385 query26 719 148 154 148 query27 2298274 285 274 query28 6673207420812074 query29 867 667 619 619 query30 249 160 156 156 query31 983 793 744 744 query32 95 53 54 53 query33 761 296 305 296 query34 1295520 509 509 query35 694 609 599 599 query36 1156998 1009998 query37 148 87 88 87 query38 3089299128462846 query39 944 865 885 865 query40 212 123 120 120 query41 49 46 43 43 query42 112 105 95 95 query43 510 482 465 465 query44 1272714 733 714 query45 195 158 168 158 query46 1098722 723 722 query47 1873175117791751 query48 370 287 296 287 query49 839 393 402 393 query50 768 398 396 396 query51 6890678166806680 query52 109 92 98 92 query53 357 284 291 284 query54 945 448 442 442 query55 74 71 72 71 query56 286 263 266 263 query57 1144104810591048 query58 233 240 252 240 query59 2837270427552704 query60 315 269 279 269 query61 95 93 96 93 query62 779 630 621 621 query63 324 286 285 285 query64 9291220916731673 query65 3148313431063106 query66 740 339 319 319 query67 15588 15202 14915 14915 query68 4460544 540 540 query69 595 422 349 349 query70 1164116511061106 query71 432 279 268 268 query72 7136531556035315 query73 749 329 325 325 query74 6050574156365636 query75 3463264627092646 query76 2895934 1008934 query77 627 299 301 299 query78 9629928991009100 query79 2171529 525 525 query80 1769463 467 463 query81 592 214 226 214 query82 762 132 132 132 query83 320 160 160 160 query84 272 84 84 84 query85 1727317 311 311 query86 458 297 294 294 query87 3269308131243081 query88 3763245124482448 query89 474 391 376 376 query90 1773192 187 187 query91 126 97 100 97 query92 57 51 46 46 query93 2234505 494 494 query94 988 214 213 213 query95 407 314 307 307 query96 607 280 274 274 query97 3263301630093009 query98 216 193 191 191 query99 1600130812691269 Total cold run time: 279166 ms Total hot run time: 172727 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s
Re: [PR] [fix](stacktrace) Add conf `enable_stacktrace` [doris]
zhiqiang- commented on code in PR #37713:
URL: https://github.com/apache/doris/pull/37713#discussion_r1678903724
##
be/src/vec/common/allocator.cpp:
##
@@ -64,9 +64,9 @@ void Allocator::sys_memory_check(size_t
[[maybe_unused]] auto stack_trace_st =
doris::Status::Error(
injection_err_msg);
-#ifndef ENABLE_STACKTRACE
-LOG(INFO) << stack_trace_st.to_string();
-#endif
+if (!doris::config::enable_stacktrace) {
Review Comment:
执行到这里说明触发了概率性AllocateFailed，为了确保这里有且仅有一个堆栈，加了一个check，如果编译代码的时候不打堆栈，那这里手动获取堆栈打印到日志里。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](audit) Optimize the sql length in audit logs [doris]
gavinchou commented on code in PR #37894:
URL: https://github.com/apache/doris/pull/37894#discussion_r1678906155
##
fe/fe-core/src/main/java/org/apache/doris/qe/AuditLogHelper.java:
##
@@ -47,15 +57,52 @@ public class AuditLogHelper {
public static void logAuditLog(ConnectContext ctx, String origStmt,
StatementBase parsedStmt,
org.apache.doris.proto.Data.PQueryStatistics statistics, boolean
printFuzzyVariables) {
try {
+origStmt = handleStmt(origStmt, parsedStmt);
logAuditLogImpl(ctx, origStmt, parsedStmt, statistics,
printFuzzyVariables);
} catch (Throwable t) {
LOG.warn("Failed to write audit log.", t);
}
}
+private static String handleStmt(String origStmt, StatementBase
parsedStmt) {
+int length = Math.min(GlobalVariable.auditPluginMaxSqlLength,
origStmt.length());
+origStmt = origStmt.replace("\n", " ").substring(0, length);
Review Comment:
\r\n
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](cloud) fix be core when using stream load record in cloud mode [doris]
sollhui opened a new pull request, #37903: URL: https://github.com/apache/doris/pull/37903 when set enable_stream_load_record=true in cloud mode, meet core: ``` F20240711 00:02:05.733184 2316 storage_engine.cpp:123] Check failed: _type == Type::LOCAL (^A vs. ^@) *** Check failure stack trace: *** @ 0x557f1becd646 google::LogMessageFatal::~LogMessageFatal() @ 0x557f128ebb9a doris::BaseStorageEngine::to_local() @ 0x557f13413005 doris::StreamLoadAction::_save_stream_load_record() @ 0x557f134125c9 doris::StreamLoadAction::handle() @ 0x557f1c6399d6 (unknown) @ 0x557f1c638ee2 (unknown) @ 0x557f1c61e626 bufferevent_run_readcb_ @ 0x557f1c639f3f (unknown) @ 0x557f1c62a679 (unknown) @ 0x557f1c626e7f event_base_loop @ 0x557f13436087 std::_Function_handler<>::_M_invoke() @ 0x557f12da2148 doris::ThreadPool::dispatch_thread() @ 0x557f12d970d1 doris::Thread::supervise_thread() @ 0x7ffb1a784ac3 (unknown) @ 0x7ffb1a815a04 clone @ (nil) (unknown) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cloud) fix be core when using stream load record in cloud mode [doris]
doris-robot commented on PR #37903: URL: https://github.com/apache/doris/pull/37903#issuecomment-2230226462 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](build index)Remove index_meta in tablet schema when the index is dropped. (#37646) [doris]
qidaye commented on PR #37902: URL: https://github.com/apache/doris/pull/37902#issuecomment-2230226733 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refine] [doris]
Mryange commented on PR #37829: URL: https://github.com/apache/doris/pull/37829#issuecomment-2230229504 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) tablet prune wrong when decimal value scale is nagtive [doris]
doris-robot commented on PR #37889: URL: https://github.com/apache/doris/pull/37889#issuecomment-2230229886 ClickBench: Total hot run time: 31.48 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 0cd67e7afca414ca0628570e28d5c7f97f1be72c, data reload: false query1 0.050.030.03 query2 0.080.040.04 query3 0.220.050.05 query4 1.680.070.07 query5 0.490.490.47 query6 1.130.730.73 query7 0.020.010.01 query8 0.050.050.05 query9 0.560.460.49 query10 0.550.550.53 query11 0.150.120.11 query12 0.140.120.13 query13 0.580.580.58 query14 0.760.780.78 query15 0.840.800.80 query16 0.380.370.35 query17 1.001.031.02 query18 0.230.220.22 query19 1.891.811.73 query20 0.010.010.01 query21 15.40 0.750.66 query22 3.896.682.89 query23 18.29 1.461.20 query24 2.090.240.21 query25 0.170.080.08 query26 0.300.210.21 query27 0.450.240.23 query28 13.24 1.020.99 query29 12.61 3.393.33 query30 0.250.050.05 query31 2.870.390.39 query32 3.270.470.47 query33 2.882.922.90 query34 17.19 4.314.39 query35 4.424.464.38 query36 0.640.460.48 query37 0.180.160.16 query38 0.150.150.14 query39 0.050.030.04 query40 0.150.120.12 query41 0.090.050.04 query42 0.060.040.05 query43 0.040.040.03 Total cold run time: 109.49 s Total hot run time: 31.48 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [test](build index)Remove index_meta in tablet schema when the index is dropped. (#37646) [doris]
github-actions[bot] commented on PR #37902: URL: https://github.com/apache/doris/pull/37902#issuecomment-2230230339 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](third) Upgrade curl 7.79.0 -> 8.2.1 [doris]
doris-robot commented on PR #37893: URL: https://github.com/apache/doris/pull/37893#issuecomment-2230231444 TPC-DS: Total hot run time: 202964 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 41ad3872e678835090d95927c0f6509b0dae5caf, data reload: false query1 929 401 415 401 query2 6526272927722729 query3 6950208 208 208 query4 19913 17940 18006 17940 query5 19737 653765546537 query6 309 223 239 223 query7 4162303 304 303 query8 381 385 385 385 query9 3159268826172617 query10 451 311 301 301 query11 11296 10812 10699 10699 query12 119 86 73 73 query13 5610709 677 677 query14 18008 13397 13605 13397 query15 367 240 250 240 query16 6458296 263 263 query17 17491449860 860 query18 2326436 419 419 query19 206 152 152 152 query20 83 81 84 81 query21 191 100 94 94 query22 5261516750235023 query23 32524 31752 31964 31752 query24 6894655064616461 query25 533 431 450 431 query26 532 162 159 159 query27 1839296 302 296 query28 6138236823362336 query29 2795281226822682 query30 241 168 165 165 query31 886 761 740 740 query32 80 66 61 61 query33 402 264 262 262 query34 846 498 480 480 query35 1126954 929 929 query36 1257106212461062 query37 87 64 61 61 query38 3111290529392905 query39 1365133513271327 query40 209 98 97 97 query41 50 54 43 43 query42 79 82 81 81 query43 665 673 681 673 query44 1131716 734 716 query45 250 231 239 231 query46 1240978 972 972 query47 1674161916991619 query48 1014732 716 716 query49 624 374 366 366 query50 865 615 651 615 query51 4765466846804668 query52 96 87 93 87 query53 454 326 322 322 query54 2663248324672467 query55 89 80 87 80 query56 249 220 218 218 query57 1109111510591059 query58 218 201 211 201 query59 4460387641843876 query60 213 217 204 204 query61 96 94 104 94 query62 812 464 444 444 query63 497 344 351 344 query64 2437157714361436 query65 3609353635483536 query66 821 377 382 377 query67 15401 15058 15231 15058 query68 10197 649 664 649 query69 576 348 366 348 query70 1694141914871419 query71 422 320 328 320 query72 6651350735303507 query73 738 324 312 312 query74 6340586458425842 query75 5369367836743674 query76 6434112111381121 query77 1098259 251 251 query78 12938 11665 11694 11665 query79 7899635 655 635 query80 1161407 408 407 query81 489 242 227 227 query82 157894 99 94 query83 185 130 129 129 query84 263 72 69 69 query85 900 338 337 337 query86 332 305 296 296 query87 3223300230773002 query88 4589229422982294 query89 387 279 291 279 query90 1918212 209 209 query91 169 137 139 137 query92 60 53 55 53 query93 6449576 576 576 query94 671 219 203 203 query95 1104107610561056 query96 641 331 324 324 query97 6593632762636263 query98 197 186 178 178 query99 2994757 1031757 Total cold run time: 314896 ms Total hot run time: 202964 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t
(doris) branch master updated (a741cd308b2 -> 90321799940)
This is an automated email from the ASF dual-hosted git repository. wangbo pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from a741cd308b2 [fix](stacktrace) Add conf `enable_stacktrace` (#37713) add 90321799940 [Fix]Add audit log event queue size limit (#37786) No new revisions were added by this update. Summary of changes: .../main/java/org/apache/doris/common/Config.java | 4 ++ .../org/apache/doris/qe/AuditEventProcessor.java | 13 +- .../WorkloadRuntimeStatusMgr.java | 47 -- 3 files changed, 49 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](test)remove redundant test cases [doris]
starocean999 commented on PR #37866: URL: https://github.com/apache/doris/pull/37866#issuecomment-2230232753 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix]Add audit log event queue size limit [doris]
wangbo merged PR #37786: URL: https://github.com/apache/doris/pull/37786 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](vault) Support alter storage vault stmt in FE [doris]
ByteYue commented on PR #37606: URL: https://github.com/apache/doris/pull/37606#issuecomment-2230232750 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [spill](regression) add regression test cases for spill [doris]
doris-robot commented on PR #37319: URL: https://github.com/apache/doris/pull/37319#issuecomment-2230236364 TPC-DS: Total hot run time: 173673 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 982fad2f60ed4b640245377e15284b147b2b6b34, data reload: false query1 919 379 362 362 query2 6457191518761876 query3 6665216 222 216 query4 28405 17461 17383 17383 query5 4213501 502 501 query6 288 184 174 174 query7 4599294 289 289 query8 243 208 194 194 query9 8327245224502450 query10 439 286 285 285 query11 11120 995799479947 query12 132 81 83 81 query13 1638363 356 356 query14 10130 760982767609 query15 211 169 171 169 query16 7091340 316 316 query17 1780605 542 542 query18 1166294 283 283 query19 204 164 157 157 query20 91 87 83 83 query21 208 131 128 128 query22 4347407740754075 query23 33705 33142 33060 33060 query24 11987 289128372837 query25 672 390 396 390 query26 1755157 151 151 query27 2644276 281 276 query28 6809203920272027 query29 975 645 643 643 query30 291 149 155 149 query31 946 774 743 743 query32 96 60 58 58 query33 786 334 313 313 query34 881 492 493 492 query35 702 601 596 596 query36 1129942 920 920 query37 230 84 87 84 query38 2928280127202720 query39 859 823 832 823 query40 291 126 124 124 query41 48 47 46 46 query42 120 101 104 101 query43 537 460 469 460 query44 1248744 731 731 query45 194 165 164 164 query46 1108744 747 744 query47 1829175317501750 query48 372 307 301 301 query49 1193454 418 418 query50 795 412 409 409 query51 6903685468486848 query52 113 98 97 97 query53 368 290 298 290 query54 1132463 464 463 query55 78 76 77 76 query56 304 280 291 280 query57 1148105610651056 query58 252 251 252 251 query59 3030285327532753 query60 320 304 305 304 query61 120 116 115 115 query62 845 670 658 658 query63 329 295 293 293 query64 10674 237816941694 query65 3301312531183118 query66 1265334 334 334 query67 15643 14943 15034 14943 query68 9650552 562 552 query69 784 477 355 355 query70 1454117011621162 query71 550 285 275 275 query72 8836578955095509 query73 2229329 330 329 query74 6279562456895624 query75 6324271326962696 query76 5506958 914 914 query77 793 311 308 308 query78 9711902789688968 query79 9221532 528 528 query80 1015491 492 491 query81 581 227 225 225 query82 286 135 135 135 query83 386 175 168 168 query84 274 88 88 88 query85 986 311 305 305 query86 367 315 309 309 query87 3347318631413141 query88 5166249724812481 query89 520 389 388 388 query90 2059198 203 198 query91 133 101 103 101 query92 67 50 51 50 query93 6209504 503 503 query94 1456224 219 219 query95 410 324 369 324 query96 631 279 274 274 query97 3145301030133010 query98 228 204 203 203 query99 1538126712611261 Total cold run time: 309769 ms Total hot run time: 173673 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th
Re: [PR] [fix](docker)Docker be register bug and add fe new interface recovery [doris]
JNSimba merged PR #37335: URL: https://github.com/apache/doris/pull/37335 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cloud) fix be core when using stream load record in cloud mode [doris]
github-actions[bot] commented on PR #37903: URL: https://github.com/apache/doris/pull/37903#issuecomment-2230236787 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](Nereids) support no-key hint parameter [doris]
doris-robot commented on PR #37720: URL: https://github.com/apache/doris/pull/37720#issuecomment-2230238213 TPC-H: Total hot run time: 40073 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 69cefc14913933431b174dfd9aa544658ebc5e73, data reload: false -- Round 1 -- q1 18084 445544124412 q2 2611197 187 187 q3 11679 120910711071 q4 10561 784 800 784 q5 7676269926942694 q6 222 139 137 137 q7 985 611 603 603 q8 9250207121042071 q9 8814661666116611 q10 8861383638203820 q11 457 242 244 242 q12 395 229 220 220 q13 17775 299729932993 q14 269 235 245 235 q15 521 486 501 486 q16 505 382 373 373 q17 950 731 674 674 q18 8097751173627362 q19 7099133212641264 q20 680 331 337 331 q21 4916322132383221 q22 349 288 282 282 Total cold run time: 120756 ms Total hot run time: 40073 ms - Round 2, with runtime_filter_mode=off - q1 4366429042904290 q2 374 277 264 264 q3 2990278127452745 q4 1906166915971597 q5 5287530353555303 q6 227 132 131 131 q7 2113168917101689 q8 3206333233053305 q9 8373838283768376 q10 3899369236443644 q11 579 499 494 494 q12 787 601 603 601 q13 16542 300229752975 q14 308 275 285 275 q15 529 475 485 475 q16 479 416 436 416 q17 1774148614861486 q18 7759765073847384 q19 1664159414791479 q20 2007178917681768 q21 4829480348094803 q22 587 475 503 475 Total cold run time: 70585 ms Total hot run time: 53975 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](docker)Docker be register bug and add fe new interface recovery (#37335)
This is an automated email from the ASF dual-hosted git repository.
diwu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new e334e407975 [fix](docker)Docker be register bug and add fe new
interface recovery (#37335)
e334e407975 is described below
commit e334e4079750461531c8237ebc79fcf2aa9e3c9d
Author: FreeOnePlus <54164178+freeonep...@users.noreply.github.com>
AuthorDate: Tue Jul 16 15:42:59 2024 +0800
[fix](docker)Docker be register bug and add fe new interface recovery
(#37335)
---
docker/runtime/be/resource/entry_point.sh | 2 +-
docker/runtime/be/resource/init_be.sh | 68 +++--
docker/runtime/fe/resource/init_fe.sh | 238 +-
3 files changed, 255 insertions(+), 53 deletions(-)
diff --git a/docker/runtime/be/resource/entry_point.sh
b/docker/runtime/be/resource/entry_point.sh
index 1ae418163fa..6e3dfaf3875 100755
--- a/docker/runtime/be/resource/entry_point.sh
+++ b/docker/runtime/be/resource/entry_point.sh
@@ -167,7 +167,7 @@ check_be_status() {
if [[ $1 == true ]]; then
docker_process_sql <<<"show frontends" | grep
"[[:space:]]${MASTER_FE_IP}[[:space:]]"
else
- docker_process_sql <<<"show backends" | grep
"[[:space:]]${CURRENT_BE_IP}[[:space:]]" | grep
"[[:space:]]${CURRENT_BE_PORT}[[:space:]]" | grep "[[:space:]]true[[:space:]]"
+ docker_process_sql <<<"show backends" | grep
"[[:space:]]${CURRENT_BE_IP}[[:space:]]" | grep
"[[:space:]]${CURRENT_BE_PORT}[[:space:]]"
fi
be_join_status=$?
if [[ "${be_join_status}" == 0 ]]; then
diff --git a/docker/runtime/be/resource/init_be.sh
b/docker/runtime/be/resource/init_be.sh
index 42afd1f6754..f9269f3ee7e 100644
--- a/docker/runtime/be/resource/init_be.sh
+++ b/docker/runtime/be/resource/init_be.sh
@@ -72,23 +72,26 @@ show_be_args(){
doris_note "CURRENT_BE_IP " ${CURRENT_BE_IP}
doris_note "CURRENT_BE_PORT " ${CURRENT_BE_PORT}
doris_note "RUN_TYPE " ${RUN_TYPE}
- doris_note "PRIORITY_NETWORKS " ${PRIORITY_NETWORKS}
}
# Execute sql script, passed via stdin
# usage: docker_process_sql sql_script
docker_process_sql() {
- set +e
- mysql -uroot -P9030 -h${MASTER_FE_IP} --comments "$@" 2>/dev/null
+set +e
+if [[ $RUN_TYPE == "ELECTION" || $RUN_TYPE == "ASSIGN" ]]; then
+mysql -uroot -P9030 -h${MASTER_FE_IP} --comments "$@" 2>/dev/null
+elif [[ $RUN_TYPE == "FQDN" ]]; then
+mysql -uroot -P9030 -h${MASTER_NODE_NAME} --comments "$@" 2>/dev/null
+fi
}
node_role_conf(){
- if [[ ${NODE_ROLE} == 'computation' ]]; then
-doris_note "this node role is computation"
-echo "be_node_role=computation" >>${DORIS_HOME}/be/conf/be.conf
- else
-doris_note "this node role is mix"
- fi
+if [[ ${NODE_ROLE} == 'computation' ]]; then
+doris_note "this node role is computation"
+echo "be_node_role=computation" >>${DORIS_HOME}/be/conf/be.conf
+else
+doris_note "this node role is mix"
+fi
}
register_be_to_fe() {
@@ -103,12 +106,25 @@ register_be_to_fe() {
fi
fi
for i in {1..300}; do
- docker_process_sql <<<"alter system add backend
'${CURRENT_BE_IP}:${CURRENT_BE_PORT}'"
+ if [[ $RUN_TYPE == "ELECTION" || $RUN_TYPE == "ASSIGN" ]]; then
+ SQL="alter system add backend '${CURRENT_BE_IP}:${CURRENT_BE_PORT}';"
+ doris_note "Executing SQL: $SQL"
+ docker_process_sql <<<"$SQL"
+ elif [[ $RUN_TYPE == "FQDN" ]]; then
+ SQL="alter system add backend
'${CURRENT_NODE_NAME}:${CURRENT_BE_PORT}';"
+ doris_note "Executing SQL: $SQL"
+ docker_process_sql <<<"$SQL"
+ fi
register_be_status=$?
if [[ $register_be_status == 0 ]]; then
doris_note "BE successfully registered to FE！"
is_fe_start=true
return
+ else
+check_be_status
+if [[ $IS_BE_JOIN_STATUS == "true" ]]; then
+ return
+fi
fi
if [[ $(( $i % 20 )) == 1 ]]; then
doris_note "Register BE to FE is failed. retry."
@@ -122,16 +138,23 @@ register_be_to_fe() {
check_be_status() {
set +e
-local is_fe_start=false
-for i in {1..300}; do
+declare -g IS_FE_START_STATUS IS_BE_JOIN_STATUS
+IS_FE_START_STATUS=false
+IS_BE_JOIN_STATUS=false
+for i in {1..100}; do
if [[ $(($i % 20)) == 1 ]]; then
doris_warn "start check be register status~"
fi
- docker_process_sql <<<"show backends;" | grep
"[[:space:]]${CURRENT_BE_IP}[[:space:]]" | grep
"[[:space:]]${CURRENT_BE_PORT}[[:space:]]"
+ if [[ $RUN_TYPE == "ELECTION" || $RUN_TYPE == "ASSIGN" ]]; then
+docker_process_sql <<<"show backends" | grep
"[[:space:]]${CURRENT_BE_IP}[[:space:]]" | grep
"[[:space:]]${CURRENT_BE_PORT}[[:space:]]"
+ elif [[ $RUN_TYPE == "FQDN" ]]; then
+docker_process_sql <<<"show backends" | grep
"[[:s
