(doris) branch master updated (44915f3513f -> b2b1cb43b45)
This is an automated email from the ASF dual-hosted git repository.
yangbowen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 44915f3513f [fix](Azure) Use robust logic to check there is no
multipart upload for Azure (#36904)
add b2b1cb43b45 [optimize](restore) update replica version after restore
(#36148)
No new revisions were added by this update.
Summary of changes:
.../java/org/apache/doris/backup/RestoreJob.java | 4 +---
.../java/org/apache/doris/catalog/Replica.java | 9
...y => test_backup_restore_version_revert.groovy} | 26 ++
3 files changed, 27 insertions(+), 12 deletions(-)
copy regression-test/suites/backup_restore/{test_backup_restore.groovy =>
test_backup_restore_version_revert.groovy} (72%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](restore) avoid NPE for restore job (#36395)
This is an automated email from the ASF dual-hosted git repository.
yangbowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 29d31a278c4 [fix](restore) avoid NPE for restore job (#36395)
29d31a278c4 is described below
commit 29d31a278c4365bd1cd7ec627c2f5ac98de95666
Author: Xujian Duan <50550370+darvend...@users.noreply.github.com>
AuthorDate: Thu Jun 27 20:15:40 2024 +0800
[fix](restore) avoid NPE for restore job (#36395)
## Proposed changes
Issue Number: close #36440
The indexes of OlapTable may not be assigned in method `readFields`:
```
@Override
public void readFields(DataInput in) throws IOException {
...
// read indexes
if (in.readBoolean()) {
this.indexes = TableIndexes.read(in);
}
...
}
```
So indexes may be null.
---
.../src/main/java/org/apache/doris/catalog/OlapTable.java | 14 --
1 file changed, 8 insertions(+), 6 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
index 490e8a7b3a5..b0103e0d68b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
@@ -743,12 +743,14 @@ public class OlapTable extends Table implements
MTMVRelatedTableIf, GsonPostProc
}
// reset the indexes and update the indexes in materialized index meta
too.
-List indexes = this.indexes.getIndexes();
-for (Index idx : indexes) {
-idx.setIndexId(env.getNextId());
-}
-for (Map.Entry entry :
indexIdToMeta.entrySet()) {
-entry.getValue().setIndexes(indexes);
+if (this.indexes != null) {
+List indexes = this.indexes.getIndexes();
+for (Index idx : indexes) {
+idx.setIndexId(env.getNextId());
+}
+for (Map.Entry entry :
indexIdToMeta.entrySet()) {
+entry.getValue().setIndexes(indexes);
+}
}
return Status.OK;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-.20 created (now ef00dad680d)
This is an automated email from the ASF dual-hosted git repository. yangbowen pushed a change to branch branch-.20 in repository https://gitbox.apache.org/repos/asf/doris.git at ef00dad680d [Fix](multi-catalog) Fix some undefined behaviors. (#38274) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-.20 deleted (was ef00dad680d)
This is an automated email from the ASF dual-hosted git repository. yangbowen pushed a change to branch branch-.20 in repository https://gitbox.apache.org/repos/asf/doris.git was ef00dad680d [Fix](multi-catalog) Fix some undefined behaviors. (#38274) The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [BugFix](TabletInvertedIndex) fix replica not found in TabletInvertedIndex (#35791)
This is an automated email from the ASF dual-hosted git repository.
yangbowen pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 993c3c376d0 [BugFix](TabletInvertedIndex) fix replica not found in
TabletInvertedIndex (#35791)
993c3c376d0 is described below
commit 993c3c376d0ca3662a1bfaf57496b32168ac3a6e
Author: shouchengShen <50309488+johnny...@users.noreply.github.com>
AuthorDate: Tue Jun 4 16:32:23 2024 +0800
[BugFix](TabletInvertedIndex) fix replica not found in TabletInvertedIndex
(#35791)
## Proposed changes
adding and dropping backend in cluster may lead to the different replica
in different backend have the same replica id,
at that time, 'delete from table partition xxx' operation will be failed
with exception:'Failed to commit txn 862, cause tablet 11653 succ
replica num 1 < quorum replica num 2. ...'
and also, with the above delete operation, drop replica
operation(class:TabletInvertedIndex) will be failed with WARN log:
' could not find tablet id for replica xxx, the tablet maybe dropped'
fix in master branch already merged: #34117
Co-authored-by: shenshoucheng
---
.../java/org/apache/doris/catalog/TabletInvertedIndex.java | 12 ++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/TabletInvertedIndex.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/TabletInvertedIndex.java
index 9f9f3fdc698..1c337740de9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/TabletInvertedIndex.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/TabletInvertedIndex.java
@@ -574,8 +574,16 @@ public class TabletInvertedIndex {
"tablet " + tabletId + " not exists, backend " +
backendId);
if (replicaMetaTable.containsRow(tabletId)) {
Replica replica = replicaMetaTable.remove(tabletId, backendId);
-replicaToTabletMap.remove(replica.getId());
-replicaMetaTable.remove(tabletId, backendId);
+if (replicaMetaTable.containsRow(tabletId)) {
+long replicaNum =
replicaMetaTable.row(tabletId).values().stream()
+.filter(c -> c.getId() == replica.getId()).count();
+if (replicaNum == 0) {
+replicaToTabletMap.remove(replica.getId());
+}
+} else {
+replicaToTabletMap.remove(replica.getId());
+}
+
backingReplicaMetaTable.remove(backendId, tabletId);
LOG.debug("delete replica {} of tablet {} in backend {}",
replica.getId(), tabletId, backendId);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (ded018689bd -> de9da3341b4)
This is an automated email from the ASF dual-hosted git repository. yangbowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from ded018689bd [fix](cloud) Fix CloudTabletRebalancer npe (#36162) add de9da3341b4 [bug](metrics) fix be metric doris_be_process_thread_num is zero (#35511) No new revisions were added by this update. Summary of changes: be/src/util/doris_metrics.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (fa4341f8526 -> a5fe4544096)
This is an automated email from the ASF dual-hosted git repository. yangbowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from fa4341f8526 [Improve](expr)first support array_contains for expr push down inverted index (#32620) add a5fe4544096 [BugFix](TabletInvertedIndex) fix replica not found in TabletInvertedIndex (#34117) No new revisions were added by this update. Summary of changes: .../org/apache/doris/catalog/TabletInvertedIndex.java | 15 +-- 1 file changed, 13 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: fix syntax error for CreateTableLikeStmt with partition properties (#34187)
This is an automated email from the ASF dual-hosted git repository.
yangbowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 958fa5c763d fix syntax error for CreateTableLikeStmt with partition
properties (#34187)
958fa5c763d is described below
commit 958fa5c763d9b96b03bac658202f65b954330b23
Author: Xujian Duan <50550370+darvend...@users.noreply.github.com>
AuthorDate: Mon Apr 29 11:03:42 2024 +0800
fix syntax error for CreateTableLikeStmt with partition properties (#34187)
fix syntax error for CreateTableLikeStmt with partition properties
---
.../apache/doris/catalog/ListPartitionInfo.java| 3 +-
.../apache/doris/catalog/RangePartitionInfo.java | 3 +-
.../suites/ddl_p0/test_create_table_like.groovy| 54 ++
3 files changed, 58 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/ListPartitionInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/ListPartitionInfo.java
index d657dc7d9d4..ce8d4303cbe 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/ListPartitionInfo.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/ListPartitionInfo.java
@@ -28,6 +28,7 @@ import org.apache.doris.analysis.SlotRef;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.util.ListUtil;
+import org.apache.doris.common.util.PropertyAnalyzer;
import com.google.common.base.Preconditions;
import com.google.common.collect.Lists;
@@ -249,7 +250,7 @@ public class ListPartitionInfo extends PartitionInfo {
Optional.ofNullable(this.idToStoragePolicy.get(entry.getKey())).ifPresent(p -> {
if (!p.equals("")) {
-sb.append("PROPERTIES (\"STORAGE POLICY\" = \"");
+sb.append(" (\"" +
PropertyAnalyzer.PROPERTIES_STORAGE_POLICY + "\" = \"");
sb.append(p).append("\")");
}
});
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/RangePartitionInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/RangePartitionInfo.java
index 9a6c5d353fd..caa77f660bd 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/RangePartitionInfo.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/RangePartitionInfo.java
@@ -25,6 +25,7 @@ import org.apache.doris.analysis.RangePartitionDesc;
import org.apache.doris.analysis.SinglePartitionDesc;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.DdlException;
+import org.apache.doris.common.util.PropertyAnalyzer;
import org.apache.doris.common.util.RangeUtils;
import com.google.common.base.Preconditions;
@@ -299,7 +300,7 @@ public class RangePartitionInfo extends PartitionInfo {
Optional.ofNullable(this.idToStoragePolicy.get(entry.getKey())).ifPresent(p -> {
if (!p.equals("")) {
-sb.append("PROPERTIES (\"STORAGE POLICY\" = \"");
+sb.append(" (\"" +
PropertyAnalyzer.PROPERTIES_STORAGE_POLICY + "\" = \"");
sb.append(p).append("\")");
}
});
diff --git a/regression-test/suites/ddl_p0/test_create_table_like.groovy
b/regression-test/suites/ddl_p0/test_create_table_like.groovy
index a1154c05a43..f4c1da02d93 100644
--- a/regression-test/suites/ddl_p0/test_create_table_like.groovy
+++ b/regression-test/suites/ddl_p0/test_create_table_like.groovy
@@ -47,4 +47,58 @@ suite("test_create_table_like") {
VALUES ("test1", 1, 123456789, 1234567891, 123456789, 1234567891,
123456789)"""
qt_select_table_like """select * from decimal_test_like"""
+
+def resource_name = "test_create_table_like_use_resource"
+def policy_name = "test_create_table_like_use_policy"
+def table1 = "create_table_partion_use_created_policy"
+def table2 = "create_table_partion_like_use_created_policy"
+
+sql """
+CREATE RESOURCE IF NOT EXISTS $resource_name
+PROPERTIES(
+"type"="s3",
+"AWS_REGION" = "bj",
+"AWS_ENDPOINT" = "bj.s3.com",
+"AWS_ROOT_PATH" = "path/to/root",
+"AWS_SECRET_KEY" = "",
+"AWS_ACCESS_KEY" = "bbba",
+"AWS_BUCKET" = "test-bucket",
+"s3_validity_check" = "false"
+);
+"""
+
(doris) branch master updated: [optimize](information_schema) optimize the data type of table information_schema.rowsets (#34739)
This is an automated email from the ASF dual-hosted git repository.
yangbowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 31db952eb98 [optimize](information_schema) optimize the data type of
table information_schema.rowsets (#34739)
31db952eb98 is described below
commit 31db952eb982648b3374c3d4d3ca2fac822aebe3
Author: Xujian Duan <50550370+darvend...@users.noreply.github.com>
AuthorDate: Tue May 21 13:28:49 2024 +0800
[optimize](information_schema) optimize the data type of table
information_schema.rowsets (#34739)
* optimize the data type of table imformation_schema.rowsets
-
Co-authored-by: duanxujian
---
be/src/exec/schema_scanner/schema_rowsets_scanner.cpp| 16 ++--
.../main/java/org/apache/doris/catalog/SchemaTable.java | 4 ++--
.../data/query_p0/system/test_query_sys_rowsets.out | 14 +++---
.../suites/query_p0/system/test_query_sys_rowsets.groovy | 9 -
4 files changed, 31 insertions(+), 12 deletions(-)
diff --git a/be/src/exec/schema_scanner/schema_rowsets_scanner.cpp
b/be/src/exec/schema_scanner/schema_rowsets_scanner.cpp
index b03a30b2d92..f66363fde1e 100644
--- a/be/src/exec/schema_scanner/schema_rowsets_scanner.cpp
+++ b/be/src/exec/schema_scanner/schema_rowsets_scanner.cpp
@@ -56,8 +56,8 @@ std::vector
SchemaRowsetsScanner::_s_tbls_columns = {
{"END_VERSION", TYPE_BIGINT, sizeof(int64_t), true},
{"INDEX_DISK_SIZE", TYPE_BIGINT, sizeof(size_t), true},
{"DATA_DISK_SIZE", TYPE_BIGINT, sizeof(size_t), true},
-{"CREATION_TIME", TYPE_BIGINT, sizeof(int64_t), true},
-{"NEWEST_WRITE_TIMESTAMP", TYPE_BIGINT, sizeof(int64_t), true},
+{"CREATION_TIME", TYPE_DATETIME, sizeof(int64_t), true},
+{"NEWEST_WRITE_TIMESTAMP", TYPE_DATETIME, sizeof(int64_t), true},
};
@@ -221,24 +221,28 @@ Status
SchemaRowsetsScanner::_fill_block_impl(vectorized::Block* block) {
}
// CREATION_TIME
{
-std::vector srcs(fill_rowsets_num);
+std::vector srcs(fill_rowsets_num);
for (int i = fill_idx_begin; i < fill_idx_end; ++i) {
RowsetSharedPtr rowset = rowsets_[i];
-srcs[i - fill_idx_begin] = rowset->creation_time();
+int64_t creation_time = rowset->creation_time();
+srcs[i - fill_idx_begin].from_unixtime(creation_time,
TimezoneUtils::default_time_zone);
datas[i - fill_idx_begin] = srcs.data() + i - fill_idx_begin;
}
RETURN_IF_ERROR(fill_dest_column_for_range(block, 10, datas));
}
// NEWEST_WRITE_TIMESTAMP
{
-std::vector srcs(fill_rowsets_num);
+std::vector srcs(fill_rowsets_num);
for (int i = fill_idx_begin; i < fill_idx_end; ++i) {
RowsetSharedPtr rowset = rowsets_[i];
-srcs[i - fill_idx_begin] = rowset->newest_write_timestamp();
+int64_t newest_write_timestamp = rowset->newest_write_timestamp();
+srcs[i - fill_idx_begin].from_unixtime(newest_write_timestamp,
+
TimezoneUtils::default_time_zone);
datas[i - fill_idx_begin] = srcs.data() + i - fill_idx_begin;
}
RETURN_IF_ERROR(fill_dest_column_for_range(block, 11, datas));
}
+
_rowsets_idx += fill_rowsets_num;
return Status::OK();
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java
index 92206dc3fdb..88349942719 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java
@@ -395,8 +395,8 @@ public class SchemaTable extends Table {
.column("END_VERSION",
ScalarType.createType(PrimitiveType.BIGINT))
.column("INDEX_DISK_SIZE",
ScalarType.createType(PrimitiveType.BIGINT))
.column("DATA_DISK_SIZE",
ScalarType.createType(PrimitiveType.BIGINT))
-.column("CREATION_TIME",
ScalarType.createType(PrimitiveType.BIGINT))
-.column("NEWEST_WRITE_TIMESTAMP",
ScalarType.createType(PrimitiveType.BIGINT))
+.column("CREATION_TIME",
ScalarType.createType(PrimitiveType.DATETIME))
+.column("NEWEST_WRITE_TIMESTAMP",
ScalarType.createType(PrimitiveType.DATETIME))
.build()))
.put("parameters", new SchemaTable(SystemIdGenerator.getNextId(),
"parameters", TableType.SCHEMA,
builder().column("SPEC
(doris) branch branch-1.2-lts updated: fix bug that replica's remote data size set to data size (#35100)
This is an automated email from the ASF dual-hosted git repository.
yangbowen pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new 1d8e763cdd5 fix bug that replica's remote data size set to data size
(#35100)
1d8e763cdd5 is described below
commit 1d8e763cdd54997163cecfe96413bb913a17f9ef
Author: GoGoWen <82132356+gogo...@users.noreply.github.com>
AuthorDate: Tue May 21 13:33:32 2024 +0800
fix bug that replica's remote data size set to data size (#35100)
fix bug that replica's remote data size set to data size
---
fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java
b/fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java
index bad60d8ec43..5c93348e96b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java
@@ -1019,7 +1019,7 @@ public class TabletSchedCtx implements
Comparable {
}
replica.updateVersionInfo(reportedTablet.getVersion(),
reportedTablet.getDataSize(),
-reportedTablet.getDataSize(),
reportedTablet.getRowCount());
+reportedTablet.getRemoteDataSize(),
reportedTablet.getRowCount());
if (reportedTablet.isSetPathHash()) {
replica.setPathHash(reportedTablet.getPathHash());
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: fix replica's remote data size set to data size (#35098)
This is an automated email from the ASF dual-hosted git repository.
yangbowen pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 0599cb2efd1 fix replica's remote data size set to data size (#35098)
0599cb2efd1 is described below
commit 0599cb2efd1a6dcc1c609db35fa0a63c99a73545
Author: GoGoWen <82132356+gogo...@users.noreply.github.com>
AuthorDate: Tue May 21 16:48:08 2024 +0800
fix replica's remote data size set to data size (#35098)
fix replica's remote data size set to data size
---
fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java
b/fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java
index 706e346c046..8401ec17bbd 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java
@@ -1178,7 +1178,7 @@ public class TabletSchedCtx implements
Comparable {
}
replica.updateVersionInfo(reportedTablet.getVersion(),
reportedTablet.getDataSize(),
-reportedTablet.getDataSize(),
reportedTablet.getRowCount());
+reportedTablet.getRemoteDataSize(),
reportedTablet.getRowCount());
if (replica.getLastFailedVersion() >
partition.getCommittedVersion()
&& reportedTablet.getVersion() >=
partition.getCommittedVersion()
//&& !(reportedTablet.isSetVersionMiss() &&
reportedTablet.isVersionMiss()
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: fix bug that replica remote data size set to data size (#35097)
This is an automated email from the ASF dual-hosted git repository.
yangbowen pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 8a16dad3289 fix bug that replica remote data size set to data size
(#35097)
8a16dad3289 is described below
commit 8a16dad3289a4c25ee3ef25c6a2fa30700552d01
Author: GoGoWen <82132356+gogo...@users.noreply.github.com>
AuthorDate: Thu May 23 14:27:44 2024 +0800
fix bug that replica remote data size set to data size (#35097)
fix bug that replica remote data size set to data size
---
fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java
b/fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java
index 912fc1bb316..3544da69e88 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java
@@ -1137,7 +1137,7 @@ public class TabletSchedCtx implements
Comparable {
}
replica.updateVersionInfo(reportedTablet.getVersion(),
reportedTablet.getDataSize(),
-reportedTablet.getDataSize(),
reportedTablet.getRowCount());
+reportedTablet.getRemoteDataSize(),
reportedTablet.getRowCount());
if (replica.getLastFailedVersion() >
partition.getCommittedVersion()
&& reportedTablet.getVersion() >=
partition.getCommittedVersion()
//&& !(reportedTablet.isSetVersionMiss() &&
reportedTablet.isVersionMiss()
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [cherrypick](schema-change) cherrypick to Reserve some memory for use by other par… (#39995)
This is an automated email from the ASF dual-hosted git repository.
yangbowen pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 683600e8b14 [cherrypick](schema-change) cherrypick to Reserve some
memory for use by other par… (#39995)
683600e8b14 is described below
commit 683600e8b1447a1785d95b3a4505a472b6baf469
Author: GoGoWen <82132356+gogo...@users.noreply.github.com>
AuthorDate: Tue Aug 27 20:35:38 2024 +0800
[cherrypick](schema-change) cherrypick to Reserve some memory for use by
other par… (#39995)
cp #27800 to fix schema change failure
Co-authored-by: Pxl
---
be/src/olap/schema_change.cpp | 6 +-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/be/src/olap/schema_change.cpp b/be/src/olap/schema_change.cpp
index 9d2b3b94a48..31d3f0a32c8 100644
--- a/be/src/olap/schema_change.cpp
+++ b/be/src/olap/schema_change.cpp
@@ -595,7 +595,11 @@ Status
VSchemaChangeWithSorting::_inner_process(RowsetReaderSharedPtr rowset_rea
}
RETURN_IF_ERROR(_changer.change_block(ref_block.get(),
new_block.get()));
-if (_mem_tracker->consumption() + new_block->allocated_bytes() >
_memory_limitation) {
+
+constexpr double HOLD_BLOCK_MEMORY_RATE =
+0.66; // Reserve some memory for use by other parts of this job
+if (_mem_tracker->consumption() + new_block->allocated_bytes() >
_memory_limitation ||
+_mem_tracker->consumption() > _memory_limitation *
HOLD_BLOCK_MEMORY_RATE) {
RETURN_IF_ERROR(create_rowset());
if (_mem_tracker->consumption() + new_block->allocated_bytes() >
_memory_limitation) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [Fix](Date)fix date cast to datetime can not pushdown (#39310)
This is an automated email from the ASF dual-hosted git repository.
yangbowen pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 009039b19b0 [Fix](Date)fix date cast to datetime can not pushdown
(#39310)
009039b19b0 is described below
commit 009039b19b06a74bbd49b838e90ea76e0dc018c4
Author: GoGoWen <82132356+gogo...@users.noreply.github.com>
AuthorDate: Tue Aug 27 20:41:43 2024 +0800
[Fix](Date)fix date cast to datetime can not pushdown (#39310)
## Proposed changes
when we upgrade from 1.2 to 2.0, following case that "date cast to
datetime" will not push down any more, which cause performance
degradation.
step to repo
step 1 (in 1.2):
CREATE TABLE `search_analysis_after_adv` ( `pin_id` BIGINT NULL COMMENT
'广告主pin_id', `date` date NULL COMMENT '点击日期', `search_date` date NULL
COMMENT '搜索时间', `pt` INT NULL COMMENT '1:点击 0:曝光', `dim_type` INT NULL
COMMENT '1:品牌 0:品牌+三级类目', `business_type` INT NULL COMMENT '业务线类型',
`campaign_type` INT NULL COMMENT '计划类型', `delivery_system_type` INT NULL
COMMENT '平台来源', `ad_plan_id` BIGINT NULL COMMENT '广告计划id', `ad_group_id`
BIGINT NULL COMMENT '广告单元id', `deal_id` BIGINT NULL COMMENT '排期id',
`branding_order_id` BIGINT NULL COMMENT '品牌广告订单id', `sku_brand_id`
BIGINT NULL COMMENT '点击sku对应的品 牌ID', `ad_sku_cid3` BIGINT NULL COMMENT
'点击sku对应的三级类目ID', `key_word` VARCHAR(512) NULL COMMENT '搜索词',
`search_num` HLL HLL_UNION NOT NULL COMMENT '搜索id' ) ENGINE=OLAP
AGGREGATE KEY(`pin_id`, `date`, `search_date`, `pt`, `dim_type`,
`business_type`, `campaign_type`, `delivery_system_type`, `ad_plan_id`,
`ad_group_id`, `deal_id`, `branding_order_id`, `sku_brand_id`,
`ad_sku_cid3`, `key_word`) COMMENT 'olap' PARTITION BY RANGE(`date`)
(PARTITION p202304 VALUES [('2023-04-01'), ('2023-05-01')), PARTITION
p202305 VALUES [('2023-05-01'), ('2023-06-01')), PARTITION p202306
VALUES [('2023-06-01'), ('2023-07-01')), PARTITION p202307 VALUES
[('2023-07-01'), ('2023-08-01')), PARTITION p202308 VALUES
[('2023-08-01'), ('2023-09-01')), PARTITION p202309 VALUES
[('2023-09-01'), ('2023-10-01')), PARTITION p202310 VALUES
[('2023-10-01'), ('2023-11-01')), PARTITION p202311 VALUES
[('2023-11-01'), ('2023-12-01')), PARTITION p202312 VALUES
[('2023-12-01'), ('2024-01-01')), PARTITION p202401 VALUES
[('2024-01-01'), ('2024-02-01')), PARTITION p202402 VALUES
[('2024-02-01'), ('2024-03-01')), PARTITION p202403 VALUES
[('2024-03-01'), ('2024-04-01')), PARTITION p202404 VALUES
[('2024-04-01'), ('2024-05-01')), PARTITION p202405 VALUES
[('2024-05-01'), ('2024-06-01')), PARTITION p202406 VALUES
[('2024-06-01'), ('2024-07-01')), PARTITION p202407 VALUES
[('2024-07-01'), ('2024-08-01')), PARTITION p202408 VALUES
[('2024-08-01'), ('2024-09-01')), PARTITION p202409 VALUES
[('2024-09-01'), ('2024-10-01')), PARTITION p202410 VALUES
[('2024-10-01'), ('2024-11-01')), PARTITION p202411 VALUES
[('2024-11-01'), ('2024-12-01')), PARTITION p202412 VALUES
[('2024-12-01'), ('2025-01-01')), PARTITION p202501 VALUES
[('2025-01-01'), ('2025-02-01'))) DISTRIBUTED BY HASH(`pin_id`) BUCKETS
32 PROPERTIES ( "replication_allocation" = "tag.location.default: 1",
"dynamic_partition.enable" = "true", "dynamic_partition.time_unit" =
"month", "dynamic_partition.time_zone" = "Asia/Shanghai",
"dynamic_partition.start" = "-2147483648", "dynamic_partition.end" =
"5", "dynamic_partition.prefix" = "p",
"dynamic_partition.replication_allocation" = "tag.location.default: 1",
"dynamic_partition.buckets" = "32",
"dynamic_partition.create_history_partition" = "true",
"dynamic_partition.history_partition_num" = "12",
"dynamic_partition.hot_partition_num" = "0",
"dynamic_partition.reserved_history_periods" = "NULL",
"dynamic_partition.storage_policy" = "",
"dynamic_partition.storage_medium&q
(doris) branch branch-2.0 updated: [Improve](Legacy planner)forbidden mvrewrite when there is no mv index in legacy planner (#40000)
This is an automated email from the ASF dual-hosted git repository.
yangbowen pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 0d9dfd8f8d0 [Improve](Legacy planner)forbidden mvrewrite when there is
no mv index in legacy planner (#4)
0d9dfd8f8d0 is described below
commit 0d9dfd8f8d0f95afe04d8a30d5bb30ff86bc2967
Author: wangqt
AuthorDate: Mon Sep 2 20:22:58 2024 +0800
[Improve](Legacy planner)forbidden mvrewrite when there is no mv index in
legacy planner (#4)
Problem:
In legacy planner mode, when do select request from a table with rollup
index; It will do mv rewrite operation, this will invoker too much
Expr.toSql that will cause cpu consumption。
Solution:
when do mv rewrite judgement，add mv index check when the tables index
number > 1；if there is no mv index, do not do mv rewrite.
-
Co-authored-by: wangqingtao6
---
.../src/main/java/org/apache/doris/analysis/BinaryPredicate.java | 4 +++-
.../src/main/java/org/apache/doris/analysis/SelectStmt.java | 9 -
fe/fe-core/src/main/java/org/apache/doris/analysis/SlotRef.java | 4 +++-
3 files changed, 14 insertions(+), 3 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/BinaryPredicate.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/BinaryPredicate.java
index 05014a4e0f0..00c67ab1203 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/BinaryPredicate.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/BinaryPredicate.java
@@ -309,7 +309,9 @@ public class BinaryPredicate extends Predicate implements
Writable {
Preconditions.checkState(match.getReturnType().getPrimitiveType() ==
PrimitiveType.BOOLEAN);
//todo(dhc): should add oppCode
//this.vectorOpcode = match.opcode;
-LOG.debug(debugString() + " opcode: " + vectorOpcode);
+if (LOG.isDebugEnabled()) {
+LOG.debug(debugString() + " opcode: " + vectorOpcode);
+}
}
private boolean canCompareDate(PrimitiveType t1, PrimitiveType t2) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
index 01352d2ab10..793ebf63e33 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
@@ -26,6 +26,7 @@ import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.DatabaseIf;
import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.FunctionSet;
+import org.apache.doris.catalog.MaterializedIndexMeta;
import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.PrimitiveType;
import org.apache.doris.catalog.Table;
@@ -512,7 +513,13 @@ public class SelectStmt extends QueryStmt {
}
OlapTable olapTable = (OlapTable) tbl.getTable();
if (olapTable.getIndexIds().size() != 1) {
-haveMv = true;
+for (MaterializedIndexMeta meta :
olapTable.getIndexIdToMeta().values()) {
+List idxColumns = meta.getSchema();
+// check the index is a mv index
+if (!idxColumns.isEmpty() && null !=
idxColumns.get(0).getDefineExpr()) {
+haveMv = true;
+}
+}
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/SlotRef.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/SlotRef.java
index bab3ac1baef..7275572c5b4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/SlotRef.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/SlotRef.java
@@ -199,7 +199,9 @@ public class SlotRef extends Expr {
@Override
public void computeOutputColumn(Analyzer analyzer) {
outputColumn = desc.getSlotOffset();
-LOG.debug("SlotRef: " + debugString() + " outputColumn: " +
outputColumn);
+if (LOG.isDebugEnabled()) {
+LOG.debug("SlotRef: " + debugString() + " outputColumn: " +
outputColumn);
+}
}
@Override
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [BugFix](StoragePolicy) fix modify partition's storage policy with different resource issue succeed with execption (#35270)
This is an automated email from the ASF dual-hosted git repository.
yangbowen pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new ec9800ae099 [BugFix](StoragePolicy) fix modify partition's storage
policy with different resource issue succeed with execption (#35270)
ec9800ae099 is described below
commit ec9800ae099b39ed21c117ed27f5c8f8a0ef9852
Author: shouchengShen <50309488+johnny...@users.noreply.github.com>
AuthorDate: Tue Jul 9 14:21:08 2024 +0800
[BugFix](StoragePolicy) fix modify partition's storage policy with
different resource issue succeed with execption (#35270)
## Proposed changes
Issue Number: close #34229
when we try to modify a partition's storage policy which have different
resource with the previous one, we get error message "currently do not
support change origin storage policy to another one with different
resource", but the modification same also succeed!
expect:
the modification fail with "currently do not support change origin
storage policy to another one with different resource".
## Further comments
If this is a relatively large or complex change, kick off the discussion
at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why
you chose the solution you did and what alternatives you considered,
etc...
-
Co-authored-by: shenshoucheng
---
.../main/java/org/apache/doris/alter/Alter.java| 4 +-
.../suites/account_p0/test_alter_user.groovy | 80 +--
.../policy/modify_partition_storage_policy.groovy | 147 +
.../suites/javaudf_p0/test_javaudf_auth.groovy | 2 +-
4 files changed, 190 insertions(+), 43 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
b/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
index bcfcfad81c1..36f82f524a1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
@@ -499,12 +499,12 @@ public class Alter {
// currently, only in memory and storage policy property could
reach here
Preconditions.checkState(properties.containsKey(PropertyAnalyzer.PROPERTIES_INMEMORY)
||
properties.containsKey(PropertyAnalyzer.PROPERTIES_STORAGE_POLICY));
-((SchemaChangeHandler)
schemaChangeHandler).updatePartitionsProperties(
-db, tableName, partitionNames, properties);
OlapTable olapTable = (OlapTable) table;
olapTable.writeLockOrDdlException();
try {
modifyPartitionsProperty(db, olapTable, partitionNames,
properties, clause.isTempPartition());
+((SchemaChangeHandler)
schemaChangeHandler).updatePartitionsProperties(
+db, tableName, partitionNames, properties);
} finally {
olapTable.writeUnlock();
}
diff --git a/regression-test/suites/account_p0/test_alter_user.groovy
b/regression-test/suites/account_p0/test_alter_user.groovy
index 9414025ef04..89ce1c4e197 100644
--- a/regression-test/suites/account_p0/test_alter_user.groovy
+++ b/regression-test/suites/account_p0/test_alter_user.groovy
@@ -20,51 +20,51 @@ suite("test_alter_user", "account") {
sql """drop user if exists test_auth_user2"""
sql """drop user if exists test_auth_user3"""
sql """drop user if exists test_auth_user4"""
-
+
// 2. test password history
sql """set global password_history=0""" // disabled
-sql """create user test_auth_user2 identified by '12345' password_history
default"""
+sql """create user test_auth_user2 identified by 'qwe678^&*'
password_history default"""
sql """grant all on *.* to test_auth_user2"""
-sql """alter user test_auth_user2 identified by '12345'"""
-sql """set password for 'test_auth_user2' = password('12345')"""
-
+sql """alter user test_auth_user2 identified by 'qwe678^&*'"""
+sql """set password for 'test_auth_user2' = password('qwe678^&*')"""
+
sql """set global password_history=1""" // set to 1
test {
-sql """alter user test_auth_user2 identified by '12345'&
(doris) branch branch-2.0 updated: [cherry-pick](branch-2.0) fix be metric doris_be_process_thread_num is zero (#36718)
This is an automated email from the ASF dual-hosted git repository.
yangbowen pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new e9f604e8001 [cherry-pick](branch-2.0) fix be metric
doris_be_process_thread_num is zero (#36718)
e9f604e8001 is described below
commit e9f604e8001010ea1f6b8bf120bef5aaf4af9169
Author: Xujian Duan <50550370+darvend...@users.noreply.github.com>
AuthorDate: Tue Jul 9 14:29:32 2024 +0800
[cherry-pick](branch-2.0) fix be metric doris_be_process_thread_num is zero
(#36718)
## Proposed changes
cherry-pick #35511
---
be/src/util/doris_metrics.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/src/util/doris_metrics.cpp b/be/src/util/doris_metrics.cpp
index 21264ed2fa2..668fb9017aa 100644
--- a/be/src/util/doris_metrics.cpp
+++ b/be/src/util/doris_metrics.cpp
@@ -354,7 +354,7 @@ void DorisMetrics::_update_process_thread_num() {
int64_t count =
std::count_if(dict_iter, std::filesystem::end(dict_iter), [](const
auto& entry) {
std::error_code error_code;
-return entry.is_regular_file(error_code) && !error_code;
+return entry.is_directory(error_code) && !error_code;
});
process_thread_num->set_value(count);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [branch-2.0](schema change) opt cooldown data schema change (#40963)
This is an automated email from the ASF dual-hosted git repository.
yangbowen pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 70e822461f2 [branch-2.0](schema change) opt cooldown data schema
change (#40963)
70e822461f2 is described below
commit 70e822461f25cf63a67b82813abb1cf2d2523771
Author: Xujian Duan <50550370+darvend...@users.noreply.github.com>
AuthorDate: Fri Sep 20 11:01:44 2024 +0800
[branch-2.0](schema change) opt cooldown data schema change (#40963)
## Proposed changes
This PR solves 3 problems of schema change for cooldown data:
1. Schema change will build a new tablet for base tablet, and all
replicas do this separately, so the cooldown data SC writes all replicas
to the remote storage, But the cooldown rowsets just need one master
replica to cooldown, the other replicas follow the master. To optimize
this issue, Doris can just write the new rowsets to the local storage,
and the tablet will cooldown this rowsets automatically for just one
mater replica.
3. In schema change job and rollup job, FE generates createReplicaTask,
which specifies the storage policy of the tablets to create, table's
storage policy may be empty if we just set the partitions' storage
policy, so we use partition's storage policy instead of table.
5. checks the target tablets in the loop of data conversion, and
terminates the data conversion thread if the tablet is been dropped.
---
be/src/olap/schema_change.cpp| 16 +++-
.../main/java/org/apache/doris/alter/RollupJobV2.java| 3 ++-
.../java/org/apache/doris/alter/SchemaChangeJobV2.java | 3 ++-
3 files changed, 19 insertions(+), 3 deletions(-)
diff --git a/be/src/olap/schema_change.cpp b/be/src/olap/schema_change.cpp
index 7d867d89460..7e6b9a3ff35 100644
--- a/be/src/olap/schema_change.cpp
+++ b/be/src/olap/schema_change.cpp
@@ -498,6 +498,13 @@ Status
VSchemaChangeDirectly::_inner_process(RowsetReaderSharedPtr rowset_reader
TabletSchemaSPtr
base_tablet_schema) {
bool eof = false;
do {
+// tablet may be dropped due to user cancel, schema change thread
should fast fail
+// and release tablet lock.
+if (new_tablet->tablet_state() == TABLET_SHUTDOWN) {
+return Status::Error(
+"fail to process tablet because it is to be deleted.
tablet_id={}",
+new_tablet->tablet_id());
+}
auto new_block =
vectorized::Block::create_unique(new_tablet->tablet_schema()->create_block());
auto ref_block =
vectorized::Block::create_unique(base_tablet_schema->create_block());
@@ -580,6 +587,13 @@ Status
VSchemaChangeWithSorting::_inner_process(RowsetReaderSharedPtr rowset_rea
bool eof = false;
do {
+// tablet may be dropped due to user cancel, schema change thread
should fast fail
+// and release tablet lock.
+if (new_tablet->tablet_state() == TABLET_SHUTDOWN) {
+return Status::Error(
+"fail to process tablet because it is to be deleted.
tablet_id={}",
+new_tablet->tablet_id());
+}
auto ref_block =
vectorized::Block::create_unique(base_tablet_schema->create_block());
auto st = rowset_reader->next_block(ref_block.get());
if (!st) {
@@ -1103,7 +1117,7 @@ Status
SchemaChangeHandler::_convert_historical_rowsets(const SchemaChangeParams
context.segments_overlap =
rs_reader->rowset()->rowset_meta()->segments_overlap();
context.tablet_schema = new_tablet->tablet_schema();
context.newest_write_timestamp = rs_reader->newest_write_timestamp();
-context.fs = rs_reader->rowset()->rowset_meta()->fs();
+context.fs = io::global_local_filesystem();
context.write_type = DataWriteType::TYPE_SCHEMA_CHANGE;
Status status = new_tablet->create_rowset_writer(context,
&rowset_writer);
if (!status.ok()) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/RollupJobV2.java
b/fe/fe-core/src/main/java/org/apache/doris/alter/RollupJobV2.java
index 19d8dc13d75..7bb59269a33 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/RollupJobV2.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/RollupJobV2.java
@@ -282,7 +282,8 @@ public class RollupJobV2 extends AlterJobV2 implements
GsonPostProcessable {
tabletType,
null,
tbl.getCompressionType(),
-tbl.getEnableUniqueKeyMergeOnWr
(doris) branch master updated (d42e8294372 -> acea8fc87ca)
This is an automated email from the ASF dual-hosted git repository. yangbowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from d42e8294372 [opt](cloud) Simplify default config file of meta-service (#43381) add acea8fc87ca [fix](resource) fix can't update BE s3 resource config (#43025) No new revisions were added by this update. Summary of changes: be/src/agent/task_worker_pool.cpp | 12 ++-- be/src/io/fs/s3_file_system.cpp | 6 -- 2 files changed, 6 insertions(+), 12 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [bug](s3) fix S3 file system gets absolute path (#44965)
This is an automated email from the ASF dual-hosted git repository.
yangbowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 0ec35de23a7 [bug](s3) fix S3 file system gets absolute path (#44965)
0ec35de23a7 is described below
commit 0ec35de23a7ae2a0e12fb6ec5be3f5d5196ee8dd
Author: Xujian Duan <50550370+darvend...@users.noreply.github.com>
AuthorDate: Mon Dec 9 17:08:31 2024 +0800
[bug](s3) fix S3 file system gets absolute path (#44965)
### What problem does this PR solve?
Issue Number: close https://github.com/apache/doris/issues/44902
Related PR: #xxx
### Problem Summary
If we create a S3 resource for cooldown data storage and the
s3.root.path has a leading slash(/), such as: /root, the remote S3 data
file will not be deleted if we drop the cooldown tablet from Doris.
### Reason
Doris will get the path of all files of a tablet and then delete those
files, The AWS `S3Client:ListObjectsV2` uses objects prefix to get the
object files, but if the prefix has a leading slash(/),
the`S3Client:ListObjectsV2` gets empty result.
### Solution
use _prefix instead of _root_path in S3FileSystem for getting absolute
path, _prefix is normalized in constructor and it is removed the first
and last '/'.
### test case
1. create a s3 resource with a leading '/' and a storage policy base on
it:
```
CREATE RESOURCE "test_resource"
PROPERTIES
(
"type" = "s3",
"s3.endpoint" = "xxx",
"s3.region" = "xxx",
"s3.bucket" = "xxx",
"s3.root.path" = "/tmp",
"s3.access_key" = "xx",
"s3.secret_key" = "xx"
);
CREATE STORAGE POLICY test_policy PROPERTIES (
"storage_resource" = "test_resource",
"cooldown_ttl" = "1"
)
```
2. create a table and set the storage_policy to `test_policy` and insert
test data:
```
CREATE TABLE `test_table` (
`k` bigint,
`v` bigint
) ENGINE=OLAP
DUPLICATE KEY(`k`)
DISTRIBUTED BY HASH(`k`) BUCKETS 1
PROPERTIES (
"replication_num" = "1",
"storage_policy" = "test_policy"
);
insert into test_table values (1,2),(2,3);
```
3. wait for the rowset cooldown to s3.
4. truncate test table
```
truncate table test_table force;
```
5. If we get a log like ` delete remote rowsets of tablet.
root_path="/tmp", tablet_id=xxx` in be.INFO, the remote tablet file
should be deleted.
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---
be/src/io/fs/s3_file_system.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/src/io/fs/s3_file_system.h b/be/src/io/fs/s3_file_system.h
index 61967a63e44..f6efa505332 100644
--- a/be/src/io/fs/s3_file_system.h
+++ b/be/src/io/fs/s3_file_system.h
@@ -121,7 +121,7 @@ protected:
abs_path = path;
} else {
// path with no schema
-abs_path = _root_path / path;
+abs_path = _prefix / path;
}
return Status::OK();
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
