morrySnow commented on code in PR #47066:
URL: https://github.com/apache/doris/pull/47066#discussion_r1917845254
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/SqlCacheContext.java:
##########
@@ -475,6 +487,7 @@ public static class TableVersion {
public final long id;
public final long version;
public final TableType type;
+
Review Comment:
useless empty line
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/SqlCacheContext.java:
##########
@@ -462,10 +463,21 @@ public static class ScanTable {
public final FullTableName fullTableName;
public final long latestVersion;
public final List<Long> scanPartitions = Lists.newArrayList();
+ public final List<Column> latestColumns = Lists.newArrayList();
+ public final Map<Long, Long> scanPartitionIdToVersion =
Maps.newHashMap();
+
public void addScanPartition(Long partitionId) {
this.scanPartitions.add(partitionId);
}
+
+ public void setLatestColumns(List<Column> columns) {
+ this.latestColumns.addAll(columns);
Review Comment:
i think we need deep copy each column here. because column is mutable
##########
fe/fe-core/src/main/java/org/apache/doris/common/cache/NereidsSqlCacheManager.java:
##########
@@ -350,13 +344,26 @@ private boolean tablesOrDataChanged(Env env,
SqlCacheContext sqlCacheContext) {
return true;
}
OlapTable olapTable = (OlapTable) tableIf;
- Collection<Long> partitionIds = scanTable.getScanPartitions();
- olapTable.getVersionInBatchForCloudMode(partitionIds);
-
- for (Long scanPartitionId : scanTable.getScanPartitions()) {
- Partition partition = olapTable.getPartition(scanPartitionId);
+ List<Column> currentFullSchema = olapTable.getFullSchema();
+ List<Column> cacheFullSchema = scanTable.latestColumns;
+ if (currentFullSchema.size() != cacheFullSchema.size()) {
+ return true;
+ }
+ for (int i = 0; i < currentFullSchema.size(); i++) {
+ Column currentColumn = currentFullSchema.get(i);
+ Column cacheColumn = cacheFullSchema.get(i);
+ if (currentColumn.hashCode() != cacheColumn.hashCode()) {
+ return true;
Review Comment:
only compare hashCode is enough?
##########
fe/fe-core/src/main/java/org/apache/doris/common/cache/NereidsSqlCacheManager.java:
##########
@@ -350,13 +344,26 @@ private boolean tablesOrDataChanged(Env env,
SqlCacheContext sqlCacheContext) {
return true;
}
OlapTable olapTable = (OlapTable) tableIf;
- Collection<Long> partitionIds = scanTable.getScanPartitions();
- olapTable.getVersionInBatchForCloudMode(partitionIds);
-
- for (Long scanPartitionId : scanTable.getScanPartitions()) {
- Partition partition = olapTable.getPartition(scanPartitionId);
+ List<Column> currentFullSchema = olapTable.getFullSchema();
+ List<Column> cacheFullSchema = scanTable.latestColumns;
+ if (currentFullSchema.size() != cacheFullSchema.size()) {
+ return true;
+ }
+ for (int i = 0; i < currentFullSchema.size(); i++) {
+ Column currentColumn = currentFullSchema.get(i);
+ Column cacheColumn = cacheFullSchema.get(i);
+ if (currentColumn.hashCode() != cacheColumn.hashCode()) {
+ return true;
+ }
+ }
+ Map<Long, Long> scanPartitionIdToVersion =
scanTable.scanPartitionIdToVersion;
+ for (Entry<Long, Long> entry :
scanPartitionIdToVersion.entrySet()) {
+ Long scanPartitionId = entry.getKey();
+ Long cachePartitionVersion = entry.getValue();
+ Partition currentPartition =
olapTable.getPartition(scanPartitionId);
Review Comment:
get partition one by one will lead to bad perfermance in cloud mode
##########
fe/fe-core/src/main/java/org/apache/doris/qe/cache/CacheAnalyzer.java:
##########
@@ -707,6 +708,7 @@ private CacheTable
buildCacheTableForOlapScanNode(OlapScanNode node) {
for (Long partitionId : node.getSelectedPartitionIds()) {
Partition partition = olapTable.getPartition(partitionId);
scanTable.addScanPartition(partitionId);
+ scanTable.addScanPartitionIdToVersion(partitionId,
partition.getVisibleVersion());
Review Comment:
could merge them into one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
