This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d321977d17b [fix](hive) Fix partition values cache sync issue in
Observer FE during insert (#59965)
d321977d17b is described below
commit d321977d17b8612ce17fe93b3e11539a3484945d
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Sat Jan 17 23:35:41 2026 +0800
[fix](hive) Fix partition values cache sync issue in Observer FE during
insert (#59965)
### What problem does this PR solve?
Related PR: #58166
Problem Summary:
When Observer FE receives edit logs from Master FE after a Hive insert
operation, it may miss adding new partitions to its partition values
cache. This happens because:
1. When inserting into a new partition, Master FE refreshes its cache
and gets the partition into its partition values cache
2. Master FE records the partition as MODIFIED (not NEW) in the edit log
if it already exists in HMS
3. Observer FE processes only newPartNames when updating partition
values cache, missing partitions marked as MODIFIED
Fix by merging both modifiedPartNames and newPartNames before adding to
the partition values cache, ensuring Observer FE stays synchronized with
Master FE.
---
.../doris/datasource/hive/HiveMetaStoreCache.java | 17 ++++++++++++++---
1 file changed, 14 insertions(+), 3 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java
index e560099a18d..8b07f3e875a 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java
@@ -595,10 +595,21 @@ public class HiveMetaStoreCache {
for (String partitionName : modifiedPartNames) {
invalidatePartitionCache(table, partitionName);
}
-
+ // Merge modifiedPartNames and newPartNames
+ // Case:
+ // 1. hive, insert into a new partition p_new
+ // 2. doris-observer, insert into same partition p_new
+ // 1. forward insert command to Master
+ // 2. Master FE will refresh its cache and get p_new into its
partition values cache
+ // 3. Insert finished and Master write edit log, but p_new is
recorded as MODIFIED not NEW.
+ // (See refreshAffectedPartitions() methods)
+ // 4. Observer FE receive edit log and refresh cache, if we don't
merge them,
+ // it will miss adding p_new to its partition values cache.
+ List<String> mergedPartNames = Lists.newArrayList(modifiedPartNames);
+ mergedPartNames.addAll(newPartNames);
// Add new partitions to partition values cache
- if (!newPartNames.isEmpty()) {
- addPartitionsCache(table.getOrBuildNameMapping(), newPartNames,
+ if (!mergedPartNames.isEmpty()) {
+ addPartitionsCache(table.getOrBuildNameMapping(), mergedPartNames,
table.getPartitionColumnTypes(Optional.empty()));
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
