github-actions[bot] commented on code in PR #68196:
URL: https://github.com/apache/doris/pull/68196#discussion_r4129218827
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaCacheMgr.java:
##########
@@ -486,8 +507,42 @@ public void removeCatalogByEngine(long catalogId, String
engine) {
}
public void invalidateDb(long catalogId, String dbName) {
- routeCatalogEngines(catalogId, cache -> safeInvalidate(
- cache, catalogId, "invalidateDb", () ->
cache.invalidateDb(catalogId, dbName)));
+ Optional<ExternalDatabase<? extends ExternalTable>> db =
getCachedDb(catalogId, dbName);
+ if (db.isPresent()) {
+ invalidateDb(catalogId, db.get().getId(), db.get().getFullName());
+ return;
+ }
+ Optional<Pair<String, Long>> identity =
getDbIdentityForReplay(catalogId, dbName);
+ if (identity.isPresent()) {
+ invalidateDb(catalogId, identity.get().second,
identity.get().first);
+ } else {
+ invalidateDb(catalogId, dbName, OptionalLong.empty(), true);
+ }
+ }
+
+ public void invalidateDb(long catalogId, long dbId, String dbName) {
+ invalidateDb(catalogId, dbName, OptionalLong.of(dbId), true);
+ }
+
+ void invalidateDb(long catalogId, long dbId, String dbName, boolean
invalidateRowCountCache) {
+ invalidateDb(catalogId, dbName, OptionalLong.of(dbId),
invalidateRowCountCache);
+ }
+
+ private void invalidateDb(long catalogId, String dbName, OptionalLong
dbId, boolean invalidateRowCountCache) {
+ try {
+ routeCatalogEngines(catalogId, cache -> safeInvalidate(
Review Comment:
[P1] Fence direct DB invalidation before evicting engine entries. Cold
REFRESH DB/TABLE replay calls this explicit-ID route after the DB object was
evicted, while its completed table row count can survive. `routeCatalogEngines`
clears Hive entries before the only row-count fence in `finally`; a concurrent
query can rebuild the cold table and read new metadata with the old count in
that gap. The unresolved replay catalog route has the same ordering. Open the
matching DB/catalog row-count fence before routed eviction, retain the
completion fence, and latch a query after engine eviction.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaCacheMgr.java:
##########
@@ -497,16 +552,28 @@ public void invalidateDb(ExternalDatabase<?> database) {
}
public void invalidateTable(long catalogId, String dbName, String
tableName) {
- invalidateLanceTableAccess(catalogId);
- routeCatalogEngines(catalogId, cache -> safeInvalidate(
- cache, catalogId, "invalidateTable",
- () -> cache.invalidateTable(catalogId, dbName, tableName)));
+ Optional<ExternalDatabase<? extends ExternalTable>> db =
Optional.empty();
+ try {
+ invalidateLanceTableAccess(catalogId);
+ db = getCachedDb(catalogId, dbName);
+ routeCatalogEngines(catalogId, cache -> safeInvalidate(
+ cache, catalogId, "invalidateTable",
+ () -> cache.invalidateTable(catalogId, dbName,
tableName)));
+ } finally {
+ invalidateTableRowCount(catalogId, db, tableName);
+ }
}
public void invalidateTableByEngine(long catalogId, String engine, String
dbName, String tableName) {
- routeSpecifiedEngine(engine, cache -> safeInvalidate(
- cache, catalogId, "invalidateTableByEngine",
- () -> cache.invalidateTable(catalogId, dbName, tableName)));
+ Optional<ExternalDatabase<? extends ExternalTable>> db =
Optional.empty();
+ try {
+ db = getCachedDb(catalogId, dbName);
+ routeSpecifiedEngine(engine, cache -> safeInvalidate(
Review Comment:
[P2] Open the row-count fence before this engine-specific table eviction.
`HMSExternalTable.getHivePartitionValues` calls this route after detecting
inconsistent Hive partition cache state. `routeSpecifiedEngine` removes
partition/file entries before `invalidateTableRowCount` runs in `finally`, so
another planner can reload corrected partition data and still consume the old
completed count during the gap. Fence the resolved table (or DB when cold)
before routing and keep the completion fence; test the ordering with an
invalidation paused after entry removal.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalog.java:
##########
@@ -173,6 +173,25 @@ public synchronized void invalidatePaimonDatabase(String
remoteDbName) throws Ex
}));
}
+ public synchronized void invalidatePaimonDatabaseByLocalName(String
localDbName) throws Exception {
+ if (!isInitialized()) {
+ return;
+ }
+ withSdkCatalogCacheWriteLock(() -> executionAuthenticator.execute(()
-> {
+ boolean caseSensitive = catalog.caseSensitive();
+ int nameMode = getLowerCaseDatabaseNames();
+ invalidateCachedPaimonTables(identifier -> {
+ String remoteName = identifier.getDatabaseName();
+ String candidateLocalName = nameMode == 2 ? remoteName :
fromRemoteDatabaseName(remoteName);
+ if (nameMode == 1) {
+ candidateLocalName =
candidateLocalName.toLowerCase(Locale.ROOT);
Review Comment:
[P2] Match the catalog's mode-1 local-name normalization here. On a
Turkish-locale FE, Doris stores remote Paimon DB `I` as local `ı` using
default-locale `toLowerCase()`, but this new SDK cache predicate converts the
same remote name to `i` with `Locale.ROOT`. For a case-sensitive Paimon
catalog, `identifierPartEquals` fails, so a cold DB refresh or DROP leaves
SDK-only table handles in `CachingCatalog.tableCache` even though the retained
DB identity was found. Use the same normalization as `ExternalCatalog` (or make
both use one locale-independent rule), and cover a cached SDK handle with a
locale-sensitive database name.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java:
##########
@@ -941,19 +954,46 @@ public Optional<ExternalDatabase<? extends
ExternalTable>> getDbForReplay(String
LOG.debug("getDbForReplay from metacache, db: {}.{}, catalog id:
{}, is catalog init: {}",
this.name, dbName, this.id, isInitialized());
}
- if (!isInitialized()) {
+ if (!isInitialized() || metaCache == null) {
return Optional.empty();
}
// Apply case-insensitive lookup with isReplay=true (no remote calls)
String localDbName = getLocalDatabaseName(dbName, true);
if (localDbName == null) {
localDbName = dbName; // Fallback to original name
+ } else if (getLowerCaseDatabaseNames() == 2 &&
!localDbName.equals(dbName)) {
+ // The current case-insensitive mapping may have rebound after a
DROP. A replay
+ // record names the historical object, not a new same-folded
replacement.
+ long historicalId = Util.genIdByName(name, dbName);
+ if (metaCache.getNameByIdIfPresent(historicalId).isPresent()) {
Review Comment:
[P1] Preserve ordinary mode-2 aliases in replay-safe lookup. With canonical
`MixedDb`, Hive can successfully run `CREATE TABLE mixeddb.t` using the
canonical remote DB, but this new check looks for a separate `mixeddb` ID slot
and returns empty. Its `afterCreateTable` hook then skips
`resetMetaCacheNames`, so a primed table list can hide `t` until refresh;
follower replay has the same caller spelling. Other post-create hooks use this
lookup too. The merge-base lookup resolved the warm canonical DB. Keep
historical DROP identity handling separate from normal alias resolution, and
cover alternate-case CREATE with a primed table-name cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
