wenzhenghu opened a new pull request, #65126: URL: https://github.com/apache/doris/pull/65126
## What problem does this PR solve? Issue Number: N/A Related PR: [#64705](https://github.com/apache/doris/pull/64705) Problem Summary: This PR refactors the FE external metadata cache from the old `MetaCache<T>` wrapper into independent `MetaCacheEntry` instances and introduces `NameCacheValue` as the immutable names snapshot used by catalog/database name resolution. The refactor keeps slow remote I/O out of Caffeine's synchronous load path, makes publication/invalidation generation-aware, and unifies the lower-case mode 0/1/2 behavior around one names snapshot. After the initial refactor, PMC review feedback identified several follow-up items. This branch now incorporates those review-driven updates: 1. Fix the mode-2 replay corner case when a names-only invalidation makes the names snapshot cold while the object cache is still hot. Replay remains cache-only and now falls back to a case-insensitive scan over hot object-cache keys instead of reloading names. 2. Optimize `NameCacheValue` hot-path lookups by precomputing local-name indexes and exposing a cached `localNames()` view, while keeping `names()` defensive because `Pair` is mutable. 3. Add comments that clarify the intended semantics for same-key load deduplication, hot-entry-only incremental names maintenance, generation-aware invalidation, and mode-2 cache-key conventions. Some review findings were intentionally not turned into extra logic changes in this PR: - The invalidate generation flow is kept as-is because the all-stripe bump and the per-key invalidation path protect different publication windows. - Hot-entry-only incremental names updates are kept as-is because the observed race leads to an extra reload, not incorrect metadata state. - The repeated catalog/database code is intentionally not extracted into a shared helper in this follow-up patch. Although the shapes are similar, the two paths still differ in replay semantics, lower-case handling, object/ID bookkeeping, and test scaffolding. For this review-driven update, extracting a helper would widen the refactor surface and increase regression risk. ### Implementation details #### 1. Overall architecture The legacy cache grouped the names cache, object cache, and ID-to-name index in one `MetaCache<T>` wrapper: ```text ExternalCatalog / ExternalDatabase -> MetaCache<T> -> namesCache: LoadingCache<String, List<Pair<remote, local>>> -> metaObjCache: LoadingCache<String, Optional<T>> -> idToName ``` The refactored structure makes each logical dataset an independent `MetaCacheEntry`: ```text ExternalCatalog -> databaseNames: MetaCacheEntry<String, NameCacheValue> -> databases: MetaCacheEntry<String, ExternalDatabase> -> dbIdToName ExternalDatabase -> tableNames: MetaCacheEntry<String, NameCacheValue> -> tables: MetaCacheEntry<String, ExternalTable> -> tableIdToName ``` The main architectural changes are: - Remove the `MetaCache<T>` wrapper that owned two `LoadingCache` instances. - Use one `MetaCacheEntry<K, V>` for each logical cache dataset. - Store the names list and its case-insensitive lookup index in one immutable `NameCacheValue` snapshot. - Move slow miss loading out of Caffeine's synchronous load path. Striped load locks deduplicate loads, while publication locks and generations coordinate explicit mutations, invalidation, and stale-result suppression. #### 2. Configuration `Config.external_meta_cache_object_entry_lock_stripes` is a non-mutable configuration with a default value of `256`. It controls the number of load-lock, publication-lock, and generation stripes used by multi-key `MetaCacheEntry` instances. Keys mapped to the same stripe share the same load and publication locks, so the value balances memory usage against unrelated-key contention. `Config.enable_external_meta_cache_name_miss_refresh` is mutable and defaults to `true`. It controls what happens when a names snapshot is already cached but a requested database or table name is absent: - When enabled, Doris invalidates the names entry, synchronously reloads it once, and retries the lookup. - When disabled, the lookup returns not found without enumerating remote names again. The option is enabled by default to preserve the existing external catalog visibility behavior. A database or table created outside Doris can therefore become visible on the next direct lookup without requiring an explicit refresh. Operators can disable the option to avoid repeated expensive `listDatabaseNames()` or `listTableNames()` calls when non-existent objects are queried frequently; with the option disabled, externally created objects become visible through an explicit refresh or metadata synchronization. The previous `enable_external_meta_cache_manual_miss_load` switch is removed. `MetaCacheEntry.get()` now always uses manual miss loading and can no longer fall back to `LoadingCache.get()` for synchronous loader execution. #### 3. `MetaCacheEntry` ##### 3.1 Concurrency state `MetaCacheEntry` replaces the fixed 128 load-lock stripes and single global invalidation generation with configurable per-stripe state: - `loadLocks` deduplicate slow miss loads. Slow external I/O runs while holding only the corresponding load stripe, not a Caffeine bin lock or publication lock. - `publishLocks` protect short cache publication, explicit mutation, and key invalidation windows. - `generations` stores one generation per stripe so mutations to one stripe do not invalidate in-flight work on every other stripe. ##### 3.2 Construction The existing common constructors remain available, and overloads accepting an explicit `stripeCount` are added. All public constructors converge on one private constructor that validates the supported combinations and builds the underlying Caffeine cache. `withSyncRemovalListener(...)` creates an entry with a synchronous removal listener. This path forces `autoRefresh=false`, because the existing synchronous-removal-listener cache construction must not be combined with `refreshAfterWrite`. Contextual-only entries must continue to use a null default loader and cannot enable automatic refresh. All entries now use the custom `CacheLoader` returned by `newCacheLoader()` so that `refreshAfterWrite` reloads participate in generation checks. ##### 3.3 Read APIs `get(K)` always calls `getWithManualLoad()` with the default loader. It no longer reads a runtime feature switch and never invokes `loadingData.get(key)` for a normal miss. `get(K, Function<K, V>)` uses the same manual path but executes the contextual loader supplied by the caller. `getIfPresent(K)` never performs synchronous load-through. It returns null when the entry is disabled. A cache hit may still cause Caffeine to schedule asynchronous `refreshAfterWrite`, but the caller never waits for remote I/O. ##### 3.4 Explicit mutations `put(K, V)` rejects null keys and values and is a no-op when caching is disabled. Otherwise, it acquires the key's publication lock, increments the stripe generation, and writes the value. Incrementing the generation prevents an earlier slow load from later overwriting the explicit value. `compute(K, BiFunction<K, V, V>)` is the generation-aware replacement for upper-layer direct access to `data.asMap().compute()`. It performs the generation bump and map mutation under the publication lock. As with `ConcurrentMap.compute`, a null remapping result removes the key. When caching is disabled, the remapping function is not executed. ##### 3.5 Invalidation `invalidateKey(K)` acquires the publication lock, bumps the key stripe generation, removes the key, and increments the invalidation count only when a cached value was actually removed. `invalidateIf(Predicate<K>)` first calls `bumpAllGenerations()` so that in-flight loads whose keys have not yet appeared in the cache map cannot publish after the invalidation. It then iterates visible keys and delegates matching keys to `invalidateKey()`. The initial all-stripe bump is intentionally conservative: an absent in-flight key cannot be evaluated against the predicate without additional tracking. `invalidateAll()` also bumps every stripe before iterating and invalidating each visible key. It no longer calls Caffeine's `invalidateAll()` directly, ensuring that published keys pass through the same publication coordination as single-key invalidation. ##### 3.6 Manual miss loading `getWithManualLoad()` implements the following sequence: 1. If caching is disabled, execute and track the loader without writing the result into the cache. 2. Check `getIfPresent()` on the fast path. 3. On a miss, acquire the key's load stripe and recheck the cache to deduplicate slow loads. 4. Briefly acquire the publication lock, recheck the cache again, and snapshot the stripe generation. This prevents an explicit mutation from occurring between miss observation and version capture. 5. Execute the slow loader outside the publication lock and outside Caffeine's synchronous load path. 6. Leave null loader results uncached. 7. Reacquire the publication lock. If the generation changed, return the loaded value to the current caller but do not cache it. 8. If the generation is unchanged, publish through `putLoadedValueWithoutGenerationBump()` and retain a post-put generation check. `putLoadedValueWithoutGenerationBump()` is deliberately separate from the public `put()`: an internal loader publication must not invalidate the generation it captured for itself. `removeLoadedValueWithoutGenerationBump()` removes the value only when the currently cached object is the same object published by that load. This avoids deleting a newer explicit replacement. ##### 3.7 Generation-aware asynchronous refresh `newCacheLoader()` keeps `load()` for Caffeine refresh support but normal misses no longer use it. Its `asyncReload()` implementation: 1. Captures the key stripe generation before submitting the loader. 2. Runs the loader on the configured executor. 3. On completion, enters the publication lock and compares the current generation. 4. Completes the reload future when the generation is unchanged, allowing Caffeine to publish the refresh result. 5. Cancels the future when the generation changed, discarding a stale refresh result. ##### 3.8 Stripe helpers `stripe(K)` maps a key hash to a bounded stripe. `loadLock(K)` and `publishLock(K)` return the corresponding lock objects. `generationOf(K)` reads the stripe generation, `bumpGeneration(K)` invalidates work on one stripe, and `bumpAllGenerations()` invalidates publication eligibility across all stripes, including loads not yet represented in the cache map. `defaultObjectStripeCount()` reads the configured multi-key stripe count. `singleKeyStripeCount()` returns one for names entries, whose only key is the empty string. #### 4. `NameCacheValue` `NameCacheValue` is a new immutable snapshot containing both: - The ordered list of `(remoteName, localName)` pairs. - A `lower-case name -> original remote name` index for case-insensitive lookup. The constructor deep-copies every `Pair`, builds the index with `Locale.ROOT`, and uses last-write-wins behavior for the index. Catalog-aware loaders remain responsible for detecting unsupported case conflicts. `of(...)` creates a non-null snapshot, and `empty()` creates an empty snapshot for explicit forced-update paths. `names()` returns fresh `Pair` objects so callers cannot mutate a published snapshot through the mutable `Pair` type. `remoteNameOfLocalName()` replaces the legacy `MetaCache.getRemoteName()` lookup and now uses a precomputed local-name index. `containsLocalName()` supports existence checks in object initialization and now uses a precomputed local-name set. `localNames()` exposes a cached immutable local-name view for hot read paths like `SHOW DATABASES` and `SHOW TABLES`. `remoteNameForCaseInsensitiveLookup()` replaces the separate `lowerCaseToDatabaseName` and `lowerCaseToTableName` maps. `withName(remoteName, localName)` performs copy-on-write insertion. It rejects an exact remote name already mapped to a different local name, removes any old entry for the same local name, adds the new pair, and returns a new snapshot. Callers update this names snapshot before object and ID state so a conflict cannot leave a partially updated object cache. `withoutLocalName(localName)` performs copy-on-write removal and returns a new snapshot. `copyPairs()` provides the deep-copy boundary used during construction and readout. #### 5. `ExternalCatalog` ##### 5.1 Cache structure and initialization The legacy `MetaCache<ExternalDatabase>` and `lowerCaseToDatabaseName` side map are replaced by `databaseNames`, `databases`, and `dbIdToName`. `buildMetaCache()` creates: - `databaseNames`, keyed only by `""`, with a `NameCacheValue`, capacity one, one stripe, and `refreshAfterWrite` enabled. - `databases`, keyed by local database name, with configurable object stripes, `refreshAfterWrite` disabled, and a synchronous removal listener that calls `resetMetaToUninitialized()` on the removed database object. `getFilteredDatabaseNames()` still performs remote enumeration, include/exclude filtering, remote-to-local mapping, system database handling, and case-conflict validation. It no longer mutates a separate lower-case side map and now acts as the pure loader for a complete names snapshot. `buildDbForInit()` routes existence checks through `resolveDatabaseNameFromSnapshot()` and routes local-to-remote resolution through `getRemoteDatabaseName()`. The system database construction branches and the existing unit-test `runningUnitTest` bypass condition are preserved. ##### 5.2 Normal and replay lookup `getDbNames()` reads local database names from `databaseNames` instead of `MetaCache.listNames()`. `getDbNullable(String)` preserves `information_schema` and `mysql` name normalization. Other names are resolved to a local name before `databases.get()` performs manual miss loading. A successful lookup updates `dbIdToName` using `db.getId()` as the single ID source. `getDbNullable(long)` resolves the local name through `dbIdToName` and then calls `databases.get()`. It returns null immediately when no ID mapping exists. `getDbForReplay(long)` checks initialization, resolves the key through `dbIdToName`, and uses `databases.getIfPresent()`. This fixes the legacy path where `getMetaObjById()` could perform load-through during replay. `getDbForReplay(String)` first checks the exact object key. On a miss, it resolves the local name only from an existing names snapshot and performs another `getIfPresent()`. If mode 2 replay sees a cold names snapshot but the object cache is still hot, it now performs a cache-only case-insensitive fallback over hot object-cache keys instead of reloading names. Replay misses never perform synchronous remote loading. A hot names entry may still schedule background `refreshAfterWrite`, but the replay thread does not wait for it. ##### 5.3 Name resolution `getLocalDatabaseName()` preserves modes 0 and 1. Mode 2 now resolves the original-case name through `NameCacheValue.remoteNameForCaseInsensitiveLookup()`. `getDatabaseNamesValue(boolean allowLoad)` centralizes the choice between `databaseNames.get("")` and `databaseNames.getIfPresent("")`. `resolveDatabaseNameFromSnapshot()` centralizes negative-lookup behavior: - A cold non-replay miss loads the names entry normally. - A cold replay miss returns null without synchronous loading. - A hit returns the resolver result from the current snapshot. - A miss in an existing snapshot invalidates the names entry, reloads once, and retries by default. When `enable_external_meta_cache_name_miss_refresh` is disabled, it returns null without enumerating remote names again. `listLocalDatabaseNamesFromCache()` loads the snapshot when necessary and extracts local names from the cached `localNames()` view. `getRemoteDatabaseName()` uses the shared resolver and `remoteNameOfLocalName()` to replace `MetaCache.getRemoteName()`. ##### 5.4 Incremental updates and invalidation `updateDatabaseCache()` updates state in names -> object -> ID order. Runtime incremental events update names and object entries only when those entries are already hot: a single create event must not materialize an incomplete names snapshot or preheat an object the FE has never consumed. The lightweight `dbIdToName` navigation index is always updated from `db.getId()`, so normal by-ID lookup can load a cold object on demand. The event path and cold object loader use the same deterministic ID rule, so the reconstructed object retains the event ID. Replay by-ID lookup remains cache-only. `invalidateDatabaseCache()` removes the local name through copy-on-write only when the names snapshot is already hot, invalidates the object key, and removes every ID mapping whose value equals that local name. Drop events therefore do not load a cold names entry. `invalidateDatabaseNamesOnly()` invalidates only the names entry. `invalidateAllDatabaseCache()` invalidates names and objects and clears the ID index. `refreshMetaCacheOnly()` now delegates to the full helper. `unregisterDatabase()` delegates local state cleanup to `invalidateDatabaseCache()` while preserving the following engine-level `ExternalMetaCacheMgr.invalidateDb()` call. `resetMetaCacheNames()` preserves the existing CREATE DATABASE behavior by invalidating only the names entry. `resetToUninitialized()` no longer clears a separate lower-case map. `gsonPostProcess()` reconstructs an empty `dbIdToName`, making the ID map explicitly runtime-only rather than image-persisted state. #### 6. `ExternalDatabase` `ExternalDatabase` applies the same structure at table scope. `MetaCache<T>` and `lowerCaseToTableName` are replaced by `tableNames`, `tables`, and `tableIdToName`. `buildMetaCache()` creates a single-key, auto-refreshing `tableNames` entry and a multi-key `tables` entry with `autoRefresh=false`. The table object entry does not need the database-level removal callback used by `ExternalCatalog.databases`. `listTableNames()` preserves system database branches, remote enumeration, remote-to-local mapping, and conflict checks, but no longer mutates a lower-case side map. `resetMetaToUninitialized()` delegates to `invalidateAllTableCache()` so names, objects, and IDs are cleared together before engine-level database cache invalidation. `buildTableForInit()` uses `resolveTableNameFromSnapshot()` for existence checks and `getRemoteTableName()` for local-to-remote resolution. The original outer condition deciding whether remote-name mapping is necessary and the unit-test bypass condition are preserved. `getTableNamesWithLock()` extracts local names from `NameCacheValue`. `getTableNullable(String)` resolves the local name, performs `tables.get()` manual loading, and updates `tableIdToName` from `table.getId()`. `getTableNullable(long)` resolves the local name from the ID index and then calls `tables.get()`. Both `getTableForReplay(long)` and `getTableForReplay(String)` require initialized state and use `getIfPresent()` for the final object lookup. The string overload checks the exact key before consulting the cached names snapshot. If mode 2 replay sees a cold names snapshot but the object cache is still hot, it now falls back to a cache-only case-insensitive scan over hot object-cache keys. Replay misses do not synchronously enumerate remote tables or load table objects. `isTableExist()` no longer calls `listTableNames()` directly to populate a side map. For case-insensitive names, it resolves the remote name through the shared names resolver and then calls the catalog's `tableExist()` implementation. `getLocalTableName()` preserves the existing lower-case storage modes and uses `NameCacheValue` for case-insensitive original-name resolution. `getTableNamesValue()`, `resolveTableNameFromSnapshot()`, `listLocalTableNamesFromCache()`, and `getRemoteTableName()` are the table-level counterparts of the database-name helpers described above. `updateTableCache()` uses the same split semantics as the database side: names and object entries are updated only when already hot, while `tableIdToName` is always updated from the actual table ID. Normal `getTableNullable(long)` can therefore resolve the name and load a cold table object on demand. The event path and loader share the same deterministic table-ID rule; Replay still uses `getIfPresent()`. `invalidateTableCache()` removes the name from a hot snapshot, invalidates the object key, and removes matching ID mappings. `unregisterTable()` derives the local cache key with the same remote-to-local rules used by register/list and performs this local cleanup even when the object entry is cold or already evicted. Engine-specific cache invalidation is invoked only when the cached `ExternalTable` object is available. `invalidateAllTableCache()` clears all three structures. `registerTable()` and `unregisterTable()` now delegate to these centralized update/invalidation helpers while preserving their surrounding timestamp behavior. `resetMetaCacheNames()` still invalidates only table names. `gsonPostProcess()` reconstructs an empty runtime-only `tableIdToName` map. #### 7. `ExternalMetaCacheMgr` The `LegacyMetaCacheFactory` field, its construction, and the `legacyMetaCacheFactory()` accessor are removed because catalog and database objects now build `MetaCacheEntry` instances directly. `commonRefreshExecutor()` exposes the existing common refresh executor to those construction paths. It replaces the old factory accessor without creating a new thread pool or changing executor ownership. #### 8. `HMSExternalCatalog` `registerDatabase(long, String)` now delegates cache maintenance to `ExternalCatalog.updateDatabaseCache()` instead of calling `metaCache.updateCache()` directly. This centralizes names/object/ID ordering, hot-only names/object updates, always-on ID navigation maintenance, and the single ID source. The duplicate `Util.genIdByName()` call is removed because the helper uses `db.getId()`. #### 9. `AbstractExternalMetaCache` `newMetaCacheEntry()` now passes `MetaCacheEntry.defaultObjectStripeCount()` explicitly when constructing existing schema, partition, and other engine-specific entries. The new stripe configuration therefore applies consistently both to the database/table object caches migrated in this PR and to multi-key entries managed by this class. ## Release note Improve external metadata cache replay lookup and hot-path name resolution in FE. ## Check List (For Author) - Test: - [x] Regression test - [x] Unit Test - [x] Manual test (add detailed scripts or steps below) - [ ] No need to test (with reason) Manual test details: - Validate mutable name miss refresh semantics on a real Doris instance with `tools/manual_external_meta_cache_regression/verify_name_miss_refresh_mutable.sh` - Validate tableNames refresh non-blocking on a real Doris instance with `tools/manual_external_meta_cache_regression/verify_table_names_refresh_non_blocking.sh` - Validate schema refresh non-blocking on a real Doris instance with `tools/manual_external_meta_cache_regression/verify_schema_refresh_non_blocking.sh` - Validate lower-case mode 0, 1 and 2 SQL paths on a real Doris instance Unit test details: - `./run-fe-ut.sh --run org.apache.doris.datasource.metacache.MetaCacheEntryTest` - `./run-fe-ut.sh --run org.apache.doris.datasource.metacache.NameCacheValueTest,org.apache.doris.datasource.ExternalDatabaseTest,org.apache.doris.datasource.ExternalCatalogTest` - Behavior changed: - [ ] No - [x] Yes. Replay lookup, names snapshot hot-path lookup, and review clarifications are aligned with the refactor design. - Does this need documentation: - [x] No - [ ] Yes ## Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
