suxiaogang223 opened a new pull request, #69665:
URL: https://github.com/apache/doris/pull/69665
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
ALTER CATALOG properties, DROP CATALOG, and catalog rename can block the
serial journal replay thread while closing old connectors, contexts, thread
pools, access controllers, or Iceberg FileIO resources. A slow close delays
subsequent journal entries, including entries for unrelated catalogs.
Apply metadata changes and cache invalidation synchronously, capture the
retired instances, and close them through an FE resource reclaimer. The
reclaimer retains ownership before dispatch, handles rejected scheduling
without running cleanup on the caller, and exposes backlog, running, failure,
age, and duration metrics. Iceberg FileIO cleanup runs after the last table
reference is released and retains the catalog lease until cleanup finishes.
Bind connector cache retirement to the owning context, including lazy
siblings and late registration, so checkpoint and serving Envs with matching
catalog IDs cannot retire each other's connector caches. Clean up cache
registrations and filesystem resources when root connector construction fails.
Journal formats and propagation remain unchanged.
Scope: catalog initialization under the object lock, cache traversal, quota
deregistration, and row-count invalidation remain synchronous. The pending
cleanup registry has no hard capacity limit, and this change does not introduce
a universal query lease protocol.
### Release note
Reduce journal replay delays caused by closing retired external catalog
resources while preserving synchronous metadata and cache invalidation.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test
- [ ] No need to test or manual test
Validation on Fedora with JDK 17 and the published third-party prebuilt:
- FE build and packaging succeeded for all 74 selected reactor modules
using the original `build.sh --fe` with Maven build cache disabled. UI, Hive
UDF, and BE Java extension packaging were disabled. Checkstyle passed.
- `run-fe-ut.sh --run` passed all 113 tests across 10 affected test
classes: `ConnectorCacheOwnerTest`, `IcebergTableCacheTest`,
`DefaultConnectorContextCacheOwnershipTest`, `CatalogFactoryTest`,
`CatalogMgrTest`, `ExternalCatalogDeadlockTest`,
`ExternalCatalogReplayCleanupTest`, `ExternalCatalogTest`,
`ExternalResourceReclaimerTest`, and
`PluginDrivenExternalCatalogConcurrencyTest`. Zero failures, errors, or skipped
tests; Maven build cache disabled.
- `git diff --check` passed. Cluster regression tests were not run.
- Behavior changed:
- [ ] No.
- [x] Yes. Retired resources close asynchronously after synchronous
metadata and cache invalidation.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
