morningman opened a new pull request, #68842:
URL: https://github.com/apache/doris/pull/68842
### What problem does this PR solve?
Issue Number: None
Related PR: #66717, #67726
Problem Summary:
**In short.** This ports four fixes from branch-4.1 #66717 (memory
governance for external metadata caches) that master's re-implementation
(#67726) does not have. Under a metadata cache weight limit, master still lets
the Iceberg SDK keep its own unbounded manifest cache, fetches the whole
snapshot history of REST `snapshot-loading-mode=refs` tables on every table
cache miss, and under-weighs cached Paimon tables by 2-5x because they keep
growing after admission. Catalogs without a weight limit are not affected.
**Background.** #66717 bounds the estimated retained memory of external
metadata caches with the FE-wide `external_meta_cache_max_weight`, the catalog
property `meta.cache.max-weight`, and per-entry
`meta.cache.<engine>.<entry>.max-weight`. A value is weighed once, when it is
admitted; if it does not fit, it is returned to the query but not cached.
master re-implemented this in #67726 on the connector cache framework, and
#68839 ports the 4.x tests. Comparing the two implementations found the gaps
below; branch-4.1 handles all of them.
**The problem, and what it cost.**
- **Iceberg SDK manifest cache outside the limit.** `IcebergCatalogFactory`
derives `io.manifest.cache-enabled=true` whenever
`meta.cache.iceberg.manifest.*` is enabled. That SDK cache keeps up to
`io.manifest.cache.max-total-bytes` (100 MB by default) of manifest content per
FileIO, outside every Doris limit, and Glue / S3 Tables catalogs have one
FileIO per table. A user who sets a limit gets a second, unbounded manifest
cache next to the bounded one.
- **Remote IO while weighing REST refs-mode tables.** A weight-limited
`IcebergTableCache` serializes the loaded metadata
(`TableMetadataParser.toJson`) and weighs it. Both call
`TableMetadata.snapshots()`, which in refs mode runs the catalog's supplier and
fetches every snapshot. So each table cache miss pays one extra full-history
REST load, exactly the cost refs mode exists to avoid, and the cached value
then holds the whole history.
- **Paimon tables grow after admission.** `paimon-table` is weighed from the
schema, location and catalog environment. A cached `FileStoreTable` then grows:
the first `latestSnapshot()` or scan builds its transient store (`RowType`
copies of the schema, a copy of the options), and field lookups build the four
lazy index maps of every `RowType`. Measured on Paimon 1.4.2, the grown graph
is 2-5x the admitted weight, so the heap the cache really holds can exceed
`MAX_WEIGHT` by that factor.
- **Paimon REST data tokens are not charged.** With data tokens,
`RESTCatalog` gives each table its own `RESTTokenFileIO`. The FileIO it
delegates to lives in Paimon's process-wide cache, but the vended token it
fetches on first data access (about 2 KB) belongs to the table and arrives
after admission.
**How this PR fixes it.**
- `IcebergConnector#hasMetaCacheWeightLimit` decides whether any limit
applies: the FE-global limit, the catalog limit, or the `max-weight` of one of
the entries the connector reads (`table`, `partition`, `manifest`,
`partition_view`, now one constant shared with the DDL validation). The three
catalog option builders take that flag, and `appendManifestCacheProperties`
does not derive `io.manifest.cache-enabled` under a limit. An explicit user
value still wins, as in branch-4.1.
- `IcebergCacheSizeEstimator.areSnapshotsLoaded` reads
`TableMetadata.snapshotsLoaded` (Iceberg exposes no accessor for it). A
weight-limited `TableOwner` checks it first and, if the list is still behind
its supplier, declines retention with
`incomplete_estimate:iceberg_snapshots_not_loaded` instead of serializing. The
statement keeps the live table, whose supplier still serves historical
snapshots on demand.
- `PaimonCacheSizeEstimator` reserves the post-admission growth of every
concrete `FileStoreTable` from its schema shape: a base, each field, each field
nested in a row, each child type of an array / map / multiset / vector, each
option and each key. The constants are calibrated against a retained-size
oracle that walks the grown graph (new test helper
`EstimatorCalibrationAssertions`). A table whose FileIO is a `RESTTokenFileIO`
additionally reserves 4 KB for its token.
**Results.**
| Case | Before | After |
|---|---|---|
| `meta.cache.iceberg.manifest.enable=true` plus any weight limit | SDK
manifest cache derived on, up to 100 MB per FileIO outside the limit | not
derived; `io.manifest.cache-enabled=true` still turns it on |
| Weight-limited REST catalog, `snapshot-loading-mode=refs`, table cache
miss | one extra remote load of the full snapshot list, full history retained |
no extra load; table not retained, statement reads the live table |
| Paimon table with 301 INT columns, after one scan | 35 KB admitted, 114 KB
retained | 134 KB admitted, 114 KB retained |
| Paimon table with a 200-field ROW column, after field lookups | 25 KB
admitted, 117 KB retained | 143 KB admitted, 117 KB retained |
| Paimon REST data-token table | token (about 2 KB) not charged | 4 KB
reserved |
Across 11 Paimon fixtures (append and primary-key tables, wide schemas, many
options, nested rows, arrays and maps of rows, multisets) the admitted weight
is now 1.2-1.7x the fully grown graph.
Not in this PR: wide schemas that exceed the 10,000-visit budget of
`ReflectiveObjectSizeEstimator.estimateComplete` (for example HMS tables with
about 1,000 primitive columns, or Paimon tables with about 3,400) are still
never admitted under a limit. The same budget is shared by several connectors
and the FE schema cache, so it needs its own change.
**Classes, and how they call each other.**
- `IcebergConnector`: `WEIGHTED_CACHE_ENTRIES`, `hasMetaCacheWeightLimit()`,
and passes the flag at the three catalog builds.
- `IcebergCatalogFactory`: `buildBaseCatalogProperties`,
`buildCatalogProperties` and `buildS3TablesCatalogProperties` take
`metaCacheWeightLimited` and stay pure.
- `IcebergCatalogProperties`: validates `max-weight` keys against the shared
entry list.
- `IcebergCacheSizeEstimator` / `IcebergTableCache.TableOwner`: snapshot
probe and the guard before serialization.
- `PaimonCacheSizeEstimator`: `estimateStoreGrowth` and
`estimateTableOwnedFileIO`.
```
IcebergConnector.createCatalog / createS3TablesCatalog / createDlfCatalog
└─ hasMetaCacheWeightLimit(): metaCache.hasEnclosingWeightLimit() ||
cacheSpec(entry).isWeightBounded()
└─ IcebergCatalogFactory.build*CatalogProperties(...,
metaCacheWeightLimited)
└─ appendManifestCacheProperties: derive io.manifest.cache-enabled
only when not limited
IcebergTableCache.borrow ─ loader ─ new TableOwner(table, ..., weighted)
└─ areSnapshotsLoaded(table) ? toJson + estimateTable :
incomplete("iceberg_snapshots_not_loaded")
PaimonMetaCacheCatalog "paimon-table" size estimator
└─ PaimonCacheSizeEstimator.estimateTable ─ estimateFileStoreTable (each
concrete table)
├─ estimateStoreGrowth(schema)
└─ estimateTableOwnedFileIO(fileIO)
```
Nothing changes for catalogs without a weight limit: the flag is false, and
the probe and the reservations only run when an entry is weighed. Independent
of #68839 (tests only); both add tests to `IcebergConnectorCacheTest` and
`IcebergTableCacheTest`, in different places.
Original author: @CalvinKirs
### Release note
Under an external metadata cache weight limit, Iceberg catalogs no longer
turn on the Iceberg SDK manifest content cache implicitly (set
`io.manifest.cache-enabled=true` to keep it), weight-limited Iceberg REST
catalogs using `snapshot-loading-mode=refs` no longer retain tables in the
table cache, and Paimon table cache weights include what a cached table grows
after admission.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- `run-fe-ut.sh --run` on `IcebergCatalogFactoryTest`,
`IcebergCatalogOptionsSnapshotTest`, `IcebergConnectorCacheTest`,
`IcebergTableCacheTest`, `PaimonCacheSizeEstimatorTest` (new) and
`PaimonMetaCacheCatalogTest`; reverting each production change turns exactly
its new tests red
- all unit tests of fe-connector-paimon (678) and
fe-connector-iceberg (1485): 0 failures
- `mvn checkstyle:check` on both modules: 0 violations
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
- See the release note. All three changes apply only to catalogs
with a metadata cache weight limit.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
