github-actions[bot] commented on code in PR #69983:
URL: https://github.com/apache/doris/pull/69983#discussion_r4446153215
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalMetaCache.java:
##########
@@ -1167,27 +1189,41 @@ private IcebergSnapshotCacheValue
loadSnapshotProjection(
}
try {
IcebergSnapshot latestIcebergSnapshot =
IcebergUtils.getLatestIcebergSnapshot(projectionTable);
- IcebergPartitionInfo icebergPartitionInfo;
- // The projection and its partition eligibility must be classified
from the same
- // retained table generation. The no-argument method performs
another cache lookup,
- // which can cross a concurrent refresh/reset boundary.
+ // Ordinary scans and SQL-cache versions need only this frozen
metadata generation.
+ // Partition consumers load the separate, generation-keyed and
budgeted view on demand.
+ IcebergPartitionInfo icebergPartitionInfo =
IcebergPartitionInfo.empty();
boolean validRelatedTable =
IcebergUtils.isValidRelatedTable(projectionTable);
- if (!validRelatedTable) {
- icebergPartitionInfo = IcebergPartitionInfo.empty();
- } else {
- icebergPartitionInfo =
IcebergUtils.loadPartitionInfo(dorisTable, projectionTable,
- latestIcebergSnapshot.getSnapshotId(),
latestIcebergSnapshot.getSchemaId(), authenticator,
- enableMappingVarbinary, enableMappingTimestampTz);
- }
Optional<Map<Integer, List<String>>> nameMapping =
IcebergUtils.getNameMapping(projectionTable);
- return isolateForQueries
+ IcebergSnapshotCacheValue value = isolateForQueries
? new IcebergSnapshotCacheValue(
icebergPartitionInfo, latestIcebergSnapshot,
nameMapping,
retainedTable, retainedCurrentSnapshotJson)
: new IcebergSnapshotCacheValue(
icebergPartitionInfo, latestIcebergSnapshot,
nameMapping,
retainedTable);
+ if (validRelatedTable) {
+ NameMapping tableIdentity = dorisTable.getOrBuildNameMapping();
+ String displayName = dorisTable.getCatalog().getName() + "." +
dorisTable.getDbName()
+ + "." + dorisTable.getName();
+ Optional<IcebergSnapshotEntryKey> key =
IcebergSnapshotEntryKey.tryCreate(tableIdentity, retainedTable);
+ MetaCacheEntry<IcebergSnapshotEntryKey, IcebergPartitionInfo>
partitions =
+ key.isPresent() ?
partitionEntry.get(key.get().getNameMapping().getCtlId()) : null;
Review Comment:
[P1] Do not require a live partition-cache catalog group while loading a
retained Iceberg projection
`loadSnapshotProjection` now publishes an empty partition view and calls
`partitionEntry.get(catalogId)` before installing the lazy loader. A statement
or rewrite can still hold a retained table/writable lease while
`IcebergExternalCatalog.resetToUninitialized` / `removeCatalogByEngine` retires
the catalog group; `AbstractExternalMetaCache.entry` then throws before the
captured loader is installed. If the projection was created before reset, its
first partition read instead reaches `IcebergUtils.loadPartitionInfo` and the
new `schemaEntry.get(catalogId)` lookup fails for the same reason. Pinned reads
therefore fail even though the retained table generation is intended to remain
usable. Capture reset-safe generation-bound partition/schema dependencies, or
retain the loaded partition view for such projections, and skip current-group
lookups for session-catalog/direct-load paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
