Noemi Pap-Takacs has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24091
Change subject: IMPALA-14823: Lazy load IcebergContentFileStore in Local Catalog mode ...................................................................... IMPALA-14823: Lazy load IcebergContentFileStore in Local Catalog mode This patch optimizes file metadata loading for Iceberg tables in the Local Catalog mode. Previously, loading an Iceberg table would eagerly fetch the entire list of content files from CatalogD. Serializing and sending `IcebergContentFileStore` caused significant overhead for metadata-only operations like DESCRIBE or SHOW TABLE STATS, which do not require file-level details. The `IcebergContentFileStore`, which holds the file list, is now lazily loaded. - The `TGetPartialCatalogObjectRequest` thrift structure and `IcebergTable.getPartialInfo` in CatalogD were updated to support requesting Iceberg metadata without content files (`DESCRIPTOR_ONLY`). - `LocalIcebergTable` now initializes with minimal metadata. Accessing `getContentFileStore()` (required for query planning) triggers a subsequent RPC to fetch the file list. - `FeIcebergTable.Utils.getTableStats` was optimized to use the Iceberg snapshot summary for file counts instead of counting loaded files, ensuring `SHOW TABLE STATS` remains lightweight. Testing: - Added `LocalIcebergContentFileStoreTest` unit test to verify that file loading is deferred until explicitly accessed. Generated by Gemini 3 Pro Change-Id: If305129a0f5a6e6fa102dfb5628c750ab5318e03 --- M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java M fe/src/main/java/org/apache/impala/catalog/local/MultiMetaProvider.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java A fe/src/test/java/org/apache/impala/catalog/local/LocalIcebergContentFileStoreTest.java M fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java 12 files changed, 166 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/24091/2 -- To view, visit http://gerrit.cloudera.org:8080/24091 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If305129a0f5a6e6fa102dfb5628c750ab5318e03 Gerrit-Change-Number: 24091 Gerrit-PatchSet: 2 Gerrit-Owner: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
