Hello Zoltan Borok-Nagy, Noemi Pap-Takacs, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24177
to look at the new patch set (#6).
Change subject: IMPALA-14806: Cache IcebergContentFileStore
......................................................................
IMPALA-14806: Cache IcebergContentFileStore
This change was split out from larger change
https://gerrit.cloudera.org/#/c/24049/
With this patch Iceberg file descriptors are cached in
IcebergFileContentStore instead of TPartialTableInfo.
TPartialInfo is still cached, but without file info.
Based on experiments this both makes mem need smaller and
planning much faster by not needing to convert TPartialTableInfo
to IcebergFileContentStore in each query. The drawback
is that +1 GetPartialCatalogObject RPC is needed to
fetch an Iceberg table from catalog.
Examples are for 1M files, 25K partitions Iceberg table
on my dev machine.
Mem:
Before:
558 MB org.apache.impala.thrift.TPartialTableInfo
After:
449 MB org.apache.impala.catalog.IcebergFileContentStore
3 MB org.apache.impala.thrift.TPartialTableInfo
Speed:
DESCRIBE t: 1.7s->10ms
EXPLAIN SELECT * FROM t: 5.5s->4.7s
EXPLAIN SELECT * FROM t WHERE part_col=100: 1.5s->0.7s
Change-Id: Ic93eeb3a90a280a7138672affd71ac6f87df71a1
---
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java
M fe/src/main/java/org/apache/impala/catalog/local/MultiMetaProvider.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M
fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java
12 files changed, 188 insertions(+), 41 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/24177/6
--
To view, visit http://gerrit.cloudera.org:8080/24177
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic93eeb3a90a280a7138672affd71ac6f87df71a1
Gerrit-Change-Number: 24177
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
