singhpk234 commented on code in PR #16891:
URL: https://github.com/apache/iceberg/pull/16891#discussion_r3468683675
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -368,6 +369,31 @@ public Identifier[] listTables(String[] namespace) {
.toArray(Identifier[]::new);
}
+ @Override
+ public TableSummary[] listTableSummaries(String[] namespace) {
+ // Build summaries directly from the catalog listings to avoid loading
every table, which the
+ // default TableCatalog.listTableSummaries implementation would do.
Iceberg tables are always
+ // reported as EXTERNAL (see SparkTable#properties), and views are
reported as VIEW.
+ List<TableSummary> summaries = Lists.newArrayList();
+
+ // Collect views first. Most catalogs return only tables from listTables,
but HiveCatalog with
+ // list-all-tables=true returns every metastore entry, including views.
De-duplicate against the
+ // view identifiers so a view is never also reported as a table.
+ Set<Identifier> viewIdents = Sets.newHashSet(listViews(namespace));
Review Comment:
Does spark supports showing view here , my understanding is not ?
https://github.com/apache/spark/commit/810a547e20e616bc3bd89e1caa8e2062818b5971#diff-76ba8535886252d11f5de158f6e465ea92dc07f3e2b3f6a976a5cb412fb19112R123
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -368,6 +369,31 @@ public Identifier[] listTables(String[] namespace) {
.toArray(Identifier[]::new);
}
+ @Override
+ public TableSummary[] listTableSummaries(String[] namespace) {
+ // Build summaries directly from the catalog listings to avoid loading
every table, which the
+ // default TableCatalog.listTableSummaries implementation would do.
Iceberg tables are always
+ // reported as EXTERNAL (see SparkTable#properties), and views are
reported as VIEW.
+ List<TableSummary> summaries = Lists.newArrayList();
+
+ // Collect views first. Most catalogs return only tables from listTables,
but HiveCatalog with
+ // list-all-tables=true returns every metastore entry, including views.
De-duplicate against the
+ // view identifiers so a view is never also reported as a table.
+ Set<Identifier> viewIdents = Sets.newHashSet(listViews(namespace));
+
+ for (Identifier ident : listTables(namespace)) {
Review Comment:
List table doesn't contain enough info in spark it seems like we do the
individual load ?
https://github.com/apache/spark/commit/810a547e20e616bc3bd89e1caa8e2062818b5971#diff-76ba8535886252d11f5de158f6e465ea92dc07f3e2b3f6a976a5cb412fb19112R128
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
