szehon-ho commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r4021505326
##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -19,20 +19,43 @@
package org.apache.iceberg;
import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.exceptions.NotFoundException;
import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.FileIO;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
import org.apache.iceberg.types.Types;
import org.apache.iceberg.util.SnapshotUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+/**
+ * A {@link Table} implementation that exposes a table's metadata log as rows.
+ *
+ * <p>Each row represents a historical or current metadata file and includes
the snapshot details
+ * and table properties recorded in that file. The current metadata is
included as the latest row.
+ *
+ * <p>Queries that reference {@code properties} read each retained historical
metadata file. These
+ * additional reads are skipped when {@code properties} is not referenced, and
the already loaded
+ * current metadata is reused.
+ */
public class MetadataLogEntriesTable extends BaseMetadataTable {
+ private static final Logger LOG =
LoggerFactory.getLogger(MetadataLogEntriesTable.class);
+
+ private static final int PROPERTIES_FIELD_ID = 6;
Review Comment:
Could we define `properties` as a `Types.NestedField` constant and use
`PROPERTIES.fieldId()` here? That keeps the field definition and projection
check together, while making a separate static assertion unnecessary. A
positive `shouldLoadProperties` boolean may also read more clearly.
##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -119,6 +150,33 @@ private static StaticDataTask.Row metadataLogEntryToRow(
// latest snapshot in this file corresponding to the log entry
latestSnapshotId,
latestSnapshot != null ? latestSnapshot.schemaId() : null,
- latestSnapshot != null ? latestSnapshot.sequenceNumber() : null);
+ latestSnapshot != null ? latestSnapshot.sequenceNumber() : null,
+ properties);
+ }
+
+ private static Map<String, String> loadTableProperties(
+ TableMetadata.MetadataLogEntry metadataLogEntry,
+ FileIO io,
+ TableMetadata current,
+ boolean skipPropertiesLoad) {
+
+ // Avoid loading metadata file when properties are not projected.
+ if (skipPropertiesLoad) {
+ return null;
+ }
+
+ // Reuse the already loaded current metadata.
+ if (metadataLogEntry.file().equals(current.metadataFileLocation())) {
+ return current.properties();
+ }
+
+ try {
+ return TableMetadataParser.read(io,
metadataLogEntry.file()).properties();
Review Comment:
Could we prefetch historical properties with `Tasks.foreach`, using the
scan’s planning executor when parallel planning is enabled? Results could be
stored by metadata-file location and rows assembled afterward in their original
order. This would parallelize remote reads while retaining Iceberg’s existing
executor controls and error handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
