stevenzwu commented on code in PR #11948:
URL: https://github.com/apache/iceberg/pull/11948#discussion_r1917465682
##########
core/src/main/java/org/apache/iceberg/TableMetadata.java:
##########
@@ -1468,6 +1515,45 @@ public Builder setPreviousFileLocation(String
previousFileLocation) {
return this;
}
+ private Builder setRowLineage(Boolean newRowLineage) {
Review Comment:
for builder, arg can probably be primitive `boolean`. then we can save the
`if` check below
##########
core/src/test/java/org/apache/iceberg/TestRowLineageMetadata.java:
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.Assumptions.assumeThat;
+
+import java.io.File;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.iceberg.exceptions.ValidationException;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.FieldSource;
+
+public class TestRowLineageMetadata {
+
+ @TempDir private File tableDir = null;
+
+ private static final String TEST_LOCATION = "s3://bucket/test/location";
+
+ private static final Schema TEST_SCHEMA =
+ new Schema(
+ 7,
+ Types.NestedField.required(1, "x", Types.LongType.get()),
+ Types.NestedField.required(2, "y", Types.LongType.get(), "comment"),
+ Types.NestedField.required(3, "z", Types.LongType.get()));
+
+ private TableMetadata baseMetadata(int formatVersion) {
+ return TableMetadata.buildFromEmpty(formatVersion)
+ .enableRowLineage()
+ .addSchema(TEST_SCHEMA)
+ .setLocation(TEST_LOCATION)
+ .addPartitionSpec(PartitionSpec.unpartitioned())
+ .addSortOrder(SortOrder.unsorted())
+ .build();
+ }
+
+ @AfterEach
+ public void cleanup() {
+ TestTables.clearTables();
+ }
+
+ @ParameterizedTest
+ @FieldSource("org.apache.iceberg.TestHelpers#ALL_VERSIONS")
+ public void testRowLineageSupported(int formatVersion) {
+ if (formatVersion == TableMetadata.MIN_FORMAT_VERSION_ROW_LINEAGE) {
Review Comment:
the condition probably should be `>=` to work for future versions like `V4`
##########
core/src/main/java/org/apache/iceberg/SnapshotProducer.java:
##########
@@ -282,6 +283,29 @@ public Snapshot apply() {
throw new RuntimeIOException(e, "Failed to write manifest list file");
}
+ Long addedRows = null;
+ Long lastRowId = null;
+ if (base.rowLineageEnabled()) {
+ addedRows =
Review Comment:
nit: extract line 289-306 to a utility method
##########
core/src/main/java/org/apache/iceberg/TableProperties.java:
##########
@@ -388,4 +388,6 @@ private TableProperties() {}
public static final int ENCRYPTION_DEK_LENGTH_DEFAULT = 16;
public static final int ENCRYPTION_AAD_LENGTH_DEFAULT = 16;
+
+ public static final String ROW_LINEAGE = "row-lineage";
Review Comment:
nit: existing properties have `enabled` for boolean flags. maybe
`row-lineage-enabled`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
