Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on PR #11947: URL: https://github.com/apache/iceberg/pull/11947#issuecomment-2605982667 Hi everyone, thanks for all the review and suggestions! Since there is an ongoing [discussion](https://github.com/apache/iceberg/pull/11947#discussion_r1917865349) on whether to add tes

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

RussellSpitzer commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1918815287 ## core/src/main/java/org/apache/iceberg/TableMetadataParser.java: ## @@ -372,6 +373,7 @@ public static TableMetadata fromJson(String metadataLocation, JsonNo

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1917873174 ## core/src/main/java/org/apache/iceberg/TableMetadataParser.java: ## @@ -372,6 +373,7 @@ public static TableMetadata fromJson(String metadataLocation, JsonNode node)

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1917865349 ## core/src/test/java/org/apache/iceberg/MetadataTestUtils.java: ## @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more co

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1917865349 ## core/src/test/java/org/apache/iceberg/MetadataTestUtils.java: ## @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more co

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

RussellSpitzer commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1917469252 ## core/src/test/java/org/apache/iceberg/MetadataTestUtils.java: ## @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

rdblue commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1917421391 ## core/src/test/java/org/apache/iceberg/MetadataTestUtils.java: ## @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more co

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

rdblue commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1917414530 ## core/src/test/java/org/apache/iceberg/TestSnapshotJson.java: ## @@ -56,13 +56,16 @@ public void testJsonConversion() throws IOException { @Test public void

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

rdblue commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1917413799 ## core/src/test/java/org/apache/iceberg/TestSnapshotJson.java: ## @@ -56,13 +56,16 @@ public void testJsonConversion() throws IOException { @Test public void

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

RussellSpitzer commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1917401890 ## core/src/test/java/org/apache/iceberg/TestSnapshotJson.java: ## @@ -37,13 +34,16 @@ public class TestSnapshotJson { @Test public void testJsonConver

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

rdblue commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1917410896 ## core/src/test/java/org/apache/iceberg/TestDataTaskParser.java: ## @@ -193,10 +193,26 @@ private DataTask createDataTask() { List snapshots = Arrays.

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

rdblue commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1917409546 ## core/src/test/java/org/apache/iceberg/MetadataTestUtils.java: ## @@ -0,0 +1,428 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more co

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

rdblue commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1917406983 ## core/src/test/java/org/apache/iceberg/MetadataTestUtils.java: ## @@ -0,0 +1,428 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more co

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

rdblue commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1917406478 ## core/src/test/java/org/apache/iceberg/MetadataTestUtils.java: ## @@ -0,0 +1,428 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more co

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

rdblue commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1917406002 ## core/src/test/java/org/apache/iceberg/MetadataTestUtils.java: ## @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more co

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

RussellSpitzer commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1917405165 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -160,31 +155,16 @@ public void testJsonConversion() throws Exception {

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

rdblue commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1917404736 ## core/src/test/java/org/apache/iceberg/MetadataTestUtils.java: ## @@ -0,0 +1,428 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more co

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

rdblue commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1917398167 ## core/src/test/java/org/apache/iceberg/MetadataTestUtils.java: ## @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more co

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

rdblue commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1917395711 ## core/src/main/java/org/apache/iceberg/TableMetadataParser.java: ## @@ -372,6 +373,7 @@ public static TableMetadata fromJson(String metadataLocation, JsonNode node)

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1916016403 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -314,31 +328,43 @@ public void testBackwardCompat() throws Exception { assertThat(metadata.

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1916012887 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -160,31 +177,26 @@ public void testJsonConversion() throws Exception { .build()

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1916012081 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -96,31 +97,47 @@ public class TestTableMetadata { public TableOperations ops = new LocalTab

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1916011232 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -96,31 +97,47 @@ public class TestTableMetadata { public TableOperations ops = new LocalTab

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1916010795 ## core/src/test/java/org/apache/iceberg/TestMetadataUpdateParser.java: ## @@ -364,15 +364,15 @@ public void testAddSnapshotToJson() throws IOException { String m

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

RussellSpitzer commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1915587999 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -357,43 +383,48 @@ public void testInvalidMainBranch() throws IOException { asser

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

RussellSpitzer commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1915586726 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -314,31 +328,43 @@ public void testBackwardCompat() throws Exception { assertThat(m

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

RussellSpitzer commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1915570223 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -160,31 +177,26 @@ public void testJsonConversion() throws Exception {

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

RussellSpitzer commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1915510199 ## core/src/test/java/org/apache/iceberg/TestSnapshotJson.java: ## @@ -42,8 +42,15 @@ public void testJsonConversion() throws IOException { String manifes

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

RussellSpitzer commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1915513619 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -96,31 +97,47 @@ public class TestTableMetadata { public TableOperations ops = new

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

RussellSpitzer commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1915512319 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -96,31 +97,47 @@ public class TestTableMetadata { public TableOperations ops = new

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

RussellSpitzer commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1915509758 ## core/src/test/java/org/apache/iceberg/TestSnapshotJson.java: ## @@ -61,8 +68,15 @@ public void testJsonConversionWithoutSchemaId() throws IOException {

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

RussellSpitzer commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1915508273 ## core/src/main/java/org/apache/iceberg/TableMetadataParser.java: ## @@ -352,6 +352,7 @@ public static TableMetadata fromJson(String metadataLocation, JsonNo

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1915460216 ## core/src/main/java/org/apache/iceberg/TableMetadataParser.java: ## @@ -352,6 +352,7 @@ public static TableMetadata fromJson(String metadataLocation, JsonNode node)

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1915460216 ## core/src/main/java/org/apache/iceberg/TableMetadataParser.java: ## @@ -352,6 +352,7 @@ public static TableMetadata fromJson(String metadataLocation, JsonNode node)

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1915429803 ## core/src/test/java/org/apache/iceberg/TestMetadataUpdateParser.java: ## @@ -364,15 +364,15 @@ public void testAddSnapshotToJson() throws IOException { String m

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

RussellSpitzer commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1915381785 ## core/src/test/java/org/apache/iceberg/TestMetadataUpdateParser.java: ## @@ -364,15 +364,15 @@ public void testAddSnapshotToJson() throws IOException {

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

RussellSpitzer commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1915381785 ## core/src/test/java/org/apache/iceberg/TestMetadataUpdateParser.java: ## @@ -364,15 +364,15 @@ public void testAddSnapshotToJson() throws IOException {

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

RussellSpitzer commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1915347026 ## core/src/test/java/org/apache/iceberg/MetadataTestUtils.java: ## @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

RussellSpitzer commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1915373560 ## core/src/test/java/org/apache/iceberg/MetadataTestUtils.java: ## @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

RussellSpitzer commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1915353274 ## core/src/main/java/org/apache/iceberg/TableMetadataParser.java: ## @@ -352,6 +352,7 @@ public static TableMetadata fromJson(String metadataLocation, JsonNo

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

RussellSpitzer commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1915347026 ## core/src/test/java/org/apache/iceberg/MetadataTestUtils.java: ## @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1915306849 ## core/src/test/java/org/apache/iceberg/MetadataTestUtils.java: ## @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more co

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1915306849 ## core/src/test/java/org/apache/iceberg/MetadataTestUtils.java: ## @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more co

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

nastra commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1914536361 ## core/src/test/java/org/apache/iceberg/MetadataTestUtils.java: ## @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more co

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

nastra commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1914536816 ## core/src/test/java/org/apache/iceberg/MetadataTestUtils.java: ## @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more co

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1913661284 ## core/src/test/resources/TableMetadataV2StatisticsFiles.json: ## Review Comment: It is referenced in the following test: https://github.com/apache/iceberg/b

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1913661284 ## core/src/test/resources/TableMetadataV2StatisticsFiles.json: ## Review Comment: It is still referenced in the following test: https://github.com/apache/ice

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1913661284 ## core/src/test/resources/TableMetadataV2StatisticsFiles.json: ## Review Comment: It is referenced in the following test: https://github.com/apache/iceberg/b

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

Fokko commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1913079100 ## core/src/test/resources/TableMetadataV2StatisticsFiles.json: ## Review Comment: I don't see this file being referenced anymore. I that correct? -- This is a

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1911913284 ## core/src/test/java/org/apache/iceberg/MetadataTestUtils.java: ## @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more co

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1911912918 ## core/src/main/java/org/apache/iceberg/TableMetadataParser.java: ## @@ -240,6 +242,13 @@ public static void toJson(TableMetadata metadata, JsonGenerator generator)

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1911912918 ## core/src/main/java/org/apache/iceberg/TableMetadataParser.java: ## @@ -240,6 +242,13 @@ public static void toJson(TableMetadata metadata, JsonGenerator generator)

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

flyrain commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1911495617 ## core/src/main/java/org/apache/iceberg/TableMetadataParser.java: ## @@ -240,6 +242,13 @@ public static void toJson(TableMetadata metadata, JsonGenerator generator)

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on PR #11947: URL: https://github.com/apache/iceberg/pull/11947#issuecomment-2584041893 cc @RussellSpitzer @flyrain May I ask for your help to review this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on code in PR #11947: URL: https://github.com/apache/iceberg/pull/11947#discussion_r1911280517 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -949,6 +972,8 @@ private Builder(int formatVersion) { this.schemasById = Maps.newHashMap();

[PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX opened a new pull request, #11947: URL: https://github.com/apache/iceberg/pull/11947 Fixes #10764 - Add new fields: `row-lineage` and `next-row-id` to TableMetadata - Update the parser to parse v3 new fields - Add new unit tests for V3 metadata - Refactor TableMetadata

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX closed pull request #11730: Core: Parsing and Writing Tests for V3 Metadata URL: https://github.com/apache/iceberg/pull/11730 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX commented on code in PR #11730: URL: https://github.com/apache/iceberg/pull/11730#discussion_r1875481834 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -307,6 +311,13 @@ public String toString() { Preconditions.checkArgument( metadataFile

[PR] Core: Parsing and Writing Tests for V3 Metadata [iceberg]

HonahX opened a new pull request, #11730: URL: https://github.com/apache/iceberg/pull/11730 Fixes #10764 - Add new fields: `row-lineage` and `next-row-id` to TableMetadata - Update the parser to parse v3 new fields - Add new unit tests for V3 metadata - Refactor TableMetadata