[I] Bug: Flink data loss after failed to refresh table [iceberg]

Aireed opened a new issue, #9753: URL: https://github.com/apache/iceberg/issues/9753 ### Apache Iceberg version 1.1.0 ### Query engine Flink ### Please describe the bug 🐞 In SnapshotProducer::commit call， if `ops.refresh()` failed, table's metadata won't

Re: [PR] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

tomtongue commented on code in PR #9670: URL: https://github.com/apache/iceberg/pull/9670#discussion_r1494143891 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java: ## @@ -1181,7 +1170,7 @@ public synchronized void testDeleteWithSnap

Re: [PR] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

tomtongue commented on code in PR #9670: URL: https://github.com/apache/iceberg/pull/9670#discussion_r1494143891 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java: ## @@ -1181,7 +1170,7 @@ public synchronized void testDeleteWithSnap

Re: [PR] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

tomtongue commented on code in PR #9670: URL: https://github.com/apache/iceberg/pull/9670#discussion_r1494143891 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java: ## @@ -1181,7 +1170,7 @@ public synchronized void testDeleteWithSnap

Re: [I] ValidationException: Missing required files to delete [iceberg]

bk-mz commented on issue #9741: URL: https://github.com/apache/iceberg/issues/9741#issuecomment-1952007211 @fengguangyuan >but at some point a Compact task committed before the Overwrite task trying to call the internal method to commit metadata, obviously these overwrite tasks will

Re: [PR] Build: Bump org.immutables:value from 2.10.0 to 2.10.1 [iceberg]

Fokko merged PR #9749: URL: https://github.com/apache/iceberg/pull/9749 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.24.0 to 2.24.5 [iceberg]

Fokko merged PR #9743: URL: https://github.com/apache/iceberg/pull/9743 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Build: Bump io.netty:netty-buffer from 4.1.68.Final to 4.1.107.Final [iceberg]

Fokko merged PR #9744: URL: https://github.com/apache/iceberg/pull/9744 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Build: Bump datamodel-code-generator from 0.25.3 to 0.25.4 [iceberg]

Fokko merged PR #9742: URL: https://github.com/apache/iceberg/pull/9742 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] refactor: remove support of manifest list format as a list of file path [iceberg-rust]

Fokko merged PR #201: URL: https://github.com/apache/iceberg-rust/pull/201 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] refactor: remove unwraps [iceberg-rust]

Fokko merged PR #196: URL: https://github.com/apache/iceberg-rust/pull/196 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Fix: add required rust version in cargo.toml [iceberg-rust]

Fokko merged PR #193: URL: https://github.com/apache/iceberg-rust/pull/193 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [I] Tracking: Reading iceberg tables. [iceberg-rust]

liurenjie1024 commented on issue #123: URL: https://github.com/apache/iceberg-rust/issues/123#issuecomment-1952067074 Hi, @sdd > If you are aiming just to have table reads _working_ first, and optimizing them afterwards, then #124 is not completely necessary to do at this stage?

Re: [PR] Fix the REST spec version [iceberg-rust]

nastra merged PR #198: URL: https://github.com/apache/iceberg-rust/pull/198 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Spec: Clarify multi-arg transform behavior for different versions [iceberg]

advancedxy commented on code in PR #9661: URL: https://github.com/apache/iceberg/pull/9661#discussion_r1494418622 ## format/spec.md: ## @@ -1134,10 +1148,9 @@ In some cases partition specs are stored using only the field list instead of th The `field-id` property was added f

Re: [PR] refactor: rm async_trait and add trait_variant [iceberg-rust]

liurenjie1024 commented on PR #186: URL: https://github.com/apache/iceberg-rust/pull/186#issuecomment-1952283128 cc @odysa Should we close this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

tomtongue commented on code in PR #9670: URL: https://github.com/apache/iceberg/pull/9670#discussion_r1494143891 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java: ## @@ -1181,7 +1170,7 @@ public synchronized void testDeleteWithSnap

Re: [PR] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

tomtongue commented on code in PR #9670: URL: https://github.com/apache/iceberg/pull/9670#discussion_r1494484011 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java: ## @@ -1181,7 +1170,7 @@ public synchronized void testDeleteWithSnap

Re: [PR] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

tomtongue commented on code in PR #9670: URL: https://github.com/apache/iceberg/pull/9670#discussion_r1494484011 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java: ## @@ -1181,7 +1170,7 @@ public synchronized void testDeleteWithSnap

Re: [PR] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

tomtongue commented on code in PR #9670: URL: https://github.com/apache/iceberg/pull/9670#discussion_r1494484011 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java: ## @@ -1181,7 +1170,7 @@ public synchronized void testDeleteWithSnap

Re: [PR] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

nastra commented on code in PR #9670: URL: https://github.com/apache/iceberg/pull/9670#discussion_r1494552868 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkRowLevelOperationsTestBase.java: ## @@ -338,7 +333,10 @@ protected void sleep(long

[PR] docs: Add missing release steps [iceberg-python]

Fokko opened a new pull request, #443: URL: https://github.com/apache/iceberg-python/pull/443 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

Re: [PR] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

tomtongue commented on code in PR #9670: URL: https://github.com/apache/iceberg/pull/9670#discussion_r1494555975 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkRowLevelOperationsTestBase.java: ## @@ -338,7 +333,10 @@ protected void sleep(lo

Re: [I] refactor: Remove support of manifest list format as a list of file paths. [iceberg-rust]

Dysprosium0626 commented on issue #158: URL: https://github.com/apache/iceberg-rust/issues/158#issuecomment-1952471592 Hi, you may close this issue now! (BTW It looks like some of the issues like https://github.com/apache/iceberg-rust/issues/192 could be close too) -- This is an automa

Re: [PR] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

nastra commented on code in PR #9670: URL: https://github.com/apache/iceberg/pull/9670#discussion_r1494620947 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java: ## @@ -1052,9 +1039,10 @@ public synchronized void testDeleteWithSeria

Re: [I] Null values in metadata_log_entries after replacing Table [iceberg]

findinpath commented on issue #9723: URL: https://github.com/apache/iceberg/issues/9723#issuecomment-1952572152 @oneonestar i'm guessing that your Trino developments from https://github.com/trinodb/trino/pull/20410 can continue now because it is clear that the inconsistency found for `CREAT

Re: [PR] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

tomtongue commented on code in PR #9670: URL: https://github.com/apache/iceberg/pull/9670#discussion_r1494655518 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java: ## @@ -1052,9 +1039,10 @@ public synchronized void testDeleteWithSe

Re: [PR] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

tomtongue commented on code in PR #9670: URL: https://github.com/apache/iceberg/pull/9670#discussion_r1494687079 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java: ## @@ -1052,9 +1039,10 @@ public synchronized void testDeleteWithSe

Re: [I] Parallel Table.append [iceberg-python]

Fokko commented on issue #428: URL: https://github.com/apache/iceberg-python/issues/428#issuecomment-1952692768 Do we know if this is for data generation, or also when writing? In the end, it would be good to be able to split the data into multiple files. The MacBooks have huge IO, so it mi

Re: [I] Add `rust-version` metadata in `Cargo.toml` [iceberg-rust]

liurenjie1024 closed issue #192: Add `rust-version` metadata in `Cargo.toml` URL: https://github.com/apache/iceberg-rust/issues/192 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [I] refactor: Remove support of manifest list format as a list of file paths. [iceberg-rust]

liurenjie1024 closed issue #158: refactor: Remove support of manifest list format as a list of file paths. URL: https://github.com/apache/iceberg-rust/issues/158 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

[PR] Core: Don't fail if catalog doesn't support views [iceberg]

nastra opened a new pull request, #9754: URL: https://github.com/apache/iceberg/pull/9754 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

RussellSpitzer commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1494773064 ## .palantir/revapi.yml: ## @@ -873,6 +873,10 @@ acceptedBreaks: new: "method void org.apache.iceberg.encryption.Ciphers::()" justification: "Stati

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

RussellSpitzer commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1494774386 ## api/src/main/java/org/apache/iceberg/actions/RewriteManifests.java: ## @@ -44,6 +45,16 @@ public interface RewriteManifests */ RewriteManifests rewrite

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

RussellSpitzer commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1494775357 ## api/src/main/java/org/apache/iceberg/actions/RewriteManifests.java: ## @@ -44,6 +45,16 @@ public interface RewriteManifests */ RewriteManifests rewrite

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

RussellSpitzer commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1494775923 ## api/src/main/java/org/apache/iceberg/actions/RewriteManifests.java: ## @@ -44,6 +45,16 @@ public interface RewriteManifests */ RewriteManifests rewrite

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

RussellSpitzer commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1494780178 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -90,13 +93,45 @@ public class RewriteManifestsSparkAct

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

RussellSpitzer commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1494777993 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -90,13 +93,45 @@ public class RewriteManifestsSparkAct

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

RussellSpitzer commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1494782065 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -90,13 +93,45 @@ public class RewriteManifestsSparkAct

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

RussellSpitzer commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1494784489 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -208,7 +243,15 @@ private RewriteManifests.Result rew

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1494784637 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

RussellSpitzer commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1494787146 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -84,6 +89,7 @@ import org.junit.jupiter.api.TestTempla

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1494791973 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1494793331 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1494791973 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

RussellSpitzer commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1494799026 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +472,164 @@ public void testRewriteLargeManifest

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1494800063 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

RussellSpitzer commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1494800722 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +472,164 @@ public void testRewriteLargeManifest

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1494791973 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1494804943 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1494791973 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1494805453 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

RussellSpitzer commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1494805599 ## api/src/main/java/org/apache/iceberg/actions/RewriteManifests.java: ## @@ -44,6 +45,16 @@ public interface RewriteManifests */ RewriteManifests rewrite

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

jackye1995 commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1494823068 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1616,12 @@ components: type: string example: [ "accounting", "tax" ] +NextPageToken: +

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

jackye1995 commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1494823068 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1616,12 @@ components: type: string example: [ "accounting", "tax" ] +NextPageToken: +

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

jackye1995 commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1494832304 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1616,12 @@ components: type: string example: [ "accounting", "tax" ] +NextPageToken: +

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

jackye1995 commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1494832304 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1616,12 @@ components: type: string example: [ "accounting", "tax" ] +NextPageToken: +

Re: [PR] Core: Don't fail if catalog doesn't support views [iceberg]

rdblue commented on code in PR #9754: URL: https://github.com/apache/iceberg/pull/9754#discussion_r1494846660 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -717,8 +718,15 @@ public Transaction createTransaction() { @Override public Tran

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

rdblue commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1494844317 ## api/src/main/java/org/apache/iceberg/expressions/Literals.java: ## @@ -501,18 +531,22 @@ public Literal to(Type type) { return (Literal) new TimeLiteral(t

Re: [PR] Core: Don't fail if catalog doesn't support views [iceberg]

nastra commented on code in PR #9754: URL: https://github.com/apache/iceberg/pull/9754#discussion_r1494850470 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -717,8 +718,15 @@ public Transaction createTransaction() { @Override public Tran

Re: [PR] Allow creating metadata tables based on SerializableTable instances [iceberg]

pvary merged PR #9735: URL: https://github.com/apache/iceberg/pull/9735 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Allow creating metadata tables based on SerializableTable instances [iceberg]

pvary commented on PR #9735: URL: https://github.com/apache/iceberg/pull/9735#issuecomment-1952897514 Thanks @szehon-ho for the review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] Core: Don't fail if catalog doesn't support views [iceberg]

amogh-jahagirdar commented on code in PR #9754: URL: https://github.com/apache/iceberg/pull/9754#discussion_r1494852170 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -717,8 +718,15 @@ public Transaction createTransaction() { @Override p

Re: [PR] Core: Don't fail if catalog doesn't support views [iceberg]

nastra commented on code in PR #9754: URL: https://github.com/apache/iceberg/pull/9754#discussion_r1494858016 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -717,8 +718,15 @@ public Transaction createTransaction() { @Override public Tran

Re: [I] Null values in metadata_log_entries after replacing Table [iceberg]

nastra commented on issue #9723: URL: https://github.com/apache/iceberg/issues/9723#issuecomment-1952909031 > @amogh-jahagirdar > > After taken a closer look, `latest_schema_id` behaves strangely for ALTER TABLE. ALTER TABLE updates the metadata file and its `current-schema-id`, but

Re: [I] Parallel Table.append [iceberg-python]

kevinjqliu commented on issue #428: URL: https://github.com/apache/iceberg-python/issues/428#issuecomment-1952910799 It was for data generation only. I can't seem to reproduce the parallelism issue for `append`, probably due to MacBook's huge IO. -- This is an automated message f

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1494861705 ## .palantir/revapi.yml: ## @@ -873,6 +873,10 @@ acceptedBreaks: new: "method void org.apache.iceberg.encryption.Ciphers::()" justification: "Static util

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1494862710 ## api/src/main/java/org/apache/iceberg/actions/RewriteManifests.java: ## @@ -44,6 +45,16 @@ public interface RewriteManifests */ RewriteManifests rewriteIf(Pre

Re: [I] Parallel Table.append [iceberg-python]

kevinjqliu commented on issue #428: URL: https://github.com/apache/iceberg-python/issues/428#issuecomment-1952912276 > we already have a bin-packing algorithm in the code @Fokko can you point me to that? I couldn't find it -- This is an automated message from the Apache Git Service

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1494866526 ## api/src/main/java/org/apache/iceberg/actions/RewriteManifests.java: ## @@ -44,6 +45,16 @@ public interface RewriteManifests */ RewriteManifests rewriteIf(Pre

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1494866876 ## api/src/main/java/org/apache/iceberg/actions/RewriteManifests.java: ## @@ -44,6 +45,16 @@ public interface RewriteManifests */ RewriteManifests rewriteIf(Pre

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1494868254 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -90,13 +93,45 @@ public class RewriteManifestsSparkAction

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1494868461 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -90,13 +93,45 @@ public class RewriteManifestsSparkAction

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1494869944 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -208,7 +243,15 @@ private RewriteManifests.Result rewriteMa

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1494870105 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -84,6 +89,7 @@ import org.junit.jupiter.api.TestTemplate; i

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1494871033 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +472,164 @@ public void testRewriteLargeManifestsParti

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1494872800 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +472,164 @@ public void testRewriteLargeManifestsParti

Re: [PR] refactor: rm async_trait and add trait_variant [iceberg-rust]

odysa commented on PR #186: URL: https://github.com/apache/iceberg-rust/pull/186#issuecomment-1952945242 @liurenjie1024 Yes, we can close this and reopen it after they release the rewrite feature. -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

jackye1995 commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1494832304 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1616,12 @@ components: type: string example: [ "accounting", "tax" ] +NextPageToken: +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1494888294 ## open-api/rest-catalog-open-api.yaml: ## @@ -2574,7 +2810,7 @@ components: allOf: - $ref: "#/components/schemas/TableRequirement" description: -

Re: [PR] Spec, Core: add a new task-type field to task JSON serialization. add data task JSON serialization spec and imp. [iceberg]

emkornfield commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1494889470 ## format/spec.md: ## @@ -1237,17 +1237,36 @@ Content file (data or delete) is serialized as a JSON object according to the fo | **`equality-ids`** |`JSON l

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

jackye1995 commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1494891220 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1616,12 @@ components: type: string example: [ "accounting", "tax" ] +NextPageToken: +

Re: [PR] docs: Add missing release steps [iceberg-python]

Fokko merged PR #443: URL: https://github.com/apache/iceberg-python/pull/443 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Parallel Table.append [iceberg-python]

Fokko commented on issue #428: URL: https://github.com/apache/iceberg-python/issues/428#issuecomment-1952962173 @kevinjqliu It is under [utils/bin_packing.py](https://github.com/apache/iceberg-python/blob/main/pyiceberg/utils/bin_packing.py). -- This is an automated message from the Apach

Re: [I] Parallel Table.append [iceberg-python]

kevinjqliu commented on issue #428: URL: https://github.com/apache/iceberg-python/issues/428#issuecomment-1952966533 thanks! I found it, had to fuzzy search in vscode :) Here's an example of bin-packing an Arrow table. https://colab.research.google.com/drive/1FM8mdr4j5KgsjBYmsp9_

Re: [I] Parallel Table.append [iceberg-python]

kevinjqliu commented on issue #428: URL: https://github.com/apache/iceberg-python/issues/428#issuecomment-1952969872 Integrating this with the write path, I have 2 approaches 1. refactoring `write_file` so that it can write multiple parquet files. This means 1 `WriteTask` can produce

Re: [I] Parallel Table.append [iceberg-python]

kevinjqliu commented on issue #428: URL: https://github.com/apache/iceberg-python/issues/428#issuecomment-195299 #444 something like this. wrote out 3 files ``` 0-0-a61f9655-0d76-45ca-b85d-4d8dc8dbcbd9.parquet 0-1-a61f9655-0d76-45ca-b85d-4d8dc8dbcbd9.parquet 00

[I] Create table from plain Parquet files [iceberg-python]

Fokko opened a new issue, #445: URL: https://github.com/apache/iceberg-python/issues/445 ### Feature Request / Improvement Today we can write to tables. During the write process we make sure that the schema is correct, and we collect column statistics during the write. It would be co

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1494914340 ## open-api/rest-catalog-open-api.yaml: ## @@ -2800,6 +3053,59 @@ components: additionalProperties: type: string +PlanContext: + type:

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1494916634 ## open-api/rest-catalog-open-api.yaml: ## @@ -2800,6 +3053,59 @@ components: additionalProperties: type: string +PlanContext: + type:

Re: [PR] OpenAPI: Add AppendDataFileUpdate to TableUpdate for rest appends [iceberg]

jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1494917690 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1494918141 ## open-api/rest-catalog-open-api.yaml: ## @@ -2800,6 +3053,59 @@ components: additionalProperties: type: string +PlanContext: + type:

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1494919511 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2165,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1494919946 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1494920473 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1494920689 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1494924023 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [I] Parallel Table.append [iceberg-python]

kevinjqliu commented on issue #428: URL: https://github.com/apache/iceberg-python/issues/428#issuecomment-1953010526 Oh interesting, the input is 1M records, 685.46 MB in memory. We bin-pack the Arrow representation into 256MB chunks (`['224.61 MB', '236.23 MB', '224.62 MB']`), but writing

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1494924550 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

rdblue commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1494925666 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile' +

  1   2   3   >