Re: [PR] Core: FileSystemCatalogTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1651106651 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -355,71 +475,90 @@ int findVersion() { * an attempt will be made to delete the sou

Re: [PR] View Spec implementation [iceberg-rust]

c-thiel commented on code in PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#discussion_r1651128747 ## crates/iceberg/testdata/view_metadata/ViewMetadataV2Valid.json: ## @@ -0,0 +1,58 @@ +{ Review Comment: Ok, thanks for the feedback @nastra. I'll add some more

Re: [PR] Core: FileSystemCatalogTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1651106651 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -355,71 +475,90 @@ int findVersion() { * an attempt will be made to delete the sou

Re: [PR] Core: FileSystemCatalogTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1651099384 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -368,58 +431,63 @@ private void renameToFinal(FileSystem fs, Path src, Path dst, int

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

stevenzwu commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1648221004 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/JVMBasedLockFactory.java: ## @@ -0,0 +1,83 @@ +/* + * Licensed to the Apache Softwa

Re: [PR] Core: FileSystemCatalogTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1651106651 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -355,71 +475,90 @@ int findVersion() { * an attempt will be made to delete the sou

Re: [PR] Core: FileSystemCatalogTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1651099384 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -368,58 +431,63 @@ private void renameToFinal(FileSystem fs, Path src, Path dst, int

Re: [PR] Core: FileSystemCatalogTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1651082514 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -368,58 +431,63 @@ private void renameToFinal(FileSystem fs, Path src, Path dst, int

Re: [PR] Core: FileSystemCatalogTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1651072649 ## core/src/test/java/org/apache/iceberg/hadoop/TestHadoopCommits.java: ## @@ -206,6 +210,139 @@ public void testFailedCommit() throws Exception { Assertions.asser

Re: [PR] Core: FileSystemCatalogTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1651047933 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -370,58 +402,70 @@ private void renameToFinal(FileSystem fs, Path src, Path dst, int

Re: [PR] DynConstructors cleanup [iceberg]

findepi commented on code in PR #10542: URL: https://github.com/apache/iceberg/pull/10542#discussion_r1651013839 ## common/src/test/java/org/apache/iceberg/common/TestDynConstructors.java: ## @@ -0,0 +1,75 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [I] Field comments are not written for timestamp field [iceberg]

ariksa1 commented on issue #4212: URL: https://github.com/apache/iceberg/issues/4212#issuecomment-2186546553 Hi, there is any update regarding the issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] DynConstructors cleanup [iceberg]

findepi commented on code in PR #10542: URL: https://github.com/apache/iceberg/pull/10542#discussion_r1651007712 ## common/src/test/java/org/apache/iceberg/common/TestDynConstructors.java: ## @@ -0,0 +1,75 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] DynConstructors cleanup [iceberg]

nastra commented on code in PR #10542: URL: https://github.com/apache/iceberg/pull/10542#discussion_r1650976468 ## common/src/test/java/org/apache/iceberg/common/TestDynConstructors.java: ## @@ -0,0 +1,75 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] DynConstructors cleanup [iceberg]

nastra commented on code in PR #10542: URL: https://github.com/apache/iceberg/pull/10542#discussion_r1650975795 ## common/src/test/java/org/apache/iceberg/common/TestDynConstructors.java: ## @@ -0,0 +1,75 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] AWS: Add FileIO tracker/closer to Glue catalog [iceberg]

findepi commented on code in PR #8315: URL: https://github.com/apache/iceberg/pull/8315#discussion_r1650952000 ## aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java: ## @@ -619,4 +635,17 @@ public void setConf(Configuration conf) { protected Map properties() {

Re: [PR] Add configurable cache expiry to caching catalog [iceberg]

findepi commented on code in PR #3543: URL: https://github.com/apache/iceberg/pull/3543#discussion_r1650929394 ## core/src/main/java/org/apache/iceberg/CachingCatalog.java: ## @@ -29,24 +34,91 @@ import org.apache.iceberg.catalog.Namespace; import org.apache.iceberg.catalog.Ta

Re: [I] Race condition in CachingCatalog [iceberg]

findepi commented on issue #10493: URL: https://github.com/apache/iceberg/issues/10493#issuecomment-2186420099 @singhpk234 indeed this would alleviate the concern, but I don't see a way to implement caching in `Tabble loadTable(TableIdentifier)` method. We don't have the snapshot ID yet, an

Re: [PR] Improve MetricsReporter loading with class loader fallback [iceberg]

findepi commented on code in PR #10459: URL: https://github.com/apache/iceberg/pull/10459#discussion_r1650899387 ## core/src/test/resources/com/example/target/custom-metrics-reporter-1.0-SNAPSHOT.jar: ## Review Comment: You can create a new classloader with e.g. `new URLCla

Re: [PR] DynFields, DynMethods code cleanup [iceberg]

findepi commented on code in PR #10543: URL: https://github.com/apache/iceberg/pull/10543#discussion_r1650894264 ## common/src/main/java/org/apache/iceberg/common/DynMethods.java: ## @@ -161,6 +161,8 @@ private BoundMethod(UnboundMethod method, Object receiver) { this.rec

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

zhongqishang commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1650864804 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,44 @@ private void commitOperation( } @Ov

Re: [I] refactor: Remove dependency of tokio. [iceberg-rust]

Xuanwo commented on issue #418: URL: https://github.com/apache/iceberg-rust/issues/418#issuecomment-2186329846 Let's me do this. I feel like it's a good win for this to remove direct dep on tokio. -- This is an automated message from the Apache Git Service. To respond to the message, plea

[PR] Refactored data clumps with the help of LLMs (research project) [iceberg]

compf opened a new pull request, #10558: URL: https://github.com/apache/iceberg/pull/10558 Hello maintainers, I am conducting a master thesis project focused on enhancing code quality through automated refactoring of data clumps, assisted by Large Language Models (LLMs).

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

pvary commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1650851730 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,44 @@ private void commitOperation( } @Override

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

pvary commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1650849668 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,44 @@ private void commitOperation( } @Override

Re: [PR] Core: FileSystemCatalogTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1650772614 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -159,18 +160,30 @@ public void commit(TableMetadata base, TableMetadata metadata) {

Re: [PR] Core: FileSystemCatalogTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1650772614 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -159,18 +160,30 @@ public void commit(TableMetadata base, TableMetadata metadata) {

Re: [PR] Core: FileSystemCatalogTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1650769239 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -159,18 +160,30 @@ public void commit(TableMetadata base, TableMetadata metadata) {

Re: [I] [Bug]: Can't create/update tables in REST nessie catalog via Spark. Iceberg Cannot read field "formatVersion" because "x0" is null [iceberg]

ajantha-bhat commented on issue #10533: URL: https://github.com/apache/iceberg/issues/10533#issuecomment-2186193549 yeah. We will check it. cc: @dimas-b, @snazy -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Core: FileSystemCatalogTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1650741211 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -370,58 +402,70 @@ private void renameToFinal(FileSystem fs, Path src, Path dst, int

Re: [PR] Core: FileSystemCatalogTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1650741211 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -370,58 +402,70 @@ private void renameToFinal(FileSystem fs, Path src, Path dst, int

Re: [PR] Core: FileSystemCatalogTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1650741211 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -370,58 +402,70 @@ private void renameToFinal(FileSystem fs, Path src, Path dst, int

Re: [PR] Core: FileSystemCatalogTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1650741211 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -370,58 +402,70 @@ private void renameToFinal(FileSystem fs, Path src, Path dst, int

Re: [PR] Core: FileSystemCatalogTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

Fokko commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1650739534 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -355,71 +475,90 @@ int findVersion() { * an attempt will be made to delete the sourc

Re: [PR] Core: FileSystemCatalogTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1650728447 ## core/src/test/java/org/apache/iceberg/hadoop/TestHadoopCommits.java: ## @@ -206,6 +210,139 @@ public void testFailedCommit() throws Exception { Assertions.asser

Re: [PR] Core: FileSystemCatalogTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

Fokko commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1650722647 ## core/src/test/java/org/apache/iceberg/hadoop/TestHadoopCommits.java: ## @@ -206,6 +210,139 @@ public void testFailedCommit() throws Exception { Assertions.assertT

Re: [I] Issue with CALL parsing [iceberg]

qianzhen0 commented on issue #8343: URL: https://github.com/apache/iceberg/issues/8343#issuecomment-2186085314 in my case, adding `spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions` solve the problem -- This is an automated message from the Apache Git

Re: [PR] Core: FileSystemCatalogTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

Fokko commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1650714966 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -370,58 +402,70 @@ private void renameToFinal(FileSystem fs, Path src, Path dst, int ne

Re: [PR] Core: FileSystemCatalogTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

Fokko commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1650711386 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -368,58 +431,63 @@ private void renameToFinal(FileSystem fs, Path src, Path dst, int ne

Re: [I] Iceberg Spark streaming skips rows of data [iceberg]

cccs-jc commented on issue #10156: URL: https://github.com/apache/iceberg/issues/10156#issuecomment-2186073673 @singhpk234 any progress on this issue ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Api: Track partition statistics via TableMetadata [iceberg]

advancedxy commented on code in PR #8502: URL: https://github.com/apache/iceberg/pull/8502#discussion_r1650705615 ## core/src/main/java/org/apache/iceberg/TableMetadataParser.java: ## @@ -481,6 +488,13 @@ public static TableMetadata fromJson(String metadataLocation, JsonNode no

Re: [PR] Core: FileSystemCatalogTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

Fokko commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1650701857 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -159,18 +160,30 @@ public void commit(TableMetadata base, TableMetadata metadata) {

Re: [I] [Bug]: Can't create/update tables in REST nessie catalog via Spark. Iceberg Cannot read field "formatVersion" because "x0" is null [iceberg]

Fokko commented on issue #10533: URL: https://github.com/apache/iceberg/issues/10533#issuecomment-2186001634 @ajantha-bhat do you have time to dig into this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[I] support bloom-filter writing [iceberg-python]

raphaelauv opened a new issue, #850: URL: https://github.com/apache/iceberg-python/issues/850 ### Feature Request / Improvement Hi, py-iceberg wannot yet write bloom-filter, would be a great feature, thanks all -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Build: Bump parquet from 1.13.1 to 1.14.1 [iceberg]

Fokko commented on PR #10553: URL: https://github.com/apache/iceberg/pull/10553#issuecomment-2185975514 Blocked by the Gradle update 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] DynFields, DynMethods code cleanup [iceberg]

Fokko commented on code in PR #10543: URL: https://github.com/apache/iceberg/pull/10543#discussion_r1650614916 ## common/src/main/java/org/apache/iceberg/common/DynMethods.java: ## @@ -161,6 +161,8 @@ private BoundMethod(UnboundMethod method, Object receiver) { this.recei

Re: [PR] DynFields, DynMethods code cleanup [iceberg]

Fokko commented on code in PR #10543: URL: https://github.com/apache/iceberg/pull/10543#discussion_r1650614392 ## common/src/main/java/org/apache/iceberg/common/DynMethods.java: ## @@ -313,6 +317,8 @@ public Builder impl(Class targetClass, Class... argClasses) { return t

Re: [PR] Build: Bump org.roaringbitmap:RoaringBitmap from 1.0.6 to 1.1.0 [iceberg]

Fokko merged PR #10552: URL: https://github.com/apache/iceberg/pull/10552 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Build: Bump nessie from 0.90.4 to 0.91.1 [iceberg]

Fokko commented on PR #10551: URL: https://github.com/apache/iceberg/pull/10551#issuecomment-2185914294 Thanks for reviewing @ajantha-bhat 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Build: Bump nessie from 0.90.4 to 0.91.1 [iceberg]

Fokko merged PR #10551: URL: https://github.com/apache/iceberg/pull/10551 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.26.3 to 2.26.7 [iceberg]

Fokko commented on PR #10554: URL: https://github.com/apache/iceberg/pull/10554#issuecomment-2185905602 Thanks for the review @singhpk234 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.26.3 to 2.26.7 [iceberg]

Fokko merged PR #10554: URL: https://github.com/apache/iceberg/pull/10554 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Build: Bump mkdocs-material from 9.5.26 to 9.5.27 [iceberg]

Fokko merged PR #10555: URL: https://github.com/apache/iceberg/pull/10555 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] feat(exprs): Adding BooleanExpressions and Predicates [iceberg-go]

Fokko commented on code in PR #91: URL: https://github.com/apache/iceberg-go/pull/91#discussion_r1650517065 ## exprs.go: ## @@ -0,0 +1,968 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distr

Re: [PR] Encryption integration and test [iceberg]

hsiang-c commented on code in PR #5544: URL: https://github.com/apache/iceberg/pull/5544#discussion_r1650529269 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -137,7 +163,47 @@ protected String tableName() { @Override public FileI

Re: [PR] Metadata Log Entries metadata table [iceberg-python]

HonahX commented on code in PR #667: URL: https://github.com/apache/iceberg-python/pull/667#discussion_r1650524201 ## pyiceberg/table/snapshots.py: ## @@ -226,7 +226,8 @@ def __eq__(self, other: Any) -> bool: class Snapshot(IcebergBaseModel): snapshot_id: int = Field(alias

Re: [PR] Add the build from source section [iceberg-go]

Fokko commented on PR #70: URL: https://github.com/apache/iceberg-go/pull/70#issuecomment-2185823387 Thanks for the PR @git-hulk and @zeroshade for the review! 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Add the build from source section [iceberg-go]

Fokko merged PR #70: URL: https://github.com/apache/iceberg-go/pull/70 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache

Re: [PR] build(deps): bump github.com/pterm/pterm from 0.12.78 to 0.12.79 [iceberg-go]

Fokko merged PR #93: URL: https://github.com/apache/iceberg-go/pull/93 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache

Re: [PR] build(deps): bump github.com/hamba/avro/v2 from 2.21.1 to 2.22.1 [iceberg-go]

Fokko merged PR #94: URL: https://github.com/apache/iceberg-go/pull/94 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache

Re: [PR] build(deps): bump github.com/apache/arrow/go/v16 from 16.0.0 to 16.1.0 [iceberg-go]

Fokko merged PR #97: URL: https://github.com/apache/iceberg-go/pull/97 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache