Re: [I] Convert row filter to arrow filter [iceberg-rust]

2024-03-18 Thread via GitHub
Dysprosium0626 commented on issue #265: URL: https://github.com/apache/iceberg-rust/issues/265#issuecomment-2003096263 Hi @liurenjie1024 I could work on this but I have no idea where is the current row filter? I only find something here: https://github.com/apache/iceberg-rust/blob/d6703df40

Re: [PR] Build: Bump com.google.errorprone:error_prone_annotations from 2.24.1 to 2.26.1 [iceberg]

2024-03-18 Thread via GitHub
nastra merged PR #9972: URL: https://github.com/apache/iceberg/pull/9972 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Build: Bump org.awaitility:awaitility from 4.2.0 to 4.2.1 [iceberg]

2024-03-18 Thread via GitHub
nastra merged PR #9970: URL: https://github.com/apache/iceberg/pull/9970 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Add 13 Dremio Blogs + Fix a few incorrect dates [iceberg]

2024-03-18 Thread via GitHub
nastra merged PR #9967: URL: https://github.com/apache/iceberg/pull/9967 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Build: Fix ignoring major version update in dependabot [iceberg]

2024-03-18 Thread via GitHub
nastra merged PR #9981: URL: https://github.com/apache/iceberg/pull/9981 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Build: Bump com.palantir.baseline:gradle-baseline-java from 4.42.0 to 5.44.0 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] commented on PR #9978: URL: https://github.com/apache/iceberg/pull/9978#issuecomment-2003110726 Looks like com.palantir.baseline:gradle-baseline-java is no longer being updated by Dependabot, so this is no longer needed. -- This is an automated message from the Apache Git

Re: [PR] Build: Bump com.github.ben-manes.caffeine:caffeine from 2.9.3 to 3.1.8 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] closed pull request #8784: Build: Bump com.github.ben-manes.caffeine:caffeine from 2.9.3 to 3.1.8 URL: https://github.com/apache/iceberg/pull/8784 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] Build: Bump org.glassfish.jaxb:jaxb-runtime from 2.3.3 to 4.0.5 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] closed pull request #9908: Build: Bump org.glassfish.jaxb:jaxb-runtime from 2.3.3 to 4.0.5 URL: https://github.com/apache/iceberg/pull/9908 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Build: Bump com.palantir.baseline:gradle-baseline-java from 4.42.0 to 5.44.0 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] closed pull request #9978: Build: Bump com.palantir.baseline:gradle-baseline-java from 4.42.0 to 5.44.0 URL: https://github.com/apache/iceberg/pull/9978 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Build: Bump slf4j from 1.7.36 to 2.0.9 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] closed pull request #8737: Build: Bump slf4j from 1.7.36 to 2.0.9 URL: https://github.com/apache/iceberg/pull/8737 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] Build: Bump slf4j from 1.7.36 to 2.0.12 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] commented on PR #9688: URL: https://github.com/apache/iceberg/pull/9688#issuecomment-2003110737 Looks like these dependencies are no longer being updated by Dependabot, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Build: Bump org.glassfish.jaxb:jaxb-runtime from 2.3.3 to 4.0.4 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] commented on PR #8898: URL: https://github.com/apache/iceberg/pull/8898#issuecomment-2003110724 Looks like org.glassfish.jaxb:jaxb-runtime is no longer being updated by Dependabot, so this is no longer needed. -- This is an automated message from the Apache Git Service. To

Re: [PR] Build: Bump slf4j from 1.7.36 to 2.0.9 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] commented on PR #8737: URL: https://github.com/apache/iceberg/pull/8737#issuecomment-2003110730 Looks like these dependencies are no longer being updated by Dependabot, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Build: Bump com.github.ben-manes.caffeine:caffeine from 2.9.3 to 3.1.8 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] commented on PR #8784: URL: https://github.com/apache/iceberg/pull/8784#issuecomment-2003110719 Looks like com.github.ben-manes.caffeine:caffeine is no longer being updated by Dependabot, so this is no longer needed. -- This is an automated message from the Apache Git Serv

Re: [PR] Build: Bump com.esotericsoftware:kryo from 4.0.2 to 5.6.0 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] commented on PR #9469: URL: https://github.com/apache/iceberg/pull/9469#issuecomment-2003110725 Looks like com.esotericsoftware:kryo is no longer being updated by Dependabot, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respo

Re: [PR] Build: Bump org.springframework:spring-web from 5.3.30 to 6.1.5 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] commented on PR #9969: URL: https://github.com/apache/iceberg/pull/9969#issuecomment-2003110720 Looks like org.springframework:spring-web is no longer being updated by Dependabot, so this is no longer needed. -- This is an automated message from the Apache Git Service. To

Re: [PR] Build: Bump org.springframework:spring-web from 5.3.30 to 6.1.5 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] closed pull request #9969: Build: Bump org.springframework:spring-web from 5.3.30 to 6.1.5 URL: https://github.com/apache/iceberg/pull/9969 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Build: Bump com.adobe.testing:s3mock-junit5 from 2.11.0 to 3.5.2 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] commented on PR #9971: URL: https://github.com/apache/iceberg/pull/9971#issuecomment-2003110723 Looks like com.adobe.testing:s3mock-junit5 is no longer being updated by Dependabot, so this is no longer needed. -- This is an automated message from the Apache Git Service. To

Re: [PR] Build: Bump jakarta.el:jakarta.el-api from 3.0.3 to 5.0.1 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] commented on PR #8791: URL: https://github.com/apache/iceberg/pull/8791#issuecomment-2003110713 Looks like jakarta.el:jakarta.el-api is no longer being updated by Dependabot, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respo

Re: [PR] Build: Bump orc from 1.9.2 to 2.0.0 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] commented on PR #9913: URL: https://github.com/apache/iceberg/pull/9913#issuecomment-2003110740 Looks like these dependencies are no longer being updated by Dependabot, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Build: Bump org.openapitools:openapi-generator-gradle-plugin from 6.6.0 to 7.4.0 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] closed pull request #9973: Build: Bump org.openapitools:openapi-generator-gradle-plugin from 6.6.0 to 7.4.0 URL: https://github.com/apache/iceberg/pull/9973 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Build: Bump jakarta.el:jakarta.el-api from 3.0.3 to 5.0.1 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] closed pull request #8791: Build: Bump jakarta.el:jakarta.el-api from 3.0.3 to 5.0.1 URL: https://github.com/apache/iceberg/pull/8791 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Build: Bump slf4j from 1.7.36 to 2.0.12 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] closed pull request #9688: Build: Bump slf4j from 1.7.36 to 2.0.12 URL: https://github.com/apache/iceberg/pull/9688 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Build: Bump org.openapitools:openapi-generator-gradle-plugin from 6.6.0 to 7.4.0 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] commented on PR #9973: URL: https://github.com/apache/iceberg/pull/9973#issuecomment-2003110722 Looks like org.openapitools:openapi-generator-gradle-plugin is no longer being updated by Dependabot, so this is no longer needed. -- This is an automated message from the Apach

Re: [PR] Build: Bump orc from 1.9.2 to 2.0.0 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] closed pull request #9913: Build: Bump orc from 1.9.2 to 2.0.0 URL: https://github.com/apache/iceberg/pull/9913 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Docs: Enhance Spark pages [iceberg]

2024-03-18 Thread via GitHub
manuzhang commented on PR #9920: URL: https://github.com/apache/iceberg/pull/9920#issuecomment-2003111009 @Fokko is fixing it in https://github.com/apache/iceberg/pull/9965 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Build: Bump com.esotericsoftware:kryo from 4.0.2 to 5.6.0 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] closed pull request #9469: Build: Bump com.esotericsoftware:kryo from 4.0.2 to 5.6.0 URL: https://github.com/apache/iceberg/pull/9469 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Build: Bump org.glassfish.jaxb:jaxb-runtime from 2.3.3 to 4.0.5 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] commented on PR #9908: URL: https://github.com/apache/iceberg/pull/9908#issuecomment-2003110716 Looks like org.glassfish.jaxb:jaxb-runtime is no longer being updated by Dependabot, so this is no longer needed. -- This is an automated message from the Apache Git Service. To

Re: [PR] Build: Bump com.adobe.testing:s3mock-junit5 from 2.11.0 to 3.5.2 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] closed pull request #9971: Build: Bump com.adobe.testing:s3mock-junit5 from 2.11.0 to 3.5.2 URL: https://github.com/apache/iceberg/pull/9971 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Build: Bump org.glassfish.jaxb:jaxb-runtime from 2.3.3 to 4.0.4 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] closed pull request #8898: Build: Bump org.glassfish.jaxb:jaxb-runtime from 2.3.3 to 4.0.4 URL: https://github.com/apache/iceberg/pull/8898 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[PR] Build: Bump jetty from 9.4.53.v20231009 to 9.4.54.v20240208 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] opened a new pull request, #9982: URL: https://github.com/apache/iceberg/pull/9982 Bumps `jetty` from 9.4.53.v20231009 to 9.4.54.v20240208. Updates `org.eclipse.jetty:jetty-server` from 9.4.53.v20231009 to 9.4.54.v20240208 Updates `org.eclipse.jetty:jetty-servlet` f

[PR] Build: Bump mkdocs-material from 9.5.9 to 9.5.14 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] opened a new pull request, #9983: URL: https://github.com/apache/iceberg/pull/9983 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.9 to 9.5.14. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mkdocs-m

Re: [PR] Build: Bump mkdocs-material from 9.5.9 to 9.5.13 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] closed pull request #9906: Build: Bump mkdocs-material from 9.5.9 to 9.5.13 URL: https://github.com/apache/iceberg/pull/9906 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Build: Bump mkdocs-material from 9.5.9 to 9.5.13 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] commented on PR #9906: URL: https://github.com/apache/iceberg/pull/9906#issuecomment-2003111838 Superseded by #9983. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Build: Bump nessie from 0.77.1 to 0.79.0 [iceberg]

2024-03-18 Thread via GitHub
nastra merged PR #9976: URL: https://github.com/apache/iceberg/pull/9976 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[PR] Build: Bump com.esotericsoftware:kryo from 4.0.2 to 4.0.3 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] opened a new pull request, #9984: URL: https://github.com/apache/iceberg/pull/9984 Bumps [com.esotericsoftware:kryo](https://github.com/EsotericSoftware/kryo) from 4.0.2 to 4.0.3. Release notes Sourced from https://github.com/EsotericSoftware/kryo/releases";>com.eso

Re: [PR] Build: Bump datamodel-code-generator from 0.25.4 to 0.25.5 [iceberg]

2024-03-18 Thread via GitHub
Fokko merged PR #9979: URL: https://github.com/apache/iceberg/pull/9979 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[PR] Build: Bump spring-boot from 2.5.4 to 2.7.18 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] opened a new pull request, #9985: URL: https://github.com/apache/iceberg/pull/9985 Bumps `spring-boot` from 2.5.4 to 2.7.18. Updates `org.springframework.boot:spring-boot-starter-jetty` from 2.5.4 to 2.7.18 Release notes Sourced from https://github.com/spring-pro

[PR] Build: Bump com.adobe.testing:s3mock-junit5 from 2.11.0 to 2.17.0 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] opened a new pull request, #9986: URL: https://github.com/apache/iceberg/pull/9986 Bumps com.adobe.testing:s3mock-junit5 from 2.11.0 to 2.17.0. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=com

[PR] Build: Bump com.palantir.baseline:gradle-baseline-java from 4.42.0 to 4.192.0 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] opened a new pull request, #9987: URL: https://github.com/apache/iceberg/pull/9987 Bumps [com.palantir.baseline:gradle-baseline-java](https://github.com/palantir/gradle-baseline) from 4.42.0 to 4.192.0. Release notes Sourced from https://github.com/palantir/gradle-

[PR] Build: Bump org.glassfish.jaxb:jaxb-runtime from 2.3.3 to 2.3.9 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] opened a new pull request, #9988: URL: https://github.com/apache/iceberg/pull/9988 Bumps org.glassfish.jaxb:jaxb-runtime from 2.3.3 to 2.3.9. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=org.g

[PR] Build: Bump org.springframework:spring-web from 5.3.30 to 5.3.33 [iceberg]

2024-03-18 Thread via GitHub
dependabot[bot] opened a new pull request, #9989: URL: https://github.com/apache/iceberg/pull/9989 Bumps [org.springframework:spring-web](https://github.com/spring-projects/spring-framework) from 5.3.30 to 5.3.33. Release notes Sourced from https://github.com/spring-projects/sprin

Re: [PR] Build: Bump mkdocs-material from 9.5.9 to 9.5.14 [iceberg]

2024-03-18 Thread via GitHub
Fokko merged PR #9983: URL: https://github.com/apache/iceberg/pull/9983 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] docs: Add links checker [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9965: URL: https://github.com/apache/iceberg/pull/9965#discussion_r1527998892 ## format/spec.md: ## @@ -57,6 +57,7 @@ In addition to row-level deletes, version 2 makes some requirements stricter for ## Overview + Review Comment: should t

Re: [PR] Build: Bump orc from 1.9.2 to 2.0.0 [iceberg]

2024-03-18 Thread via GitHub
manuzhang commented on PR #9913: URL: https://github.com/apache/iceberg/pull/9913#issuecomment-2003126279 @nastra Thanks, this was working immediately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Migrate Manifest, FormatVersion and LocationProvider files in Core to JUnit5 [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9964: URL: https://github.com/apache/iceberg/pull/9964#discussion_r1528013566 ## core/src/test/java/org/apache/iceberg/TestManifestWriter.java: ## @@ -89,48 +80,44 @@ public void testManifestPartitionStats() throws IOException { mani

Re: [PR] Migrate Manifest, FormatVersion and LocationProvider files in Core to JUnit5 [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9964: URL: https://github.com/apache/iceberg/pull/9964#discussion_r1528020644 ## core/src/test/java/org/apache/iceberg/TestManifestWriter.java: ## @@ -89,48 +80,44 @@ public void testManifestPartitionStats() throws IOException { mani

Re: [PR] Migrate Manifest, FormatVersion and LocationProvider files in Core to JUnit5 [iceberg]

2024-03-18 Thread via GitHub
nastra merged PR #9964: URL: https://github.com/apache/iceberg/pull/9964 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Migrate Manifest, FormatVersion and LocationProvider files in Core to JUnit5 [iceberg]

2024-03-18 Thread via GitHub
tomtongue commented on PR #9964: URL: https://github.com/apache/iceberg/pull/9964#issuecomment-2003146004 @nastra Thanks for the quick review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Spark: Add a test to check if the bloom filters are added to the parquet files [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9902: URL: https://github.com/apache/iceberg/pull/9902#discussion_r1528029346 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java: ## @@ -174,170 +156,62 @@ public static Object[][] parameters() {

[I] CI looks like broken [iceberg-rust]

2024-03-18 Thread via GitHub
viirya opened a new issue, #279: URL: https://github.com/apache/iceberg-rust/issues/279 https://github.com/apache/iceberg-rust/actions/runs/8323042113/job/22771908103?pr=258 ``` warning: use of deprecated function `arrow_arith::temporal::month_dyn`: Use `date_part` instead --

Re: [PR] Spark: Add a test to check if the bloom filters are added to the parquet files [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9902: URL: https://github.com/apache/iceberg/pull/9902#discussion_r1528029935 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java: ## @@ -18,56 +18,43 @@ */ package org.apache.iceberg.spark.sour

Re: [PR] Spark: Add a test to check if the bloom filters are added to the parquet files [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9902: URL: https://github.com/apache/iceberg/pull/9902#discussion_r1528030672 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java: ## @@ -116,54 +114,38 @@ public class TestSparkReaderWithBloomFilt

Re: [PR] Spark: Add a test to check if the bloom filters are added to the parquet files [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9902: URL: https://github.com/apache/iceberg/pull/9902#discussion_r1528031076 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java: ## @@ -116,54 +114,38 @@ public class TestSparkReaderWithBloomFilt

Re: [PR] Spark: Add a test to check if the bloom filters are added to the parquet files [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9902: URL: https://github.com/apache/iceberg/pull/9902#discussion_r1528034527 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java: ## @@ -95,18 +80,31 @@ public class TestSparkReaderWithBloomFilter

Re: [I] CI looks like broken [iceberg-rust]

2024-03-18 Thread via GitHub
Fokko commented on issue #279: URL: https://github.com/apache/iceberg-rust/issues/279#issuecomment-2003161284 Thanks for reporting this @viirya. I'm seeing something similar on my local machine: ``` error[E0308]: mismatched types --> /Users/fokkodriesprong/.cargo/git/checkou

Re: [PR] Spark: Add a test to check if the bloom filters are added to the parquet files [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9902: URL: https://github.com/apache/iceberg/pull/9902#discussion_r1528037506 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java: ## @@ -351,12 +225,13 @@ public void testReadWithFilter() {

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1528044340 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -149,26 +183,71 @@ public void commit(TableMetadata base, TableMetadata metadata) {

Re: [PR] feat: Implement the conversion from Arrow Schema to Iceberg Schema [iceberg-rust]

2024-03-18 Thread via GitHub
Fokko commented on code in PR #258: URL: https://github.com/apache/iceberg-rust/pull/258#discussion_r1528045350 ## crates/iceberg/src/arrow.rs: ## @@ -106,3 +114,560 @@ impl ArrowReader { ProjectionMask::all() } } + +/// A post order arrow schema visitor. +/// +//

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1528044340 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -149,26 +183,71 @@ public void commit(TableMetadata base, TableMetadata metadata) {

Re: [PR] Add Snapshots table metadata [iceberg-python]

2024-03-18 Thread via GitHub
Fokko commented on code in PR #524: URL: https://github.com/apache/iceberg-python/pull/524#discussion_r1528049941 ## tests/integration/test_writes.py: ## @@ -664,3 +668,55 @@ def test_table_properties_raise_for_none_value( session_catalog, identifier, {"format-versi

Re: [PR] build: Restore CI by making parquet and arrow version consistent [iceberg-rust]

2024-03-18 Thread via GitHub
viirya commented on code in PR #280: URL: https://github.com/apache/iceberg-rust/pull/280#discussion_r1528050445 ## crates/iceberg/src/writer/file_writer/parquet_writer.rs: ## @@ -112,20 +103,14 @@ impl FileWriterBuilder for ParquetWr .generate_location(&self.

Re: [PR] Add partition stats in snapshot summary [iceberg-python]

2024-03-18 Thread via GitHub
Fokko merged PR #521: URL: https://github.com/apache/iceberg-python/pull/521 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Add partition stats in snapshot summary [iceberg-python]

2024-03-18 Thread via GitHub
Fokko commented on PR #521: URL: https://github.com/apache/iceberg-python/pull/521#issuecomment-2003194763 This is great, thanks for working on this @jqin61 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [I] CI looks like broken [iceberg-rust]

2024-03-18 Thread via GitHub
Xuanwo commented on issue #279: URL: https://github.com/apache/iceberg-rust/issues/279#issuecomment-2003224806 I believe we should commit `Cargo.lock` to ensure we test against the same version. And we need to establish a `MSAV` (Minimum Supported Arrow Version) to make sure our users align

Re: [PR] Spark: Add a test to check if the bloom filters are added to the parquet files [iceberg]

2024-03-18 Thread via GitHub
hussein-awala commented on PR #9902: URL: https://github.com/apache/iceberg/pull/9902#issuecomment-2003228773 > there are a bunch of changes that seem unrelated to what's being proposed to be done (aka adding a check to see if blook filters are added) Thanks @nastra for the review. Th

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-03-18 Thread via GitHub
BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1528103119 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -149,26 +183,71 @@ public void commit(TableMetadata base, TableMetadata metadata) {

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-03-18 Thread via GitHub
BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1528109766 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -149,26 +183,71 @@ public void commit(TableMetadata base, TableMetadata metadata) {

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-03-18 Thread via GitHub
BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1528142417 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -149,26 +183,71 @@ public void commit(TableMetadata base, TableMetadata metadata) {

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-18 Thread via GitHub
tomfans commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2003359267 more information: metadata managed by HMS, not HDFS. it works when metadata managed by HDFS -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1528220605 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -149,26 +183,71 @@ public void commit(TableMetadata base, TableMetadata metadata) {

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1528227235 ## core/src/test/java/org/apache/iceberg/hadoop/TestHadoopCommits.java: ## @@ -206,6 +210,133 @@ public void testFailedCommit() throws Exception { Assertions.assert

Re: [PR] Build: Bump spring-boot from 2.5.4 to 2.7.18 [iceberg]

2024-03-18 Thread via GitHub
nastra merged PR #9985: URL: https://github.com/apache/iceberg/pull/9985 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-18 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1528242070 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -222,53 +231,203 @@ public boolean dropTable(TableIdentifier identifier, boolean purge)

Re: [PR] Spark: Add a test to check if the bloom filters are added to the parquet files [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9902: URL: https://github.com/apache/iceberg/pull/9902#discussion_r1528243529 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java: ## @@ -81,12 +68,10 @@ public class TestSparkReaderWithBloomFilter

Re: [PR] Spark: Add a test to check if the bloom filters are added to the parquet files [iceberg]

2024-03-18 Thread via GitHub
nastra commented on PR #9902: URL: https://github.com/apache/iceberg/pull/9902#issuecomment-2003444989 > > there are a bunch of changes that seem unrelated to what's being proposed to be done (aka adding a check to see if blook filters are added) > > Thanks @nastra for the review. The

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-18 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1528253765 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveOperationsBase.java: ## @@ -181,4 +279,230 @@ default Table newHmsTable(String hmsTableOwner) { return

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-18 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1528257514 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveOperationsBase.java: ## @@ -181,4 +279,230 @@ default Table newHmsTable(String hmsTableOwner) { return

Re: [I] Metadata not found for table created by flink [iceberg]

2024-03-18 Thread via GitHub
nastra closed issue #9958: Metadata not found for table created by flink URL: https://github.com/apache/iceberg/issues/9958 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-03-18 Thread via GitHub
BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1528260699 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -149,26 +183,71 @@ public void commit(TableMetadata base, TableMetadata metadata) {

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-03-18 Thread via GitHub
BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1528264110 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -149,26 +183,71 @@ public void commit(TableMetadata base, TableMetadata metadata) {

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1528265184 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveOperationsBase.java: ## @@ -181,4 +279,230 @@ default Table newHmsTable(String hmsTableOwner) { return

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-03-18 Thread via GitHub
BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1528264110 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -149,26 +183,71 @@ public void commit(TableMetadata base, TableMetadata metadata) {

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-03-18 Thread via GitHub
BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1528264110 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -149,26 +183,71 @@ public void commit(TableMetadata base, TableMetadata metadata) {

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-18 Thread via GitHub
nastra commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1528270018 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -113,6 +124,24 @@ public void initialize(String inputName, Map properties) { this.cl

Re: [I] Open-API: How about adding table change detection in Iceberg Catalog [iceberg]

2024-03-18 Thread via GitHub
mingnuj commented on issue #9942: URL: https://github.com/apache/iceberg/issues/9942#issuecomment-2003484990 > There's an option that you can use to only fetch the latest snapshot via `snapshot-loading-mode=refs` (defaults to `all`) when the table is loaded instead of loading all snapshots.

Re: [PR] Add Snapshots table metadata [iceberg-python]

2024-03-18 Thread via GitHub
Fokko commented on code in PR #524: URL: https://github.com/apache/iceberg-python/pull/524#discussion_r1528301582 ## tests/integration/test_writes.py: ## @@ -664,3 +668,55 @@ def test_table_properties_raise_for_none_value( session_catalog, identifier, {"format-versi

Re: [PR] Build: Bump org.springframework:spring-web from 5.3.30 to 5.3.33 [iceberg]

2024-03-18 Thread via GitHub
nastra merged PR #9989: URL: https://github.com/apache/iceberg/pull/9989 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Build: Bump jetty from 9.4.53.v20231009 to 9.4.54.v20240208 [iceberg]

2024-03-18 Thread via GitHub
nastra merged PR #9982: URL: https://github.com/apache/iceberg/pull/9982 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Build: Bump guava from 33.0.0-jre to 33.1.0-jre [iceberg]

2024-03-18 Thread via GitHub
nastra commented on PR #9977: URL: https://github.com/apache/iceberg/pull/9977#issuecomment-2003593823 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] [WIP] Implement project for Transform. #264 [iceberg-rust]

2024-03-18 Thread via GitHub
marvinlanhenke commented on code in PR #269: URL: https://github.com/apache/iceberg-rust/pull/269#discussion_r1528347743 ## crates/iceberg/src/spec/transform.rs: ## @@ -261,6 +269,50 @@ impl Transform { _ => self == other, } } +/// Projects predica

Re: [I] Discussion: How to handle S3 Config in Catalog [iceberg-rust]

2024-03-18 Thread via GitHub
marvinlanhenke closed issue #273: Discussion: How to handle S3 Config in Catalog URL: https://github.com/apache/iceberg-rust/issues/273 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [I] Discussion: How to handle S3 Config in Catalog [iceberg-rust]

2024-03-18 Thread via GitHub
marvinlanhenke commented on issue #273: URL: https://github.com/apache/iceberg-rust/issues/273#issuecomment-2003673001 @odysa @Xuanwo this is the way I implemented it for now - and I think its fine. PR will be up for review once #271 has been resolved. thanks again for your feedback.

Re: [I] Loading table from metadata file directly. [iceberg-rust]

2024-03-18 Thread via GitHub
liurenjie1024 closed issue #246: Loading table from metadata file directly. URL: https://github.com/apache/iceberg-rust/issues/246 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [I] Loading table from metadata file directly. [iceberg-rust]

2024-03-18 Thread via GitHub
liurenjie1024 commented on issue #246: URL: https://github.com/apache/iceberg-rust/issues/246#issuecomment-2003753550 Close by #259 Feel free to open when if necessary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-03-18 Thread via GitHub
BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1528470303 ## core/src/test/java/org/apache/iceberg/hadoop/TestHadoopCommits.java: ## @@ -206,6 +210,133 @@ public void testFailedCommit() throws Exception { Assertions.asser

[PR] AWS: Add Option to don't write non current columns in glue schema closes #7584 [iceberg]

2024-03-18 Thread via GitHub
Raphael-Vignes opened a new pull request, #9420: URL: https://github.com/apache/iceberg/pull/9420 This PR aims to close this [issue](https://github.com/apache/iceberg/issues/7584) and would resolve this [issue](https://github.com/apache/iceberg/issues/6340) too. We want to provide a

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-03-18 Thread via GitHub
BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1528501169 ## core/src/test/java/org/apache/iceberg/hadoop/TestHadoopCommits.java: ## @@ -206,6 +210,133 @@ public void testFailedCommit() throws Exception { Assertions.asser

Re: [PR] Build: Bump guava from 33.0.0-jre to 33.1.0-jre [iceberg]

2024-03-18 Thread via GitHub
nastra merged PR #9977: URL: https://github.com/apache/iceberg/pull/9977 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] [WIP] Implement project for Transform. #264 [iceberg-rust]

2024-03-18 Thread via GitHub
marvinlanhenke commented on PR #269: URL: https://github.com/apache/iceberg-rust/pull/269#issuecomment-2003886408 @liurenjie1024 @sdd @Xuanwo @ZENOTME I went on and implemented `fn project` for `Transform::Bucket` with some design assumptions. PTAL and tell me what you think, before we c

  1   2   3   >