[PR] Build: Bump com.palantir.baseline:gradle-baseline-java from 5.61.0 to 5.66.0 [iceberg]

2024-09-14 Thread via GitHub
dependabot[bot] opened a new pull request, #11141: URL: https://github.com/apache/iceberg/pull/11141 Bumps [com.palantir.baseline:gradle-baseline-java](https://github.com/palantir/gradle-baseline) from 5.61.0 to 5.66.0. Release notes Sourced from https://github.com/palantir/gradle

[PR] Build: Bump org.xerial.snappy:snappy-java from 1.1.10.6 to 1.1.10.7 [iceberg]

2024-09-14 Thread via GitHub
dependabot[bot] opened a new pull request, #11140: URL: https://github.com/apache/iceberg/pull/11140 Bumps [org.xerial.snappy:snappy-java](https://github.com/xerial/snappy-java) from 1.1.10.6 to 1.1.10.7. Release notes Sourced from https://github.com/xerial/snappy-java/releases";>o

[PR] Build: Bump nessie from 0.95.0 to 0.96.1 [iceberg]

2024-09-14 Thread via GitHub
dependabot[bot] opened a new pull request, #11136: URL: https://github.com/apache/iceberg/pull/11136 Bumps `nessie` from 0.95.0 to 0.96.1. Updates `org.projectnessie.nessie:nessie-client` from 0.95.0 to 0.96.1 Updates `org.projectnessie.nessie:nessie-jaxrs-testextension` from 0.95.0

[PR] Build: Bump com.google.errorprone:error_prone_annotations from 2.31.0 to 2.32.0 [iceberg]

2024-09-14 Thread via GitHub
dependabot[bot] opened a new pull request, #11139: URL: https://github.com/apache/iceberg/pull/11139 Bumps [com.google.errorprone:error_prone_annotations](https://github.com/google/error-prone) from 2.31.0 to 2.32.0. Release notes Sourced from https://github.com/google/error-prone

[PR] Build: Bump org.apache.datasketches:datasketches-java from 6.0.0 to 6.1.0 [iceberg]

2024-09-14 Thread via GitHub
dependabot[bot] opened a new pull request, #11137: URL: https://github.com/apache/iceberg/pull/11137 Bumps org.apache.datasketches:datasketches-java from 6.0.0 to 6.1.0. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependenc

[PR] Build: Bump software.amazon.awssdk:bom from 2.27.21 to 2.28.1 [iceberg]

2024-09-14 Thread via GitHub
dependabot[bot] opened a new pull request, #11138: URL: https://github.com/apache/iceberg/pull/11138 Bumps software.amazon.awssdk:bom from 2.27.21 to 2.28.1. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=softw

Re: [I] Implement Synchronous partition stats writing during write operation (controlled by table property). [iceberg]

2024-09-14 Thread via GitHub
github-actions[bot] commented on issue #8458: URL: https://github.com/apache/iceberg/issues/8458#issuecomment-2351239786 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Build a util to read and write partition stats file for a table on a single node. [iceberg]

2024-09-14 Thread via GitHub
github-actions[bot] commented on issue #8456: URL: https://github.com/apache/iceberg/issues/8456#issuecomment-2351239776 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Introduce PartitionEntry class to represent stats per partition [iceberg]

2024-09-14 Thread via GitHub
github-actions[bot] commented on issue #8455: URL: https://github.com/apache/iceberg/issues/8455#issuecomment-2351239757 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Iceberg spark procedure argument does not support empty map or empty array. [iceberg]

2024-09-14 Thread via GitHub
github-actions[bot] commented on issue #8448: URL: https://github.com/apache/iceberg/issues/8448#issuecomment-2351239731 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Spark 3.4: Supports empty map and empty array expressions [iceberg]

2024-09-14 Thread via GitHub
github-actions[bot] commented on PR #8449: URL: https://github.com/apache/iceberg/pull/8449#issuecomment-2351239744 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] Can we do Client side Encryption with Iceberg format? [iceberg]

2024-09-14 Thread via GitHub
github-actions[bot] commented on issue #8431: URL: https://github.com/apache/iceberg/issues/8431#issuecomment-2351239707 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Iceberg containing Parquet v2 files cannot be read unless read.parquet.vectorization.enabled is set to false [iceberg]

2024-09-14 Thread via GitHub
github-actions[bot] commented on issue #8430: URL: https://github.com/apache/iceberg/issues/8430#issuecomment-2351239695 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] ICEBERG_CANNOT_OPEN_SPLIT: Error opening Iceberg split s3 [iceberg]

2024-09-14 Thread via GitHub
github-actions[bot] commented on issue #8427: URL: https://github.com/apache/iceberg/issues/8427#issuecomment-2351239685 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] spark-procedures migrating tables can pose fatal problems [iceberg]

2024-09-14 Thread via GitHub
github-actions[bot] commented on issue #8425: URL: https://github.com/apache/iceberg/issues/8425#issuecomment-2351239671 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Spark 3.4: Remove deprecated AssertHelpers [iceberg]

2024-09-14 Thread via GitHub
github-actions[bot] closed pull request #8129: Spark 3.4: Remove deprecated AssertHelpers URL: https://github.com/apache/iceberg/pull/8129 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] Remove unnecessary Failure check in doCommit path [iceberg]

2024-09-14 Thread via GitHub
github-actions[bot] closed pull request #8120: Remove unnecessary Failure check in doCommit path URL: https://github.com/apache/iceberg/pull/8120 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Core, API: Add non-nullable constraint checks to support making optional columns… [iceberg]

2024-09-14 Thread via GitHub
github-actions[bot] commented on PR #8128: URL: https://github.com/apache/iceberg/pull/8128#issuecomment-2351239444 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Spark 3.4: Remove deprecated AssertHelpers [iceberg]

2024-09-14 Thread via GitHub
github-actions[bot] commented on PR #8129: URL: https://github.com/apache/iceberg/pull/8129#issuecomment-2351239452 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Core, API: Add non-nullable constraint checks to support making optional columns… [iceberg]

2024-09-14 Thread via GitHub
github-actions[bot] closed pull request #8128: Core, API: Add non-nullable constraint checks to support making optional columns… URL: https://github.com/apache/iceberg/pull/8128 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Remove unnecessary Failure check in doCommit path [iceberg]

2024-09-14 Thread via GitHub
github-actions[bot] commented on PR #8120: URL: https://github.com/apache/iceberg/pull/8120#issuecomment-2351239429 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] fix: support MonthTransform for partitioning [iceberg-python]

2024-09-14 Thread via GitHub
felixscherz commented on code in PR #1176: URL: https://github.com/apache/iceberg-python/pull/1176#discussion_r1759784424 ## tests/integration/test_partition_evolution.py: ## @@ -152,6 +201,36 @@ def test_multiple_adds(catalog: Catalog) -> None: ) +@pytest.mark.integrat

Re: [PR] fix: support MonthTransform for partitioning [iceberg-python]

2024-09-14 Thread via GitHub
kevinjqliu commented on code in PR #1176: URL: https://github.com/apache/iceberg-python/pull/1176#discussion_r1759766338 ## tests/integration/test_partition_evolution.py: ## @@ -152,6 +201,36 @@ def test_multiple_adds(catalog: Catalog) -> None: ) +@pytest.mark.integrati

Re: [PR] fix: support MonthTransform for partitioning [iceberg-python]

2024-09-14 Thread via GitHub
felixscherz commented on code in PR #1176: URL: https://github.com/apache/iceberg-python/pull/1176#discussion_r1759758412 ## tests/integration/test_partition_evolution.py: ## @@ -100,6 +100,13 @@ def test_add_month(catalog: Catalog) -> None: _validate_new_partition_fields(t

Re: [I] Minimum required pyarrow version [iceberg-python]

2024-09-14 Thread via GitHub
kevinjqliu commented on issue #1174: URL: https://github.com/apache/iceberg-python/issues/1174#issuecomment-2351043545 Thanks for reporting this. That makes sense to me. Is this something you would like to help contribute? -- This is an automated message from the Apache Git Service. To

Re: [PR] fix: support MonthTransform for partitioning [iceberg-python]

2024-09-14 Thread via GitHub
kevinjqliu commented on code in PR #1176: URL: https://github.com/apache/iceberg-python/pull/1176#discussion_r1759758077 ## tests/integration/test_partition_evolution.py: ## @@ -100,6 +100,13 @@ def test_add_month(catalog: Catalog) -> None: _validate_new_partition_fields(ta

Re: [PR] feat(iceberg): support sql catalog interface [iceberg-rust]

2024-09-14 Thread via GitHub
Li0k closed pull request #631: feat(iceberg): support sql catalog interface URL: https://github.com/apache/iceberg-rust/pull/631 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[PR] fix: support MonthTransform for partitioning [iceberg-python]

2024-09-14 Thread via GitHub
felixscherz opened a new pull request, #1176: URL: https://github.com/apache/iceberg-python/pull/1176 Hi, this is fixing a minor bug where partition evolution in combination with a `MonthTransform` could raise an exception: https://github.com/apache/iceberg-python/issues/1156 --

[I] Name Mapping Serialisation Spec lists field `field_id` but examples use `field-id` [iceberg]

2024-09-14 Thread via GitHub
jonaswk opened a new issue, #11134: URL: https://github.com/apache/iceberg/issues/11134 ### Apache Iceberg version 1.6.1 (latest release) ### Query engine None ### Please describe the bug 🐞 The spec section on the name mapping serialisation (https://github.

Re: [PR] Updating SparkScan to only read Apache DataSketches [iceberg]

2024-09-14 Thread via GitHub
guykhazma commented on PR #11035: URL: https://github.com/apache/iceberg/pull/11035#issuecomment-2350971627 @karuppayya @huaxingao @szehon-ho can you please help review this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] core:Refactor the code of HadoopTableOptions [iceberg]

2024-09-14 Thread via GitHub
BsoBird closed pull request #10623: core:Refactor the code of HadoopTableOptions URL: https://github.com/apache/iceberg/pull/10623 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [I] [feat] add missing metadata tables [iceberg-python]

2024-09-14 Thread via GitHub
soumya-ghosh commented on issue #1053: URL: https://github.com/apache/iceberg-python/issues/1053#issuecomment-2350947937 > What if you just return all unique (data+delete) files? In this case, output will not match with Spark. Will that be okay? Also found this [PR from Iceber

Re: [I] Delete Files in Table Scans [iceberg-rust]

2024-09-14 Thread via GitHub
xxhZs commented on issue #630: URL: https://github.com/apache/iceberg-rust/issues/630#issuecomment-2350911899 Hi, I've recently implemented merge on read in my library using iceberg rust and submitted a working simplified version of the code, which looks somewhat similar to the `A naive app

Re: [PR] Support changelog scan for table with delete files [iceberg]

2024-09-14 Thread via GitHub
pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1759686766 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,139 @@ public void testFileDeletes() { assertThat(t1.existingDele

Re: [PR] Support changelog scan for table with delete files [iceberg]

2024-09-14 Thread via GitHub
pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1759685604 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -63,33 +60,43 @@ protected CloseableIterable doPlanFiles( return CloseableItera