Re: [PR] Fix NOTICE and LICENSE in the flink-runtime jar [iceberg]

2025-02-01 Thread via GitHub
jbonofre commented on code in PR #12145: URL: https://github.com/apache/iceberg/pull/12145#discussion_r1938424216 ## flink/v1.20/flink-runtime/NOTICE: ## @@ -63,29 +63,277 @@ NOTICE file: -Thi

Re: [PR] Fix NOTICE and LICENSE in the flink-runtime jar [iceberg]

2025-02-01 Thread via GitHub
jbonofre commented on code in PR #12145: URL: https://github.com/apache/iceberg/pull/12145#discussion_r1938423899 ## flink/v1.20/flink-runtime/LICENSE: ## @@ -464,47 +425,92 @@ License text: -

Re: [PR] Fix NOTICE and LICENSE in the gcp-bundle jar [iceberg]

2025-02-01 Thread via GitHub
jbonofre commented on code in PR #12144: URL: https://github.com/apache/iceberg/pull/12144#discussion_r1938423547 ## gcp-bundle/LICENSE: ## @@ -325,24 +315,24 @@ License: The Apache Software License, Version 2.0 - http://www.apache.org/licens Review Comment: Yes, it's wha

Re: [PR] Build: Bump nessie from 0.102.2 to 0.102.4 [iceberg]

2025-02-01 Thread via GitHub
Fokko commented on PR #12153: URL: https://github.com/apache/iceberg/pull/12153#issuecomment-2629279496 Thanks, I was just reading up on it. I was at DuckCon last Friday, so I was a bit out of the loop. -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] chore: make integration tests run concurrently [iceberg-rust]

2025-02-01 Thread via GitHub
gruuya commented on code in PR #924: URL: https://github.com/apache/iceberg-rust/pull/924#discussion_r1938418275 ## crates/integration_tests/tests/shared_tests/scan_all_type.rs: ## @@ -40,29 +40,21 @@ use iceberg::writer::file_writer::location_generator::{ }; use iceberg::writ

[I] Support pushdown filters for non-cast date conversion functions (e.g. to_date) [iceberg-rust]

2025-02-01 Thread via GitHub
omerhadari opened a new issue, #933: URL: https://github.com/apache/iceberg-rust/issues/933 Currently, for queries that compare timestamps/dates using `TO_DATE` in order to for example truncate a timestamp column, no pushdown predicates are applied. This is because the functions `TO_DATE`,

Re: [PR] chore: make integration tests run concurrently [iceberg-rust]

2025-02-01 Thread via GitHub
gruuya commented on code in PR #924: URL: https://github.com/apache/iceberg-rust/pull/924#discussion_r1938418275 ## crates/integration_tests/tests/shared_tests/scan_all_type.rs: ## @@ -40,29 +40,21 @@ use iceberg::writer::file_writer::location_generator::{ }; use iceberg::writ

Re: [PR] Build: Bump nessie from 0.102.2 to 0.102.4 [iceberg]

2025-02-01 Thread via GitHub
jbonofre commented on PR #12153: URL: https://github.com/apache/iceberg/pull/12153#issuecomment-2629270774 @Fokko thanks ! I'm updating the PR about `LICENSE` and `NOTICE` in this regard šŸ˜„ -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] chore: make integration tests run concurrently [iceberg-rust]

2025-02-01 Thread via GitHub
gruuya commented on PR #924: URL: https://github.com/apache/iceberg-rust/pull/924#issuecomment-2629270601 > * Before this PR, tests are compiled into separated binaries. In theory they can run concurrently (e.g., if use `cargo nextest`), but just `cargo test` will run them in serial.

Re: [PR] Build: Bump com.azure:azure-sdk-bom from 1.2.30 to 1.2.31 [iceberg]

2025-02-01 Thread via GitHub
jbonofre commented on PR #12154: URL: https://github.com/apache/iceberg/pull/12154#issuecomment-2629270475 This update requires update in `LICENSE` and `NOTICE`. As I can't merge this PR myself, I assign it to me, to be notified when merged and update `LICENSE` and `NOTICE`. -- Thi

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.30.6 to 2.30.11 [iceberg]

2025-02-01 Thread via GitHub
jbonofre commented on PR #12156: URL: https://github.com/apache/iceberg/pull/12156#issuecomment-2629270093 This update would need to update `LICENSE` and `NOTICE`. As I can't merge this PR, I assign this issue to me to be notified when merged: I will update `LICENSE` and `NOTICE`. -

Re: [PR] Build: Bump mkdocs-material from 9.5.50 to 9.6.1 [iceberg]

2025-02-01 Thread via GitHub
Fokko merged PR #12157: URL: https://github.com/apache/iceberg/pull/12157 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Build: Bump nessie from 0.102.2 to 0.102.4 [iceberg]

2025-02-01 Thread via GitHub
Fokko commented on PR #12153: URL: https://github.com/apache/iceberg/pull/12153#issuecomment-2629269408 @jbonofre Perfect, thanks for fixing this with the Nessie team šŸ‘ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Build: Bump nessie from 0.102.2 to 0.102.4 [iceberg]

2025-02-01 Thread via GitHub
Fokko merged PR #12153: URL: https://github.com/apache/iceberg/pull/12153 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Build: Bump nessie from 0.102.2 to 0.102.4 [iceberg]

2025-02-01 Thread via GitHub
jbonofre commented on PR #12153: URL: https://github.com/apache/iceberg/pull/12153#issuecomment-2629269259 @rdblue @amogh-jahagirdar @Fokko This Nessie version update fixes the GPL "issue" in `NOTICE`. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Spec: Update partition stats for V3 [iceberg]

2025-02-01 Thread via GitHub
stevenzwu commented on code in PR #12098: URL: https://github.com/apache/iceberg/pull/12098#discussion_r1938400073 ## format/spec.md: ## @@ -927,20 +927,21 @@ These rows must be sorted (in ascending manner with NULL FIRST) by `partition` f The schema of the partition statist

Re: [PR] fix: use upstream avro tag [iceberg-cpp]

2025-02-01 Thread via GitHub
Xuanwo merged PR #42: URL: https://github.com/apache/iceberg-cpp/pull/42 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] fix: use upstream avro tag [iceberg-cpp]

2025-02-01 Thread via GitHub
Xuanwo commented on PR #42: URL: https://github.com/apache/iceberg-cpp/pull/42#issuecomment-2629241456 > The CI failure is unrelated which timed out to download Arrow tar ball. Retried and passed. -- This is an automated message from the Apache Git Service. To respond to the message

[PR] Build: Bump mkdocs-material from 9.5.50 to 9.6.1 [iceberg]

2025-02-01 Thread via GitHub
dependabot[bot] opened a new pull request, #12157: URL: https://github.com/apache/iceberg/pull/12157 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.50 to 9.6.1. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mkdocs

[PR] Build: Bump software.amazon.awssdk:bom from 2.30.6 to 2.30.11 [iceberg]

2025-02-01 Thread via GitHub
dependabot[bot] opened a new pull request, #12156: URL: https://github.com/apache/iceberg/pull/12156 Bumps software.amazon.awssdk:bom from 2.30.6 to 2.30.11. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=softw

[PR] Build: Bump net.snowflake:snowflake-jdbc from 3.21.0 to 3.22.0 [iceberg]

2025-02-01 Thread via GitHub
dependabot[bot] opened a new pull request, #12155: URL: https://github.com/apache/iceberg/pull/12155 Bumps [net.snowflake:snowflake-jdbc](https://github.com/snowflakedb/snowflake-jdbc) from 3.21.0 to 3.22.0. Release notes Sourced from https://github.com/snowflakedb/snowflake-jdbc/

[PR] Build: Bump com.azure:azure-sdk-bom from 1.2.30 to 1.2.31 [iceberg]

2025-02-01 Thread via GitHub
dependabot[bot] opened a new pull request, #12154: URL: https://github.com/apache/iceberg/pull/12154 Bumps [com.azure:azure-sdk-bom](https://github.com/azure/azure-sdk-for-java) from 1.2.30 to 1.2.31. Commits https://github.com/Azure/azure-sdk-for-java/commit/e23497516c52edc449

[PR] Build: Bump nessie from 0.102.2 to 0.102.4 [iceberg]

2025-02-01 Thread via GitHub
dependabot[bot] opened a new pull request, #12153: URL: https://github.com/apache/iceberg/pull/12153 Bumps `nessie` from 0.102.2 to 0.102.4. Updates `org.projectnessie.nessie:nessie-client` from 0.102.2 to 0.102.4 Updates `org.projectnessie.nessie:nessie-jaxrs-testextension` from 0.

[PR] Build: Bump me.champeau.jmh:jmh-gradle-plugin from 0.7.2 to 0.7.3 [iceberg]

2025-02-01 Thread via GitHub
dependabot[bot] opened a new pull request, #12152: URL: https://github.com/apache/iceberg/pull/12152 Bumps me.champeau.jmh:jmh-gradle-plugin from 0.7.2 to 0.7.3. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=m

Re: [PR] Fix NOTICE and LICENSE in the gcp-bundle jar [iceberg]

2025-02-01 Thread via GitHub
amogh-jahagirdar commented on code in PR #12144: URL: https://github.com/apache/iceberg/pull/12144#discussion_r1938393559 ## gcp-bundle/LICENSE: ## @@ -325,24 +315,24 @@ License: The Apache Software License, Version 2.0 - http://www.apache.org/licens Review Comment: Findb

Re: [PR] Core: add variant builder implementation [iceberg]

2025-02-01 Thread via GitHub
aihuaxu commented on code in PR #11857: URL: https://github.com/apache/iceberg/pull/11857#discussion_r1938392850 ## core/src/main/java/org/apache/iceberg/variants/VariantBuilderBase.java: ## @@ -0,0 +1,504 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] chore: make integration tests run concurrently [iceberg-rust]

2025-02-01 Thread via GitHub
xxchan commented on code in PR #924: URL: https://github.com/apache/iceberg-rust/pull/924#discussion_r1938389715 ## crates/integration_tests/tests/shared_tests/scan_all_type.rs: ## @@ -40,29 +40,21 @@ use iceberg::writer::file_writer::location_generator::{ }; use iceberg::writ

Re: [PR] feat: Support metadata table "Entries" [iceberg-rust]

2025-02-01 Thread via GitHub
xxchan commented on PR #863: URL: https://github.com/apache/iceberg-rust/pull/863#issuecomment-2629214547 > But when constructing StructArray or RecordBatch, Arrow checks that the schema matches that of the data [(e.g.)](https://github.com/apache/arrow-rs/blob/0c07ec79cd4b28e7aa9d15d1d58b5c

Re: [I] feat: support metadata tables [iceberg-rust]

2025-02-01 Thread via GitHub
xxchan commented on issue #823: URL: https://github.com/apache/iceberg-rust/issues/823#issuecomment-2629208541 Hi, feel free to take subtasks. But kindly reminder that we need to resolve https://github.com/apache/iceberg-rust/issues/868 before proceeding -- This is an automated message fr

Re: [PR] Fix NOTICE and LICENSE in the flink-runtime jar [iceberg]

2025-02-01 Thread via GitHub
amogh-jahagirdar commented on code in PR #12145: URL: https://github.com/apache/iceberg/pull/12145#discussion_r1938382732 ## flink/v1.20/flink-runtime/NOTICE: ## @@ -63,29 +63,277 @@ NOTICE file: ---

Re: [PR] Data: open file using stats in scan [iceberg]

2025-02-01 Thread via GitHub
amogh-jahagirdar merged PR #12151: URL: https://github.com/apache/iceberg/pull/12151 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Spec: Update partition stats for V3 [iceberg]

2025-02-01 Thread via GitHub
amogh-jahagirdar commented on code in PR #12098: URL: https://github.com/apache/iceberg/pull/12098#discussion_r1938383087 ## format/spec.md: ## @@ -927,20 +927,21 @@ These rows must be sorted (in ascending manner with NULL FIRST) by `partition` f The schema of the partition

Re: [PR] Data: open file using stats in scan [iceberg]

2025-02-01 Thread via GitHub
amogh-jahagirdar commented on PR #12151: URL: https://github.com/apache/iceberg/pull/12151#issuecomment-2629205236 Thanks @bryanck this is a good find! I'll go ahead and merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [PR] Fix for Timestamp and Date Conversion to Long in Iceberg Records [iceberg]

2025-02-01 Thread via GitHub
github-actions[bot] commented on PR #11841: URL: https://github.com/apache/iceberg/pull/11841#issuecomment-2629173290 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] MERGE INTO TABLE is not supported temporarily. [iceberg]

2025-02-01 Thread via GitHub
github-actions[bot] commented on issue #10882: URL: https://github.com/apache/iceberg/issues/10882#issuecomment-2629173259 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Athena Iceberg does not delete orphan files [iceberg]

2025-02-01 Thread via GitHub
github-actions[bot] commented on issue #10878: URL: https://github.com/apache/iceberg/issues/10878#issuecomment-2629173256 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Spark: Add read/write support for UUIDs from bytes [iceberg]

2025-02-01 Thread via GitHub
github-actions[bot] commented on issue #10635: URL: https://github.com/apache/iceberg/issues/10635#issuecomment-2629173227 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] throw exception : InvalidOperationException(message:The following columns have types incompatible with the existing columns in their respective positions : idd1) when add column [iceberg]

2025-02-01 Thread via GitHub
github-actions[bot] commented on issue #3747: URL: https://github.com/apache/iceberg/issues/3747#issuecomment-2629173218 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

[PR] Data: open file using stats in scan [iceberg]

2025-02-01 Thread via GitHub
bryanck opened a new pull request, #12151: URL: https://github.com/apache/iceberg/pull/12151 When using `GenericReader`, e.g. via `IcebergGenerics`, to scan data in a table, the data file is opened using the location only. This PR passes in the `DataFile` object instead, so that the file le

Re: [I] feat: support metadata tables [iceberg-rust]

2025-02-01 Thread via GitHub
felixscherz commented on issue #823: URL: https://github.com/apache/iceberg-rust/issues/823#issuecomment-2629135939 Hi, I'm going to have a look at the PARTITIONS table:) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Core: Add InternalData read and write builders [iceberg]

2025-02-01 Thread via GitHub
rdblue commented on code in PR #12060: URL: https://github.com/apache/iceberg/pull/12060#discussion_r1938350796 ## core/src/main/java/org/apache/iceberg/InternalData.java: ## @@ -0,0 +1,159 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contrib

Re: [PR] Core: Add InternalData read and write builders [iceberg]

2025-02-01 Thread via GitHub
rdblue commented on code in PR #12060: URL: https://github.com/apache/iceberg/pull/12060#discussion_r1938350909 ## core/src/main/java/org/apache/iceberg/InternalData.java: ## @@ -0,0 +1,159 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contrib

[I] question on iceberg table [iceberg]

2025-02-01 Thread via GitHub
Madhav064 opened a new issue, #12150: URL: https://github.com/apache/iceberg/issues/12150 ### Query engine _No response_ ### Question What are the best practices for maintaining Iceberg tables, particularly regarding retention, orphan file deletion, and compaction? How d

Re: [PR] Core: add variant builder implementation [iceberg]

2025-02-01 Thread via GitHub
aihuaxu commented on code in PR #11857: URL: https://github.com/apache/iceberg/pull/11857#discussion_r1938340251 ## core/src/main/java/org/apache/iceberg/variants/VariantBuilderBase.java: ## @@ -0,0 +1,504 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

[PR] Core: Delegate `resetMainBranch` to `removeRef` in `TableMetadata::Builder` [iceberg]

2025-02-01 Thread via GitHub
smaheshwar-pltr opened a new pull request, #12149: URL: https://github.com/apache/iceberg/pull/12149 Tiny change to prefer delegation to duplication as https://github.com/apache/iceberg/issues/11109 suggests that keeping these method bodies separate can be risky. -- This is an automated

Re: [PR] Core: Fix cleanup of orphaned statistics files in dropTableData [iceberg]

2025-02-01 Thread via GitHub
lliangyu-lin commented on PR #12132: URL: https://github.com/apache/iceberg/pull/12132#issuecomment-2629089272 @gaborkaszab @ebyhr I did some more search and found that it seems Iceberg already have mechanism for clean up unreferenced statistics and partition statistics files as part of

Re: [PR] feat: search current working directory for config file [iceberg-python]

2025-02-01 Thread via GitHub
IndexSeek commented on PR #1464: URL: https://github.com/apache/iceberg-python/pull/1464#issuecomment-2629027304 > hey @IndexSeek could you rebase this PR and run `make lint`? would be great to include this as part of the 0.9.0 release I just rebased and pre-commit is passing. Please

Re: [PR] Feature: Write to branches [iceberg-python]

2025-02-01 Thread via GitHub
kevinjqliu commented on PR #941: URL: https://github.com/apache/iceberg-python/pull/941#issuecomment-2629019931 thank you! @vinjai feel free to tag me again for review :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-02-01 Thread via GitHub
felixscherz commented on PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#issuecomment-2628985739 I rebased onto the main. I prepared the unit tests using the new `moto` features and I will commit them once they are released:) -- This is an automated message from the Apac

Re: [PR] Feature: Write to branches [iceberg-python]

2025-02-01 Thread via GitHub
vinjai commented on PR #941: URL: https://github.com/apache/iceberg-python/pull/941#issuecomment-2628949861 Hey @kevinjqliu I was traveling for the past two months and couldn’t complete the review comments. I’ll be back in a week, will address the comments then, and get this moving.

Re: [PR] Flink: Add null check to writers to prevent resurrecting null values [iceberg]

2025-02-01 Thread via GitHub
mxm commented on code in PR #12049: URL: https://github.com/apache/iceberg/pull/12049#discussion_r1938268635 ## data/src/test/java/org/apache/iceberg/data/DataTest.java: ## @@ -486,4 +490,22 @@ public void testPrimitiveTypeDefaultValues(Type.PrimitiveType type, Object defau

Re: [I] ImportError: cannot import name 'DeclarativeBase' from 'sqlalchemy.orm' [iceberg-python]

2025-02-01 Thread via GitHub
chespinoza commented on issue #1514: URL: https://github.com/apache/iceberg-python/issues/1514#issuecomment-2628934786 @bsce20007 If you are using pyiceberg in an Airflow pipeline, you should be able (it's a good practice) to isolate your dependencies using the correct setup for your opera

Re: [PR] Test Hive: Fix TestHiveMetastore [iceberg]

2025-02-01 Thread via GitHub
deniskuzZ commented on PR #12140: URL: https://github.com/apache/iceberg/pull/12140#issuecomment-2628929186 @2MD, please run `./gradlew :iceberg-hive-metastore:spotlessApply` to resolve formatting issues -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [PR] fix: use upstream avro tag [iceberg-cpp]

2025-02-01 Thread via GitHub
wgtmac commented on PR #42: URL: https://github.com/apache/iceberg-cpp/pull/42#issuecomment-2628928197 @Fokko @Xuanwo Could you help merge this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] Test Hive: Fix TestHiveMetastore [iceberg]

2025-02-01 Thread via GitHub
RussellSpitzer commented on code in PR #12140: URL: https://github.com/apache/iceberg/pull/12140#discussion_r1938259640 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveMetastore.java: ## @@ -273,13 +274,13 @@ private void initConf(HiveConf conf, int port) { }

Re: [PR] Parquet: Implement Variant readers [iceberg]

2025-02-01 Thread via GitHub
RussellSpitzer commented on code in PR #12139: URL: https://github.com/apache/iceberg/pull/12139#discussion_r1938261512 ## core/src/main/java/org/apache/iceberg/variants/Variants.java: ## @@ -209,59 +213,59 @@ public static VariantPrimitive ofNull() { return new PrimitiveWr

Re: [PR] Test Hive: Fix TestHiveMetastore [iceberg]

2025-02-01 Thread via GitHub
RussellSpitzer commented on code in PR #12140: URL: https://github.com/apache/iceberg/pull/12140#discussion_r1938259640 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveMetastore.java: ## @@ -273,13 +274,13 @@ private void initConf(HiveConf conf, int port) { }

Re: [PR] Test Hive: Fix TestHiveMetastore [iceberg]

2025-02-01 Thread via GitHub
RussellSpitzer commented on code in PR #12140: URL: https://github.com/apache/iceberg/pull/12140#discussion_r1938260250 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveMetastore.java: ## @@ -273,13 +274,13 @@ private void initConf(HiveConf conf, int port) { }

Re: [PR] Spark: make delete file ratio configurable [iceberg]

2025-02-01 Thread via GitHub
nastra commented on code in PR #12148: URL: https://github.com/apache/iceberg/pull/12148#discussion_r1938256954 ## core/src/main/java/org/apache/iceberg/actions/SizeBasedDataRewriter.java: ## @@ -60,13 +76,26 @@ public Set validOptions() { return ImmutableSet.builder()

Re: [PR] Spark: make delete file ratio configurable [iceberg]

2025-02-01 Thread via GitHub
nastra commented on code in PR #12148: URL: https://github.com/apache/iceberg/pull/12148#discussion_r1938256751 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestSparkFileRewriter.java: ## @@ -375,6 +400,17 @@ public void testInvalidValuesForZOrderDataOption

Re: [PR] Spark: make delete file ratio configurable [iceberg]

2025-02-01 Thread via GitHub
nastra commented on code in PR #12148: URL: https://github.com/apache/iceberg/pull/12148#discussion_r1938255953 ## core/src/main/java/org/apache/iceberg/actions/SizeBasedDataRewriter.java: ## @@ -60,13 +76,26 @@ public Set validOptions() { return ImmutableSet.builder()

Re: [PR] Spark: make delete file ratio configurable [iceberg]

2025-02-01 Thread via GitHub
nastra commented on code in PR #12148: URL: https://github.com/apache/iceberg/pull/12148#discussion_r1938256698 ## spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/actions/TestSparkFileRewriter.java: ## @@ -304,6 +307,17 @@ public void testInvalidValuesForBinPackDataOptio

Re: [PR] Spark: make delete file ratio configurable [iceberg]

2025-02-01 Thread via GitHub
nastra commented on code in PR #12148: URL: https://github.com/apache/iceberg/pull/12148#discussion_r1938256603 ## core/src/main/java/org/apache/iceberg/actions/SizeBasedDataRewriter.java: ## @@ -60,13 +76,26 @@ public Set validOptions() { return ImmutableSet.builder()

Re: [PR] Spark: make delete file ratio configurable [iceberg]

2025-02-01 Thread via GitHub
nastra commented on code in PR #12148: URL: https://github.com/apache/iceberg/pull/12148#discussion_r1938256555 ## core/src/main/java/org/apache/iceberg/actions/SizeBasedDataRewriter.java: ## @@ -60,13 +76,26 @@ public Set validOptions() { return ImmutableSet.builder()

Re: [PR] Spark: make delete file ratio configurable [iceberg]

2025-02-01 Thread via GitHub
nastra commented on code in PR #12148: URL: https://github.com/apache/iceberg/pull/12148#discussion_r1938256175 ## docs/docs/spark-procedures.md: ## @@ -388,23 +388,24 @@ Iceberg can compact data files in parallel using Spark with the `rewriteDataFile Options # Gen

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-01 Thread via GitHub
mattmartin14 commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2628911415 @tscottcoombes1 - is it possible to modify the get_filter_list function I have above to handle this conversion without having to actually modify the entire table and create a

Re: [PR] Spark: make delete file ratio configurable [iceberg]

2025-02-01 Thread via GitHub
nastra commented on code in PR #12148: URL: https://github.com/apache/iceberg/pull/12148#discussion_r1938256383 ## spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/actions/TestSparkFileRewriter.java: ## @@ -302,6 +305,16 @@ public void testInvalidValuesForBinPackDataOptio

Re: [PR] #12081: "Add deleteFileThreshold parameter to SizeBasedDataRewriter, update logic, and include tests" [iceberg]

2025-02-01 Thread via GitHub
RussellSpitzer commented on PR #12133: URL: https://github.com/apache/iceberg/pull/12133#issuecomment-2628909228 > Could you please help me out in resolving the error. > > This build must be run with JDK 11 or 17 or 21 but was executed with JDK 23 > It looks like you may hav

Re: [PR] Spec: Fix current-version-id in View Spec example [iceberg]

2025-02-01 Thread via GitHub
nastra merged PR #12146: URL: https://github.com/apache/iceberg/pull/12146 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Core, API, Spec: Metadata Row Lineage [iceberg]

2025-02-01 Thread via GitHub
nastra merged PR #11948: URL: https://github.com/apache/iceberg/pull/11948 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Core, API, Spec: Metadata Row Lineage [iceberg]

2025-02-01 Thread via GitHub
nastra commented on code in PR #11948: URL: https://github.com/apache/iceberg/pull/11948#discussion_r1938252562 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -312,6 +341,8 @@ public void testBackwardCompat() throws Exception { .isEqualTo(previou

Re: [I] Iceberg-rust Write support [iceberg-rust]

2025-02-01 Thread via GitHub
ZENOTME commented on issue #700: URL: https://github.com/apache/iceberg-rust/issues/700#issuecomment-2628898503 Hi @Fokko, CMIIW, I think for Add files to add existing Parquet files to a table, #345 is not the corresponding issue? I create a new issue #932 to track appending existing data

Re: [PR] Core, API, Spec: Metadata Row Lineage [iceberg]

2025-02-01 Thread via GitHub
RussellSpitzer commented on code in PR #11948: URL: https://github.com/apache/iceberg/pull/11948#discussion_r1938246839 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -312,6 +341,8 @@ public void testBackwardCompat() throws Exception { .isEqualTo

Re: [PR] Core, API, Spec: Metadata Row Lineage [iceberg]

2025-02-01 Thread via GitHub
RussellSpitzer commented on code in PR #11948: URL: https://github.com/apache/iceberg/pull/11948#discussion_r1938246608 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -312,6 +341,8 @@ public void testBackwardCompat() throws Exception { .isEqualTo

Re: [PR] 1191:Added Docstring and Fixed linter error [iceberg-python]

2025-02-01 Thread via GitHub
gayatrikate04 commented on PR #1590: URL: https://github.com/apache/iceberg-python/pull/1590#issuecomment-2628893413 I've removed the mkdocs = "^1.6.1" dependency -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Spark 3.5: Remove use of File.Separator in RewriteTablePath [iceberg]

2025-02-01 Thread via GitHub
RussellSpitzer commented on PR #12066: URL: https://github.com/apache/iceberg/pull/12066#issuecomment-2628890720 Thanks @manuzhang for the PR. Big thanks to @szehon-ho for noting the issue with File.separator , also thanks to @nastra for the review. -- This is an automated message from th

Re: [PR] Spark 3.5: Remove use of File.Separator in RewriteTablePath [iceberg]

2025-02-01 Thread via GitHub
RussellSpitzer merged PR #12066: URL: https://github.com/apache/iceberg/pull/12066 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Implement update for `remove-snapshots` action [iceberg-python]

2025-02-01 Thread via GitHub
grihabor commented on code in PR #1561: URL: https://github.com/apache/iceberg-python/pull/1561#discussion_r1938238860 ## pyiceberg/table/update/__init__.py: ## @@ -455,6 +455,19 @@ def _(update: SetSnapshotRefUpdate, base_metadata: TableMetadata, context: _Tabl return bas

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-01 Thread via GitHub
tscottcoombes1 commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2628873390 @mattmartin14 datetime.date is not handled in this function: https://github.com/apache/iceberg-python/blob/main/pyiceberg/expressions/literals.py#L124 from the

[PR] Implement update for `remove-snapshot-ref` action [iceberg-python]

2025-02-01 Thread via GitHub
grihabor opened a new pull request, #1598: URL: https://github.com/apache/iceberg-python/pull/1598 Part of https://github.com/apache/iceberg-python/pull/1561 Closes https://github.com/apache/iceberg-python/pull/822 -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Flink: Add null check to writers to prevent resurrecting null values [iceberg]

2025-02-01 Thread via GitHub
mxm commented on code in PR #12049: URL: https://github.com/apache/iceberg/pull/12049#discussion_r1938226630 ## data/src/test/java/org/apache/iceberg/data/DataTest.java: ## @@ -486,4 +490,22 @@ public void testPrimitiveTypeDefaultValues(Type.PrimitiveType type, Object defau

[PR] add: make delete file ratio configurable [iceberg]

2025-02-01 Thread via GitHub
MichaelDeSteven opened a new pull request, #12148: URL: https://github.com/apache/iceberg/pull/12148 #12081 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,