Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-06 Thread via GitHub
corleyma commented on code in PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#discussion_r1946056996 ## pyiceberg/table/upsert_util.py: ## @@ -0,0 +1,153 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] chore: use shared containers for integration tests [iceberg-rust]

2025-02-06 Thread via GitHub
gruuya commented on code in PR #924: URL: https://github.com/apache/iceberg-rust/pull/924#discussion_r1946050715 ## crates/integration_tests/tests/shared_tests/mod.rs: ## @@ -0,0 +1,65 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lice

Re: [PR] Auth Manager API part 5: SigV4 Auth Manager [iceberg]

2025-02-06 Thread via GitHub
nastra commented on code in PR #11995: URL: https://github.com/apache/iceberg/pull/11995#discussion_r1946042519 ## aws/src/test/java/org/apache/iceberg/aws/TestRESTSigV4AuthSession.java: ## @@ -0,0 +1,294 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] support source watermark for flink sql windows [iceberg]

2025-02-06 Thread via GitHub
pvary commented on code in PR #12191: URL: https://github.com/apache/iceberg/pull/12191#discussion_r1946014902 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceSql.java: ## @@ -162,4 +173,74 @@ public void testWatermarkOptionsDescending() throw

Re: [PR] support source watermark for flink sql windows [iceberg]

2025-02-06 Thread via GitHub
pvary commented on code in PR #12191: URL: https://github.com/apache/iceberg/pull/12191#discussion_r1946012368 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/SqlBase.java: ## @@ -86,7 +86,7 @@ protected void dropDatabase(String database, boolean ifExists) { sq

Re: [PR] Support changelog scan for table with delete files [iceberg]

2025-02-06 Thread via GitHub
czy006 commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2642038961 Looking forward to this feature. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Core, Spark: Exclude non live content file in RewriteTablePathUtil [iceberg]

2025-02-06 Thread via GitHub
szehon-ho commented on PR #12006: URL: https://github.com/apache/iceberg/pull/12006#issuecomment-2641990129 Merged, thanks @dramaticlly ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Spark 3.4: Remove use of File.Separator in RewriteTablePath [iceberg]

2025-02-06 Thread via GitHub
szehon-ho commented on PR #12173: URL: https://github.com/apache/iceberg/pull/12173#issuecomment-2641999247 Merged, thanks @manuzhang and also @dramaticlly @ebyhr for previous reviews! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Spark 3.4: Remove use of File.Separator in RewriteTablePath [iceberg]

2025-02-06 Thread via GitHub
szehon-ho merged PR #12173: URL: https://github.com/apache/iceberg/pull/12173 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Core, Spark: Exclude non live content file in RewriteTablePathUtil [iceberg]

2025-02-06 Thread via GitHub
dramaticlly commented on PR #12006: URL: https://github.com/apache/iceberg/pull/12006#issuecomment-2641991012 Thank you @szehon-ho for reviewing my change! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Core, Spark: Exclude non live content file in RewriteTablePathUtil [iceberg]

2025-02-06 Thread via GitHub
szehon-ho merged PR #12006: URL: https://github.com/apache/iceberg/pull/12006 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Docs: add apache amoro(incubating) with iceberg (#11965) [iceberg]

2025-02-06 Thread via GitHub
czy006 commented on PR #11966: URL: https://github.com/apache/iceberg/pull/11966#issuecomment-2641949029 Hi @Fokko, can help look amoro project description with iceberg? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [I] ClassNotFoundException when using the flinksql to write iceberg table [iceberg]

2025-02-06 Thread via GitHub
tmph2003 commented on issue #8947: URL: https://github.com/apache/iceberg/issues/8947#issuecomment-2641917177 how do you fix it ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Spark 3.4: Remove use of File.Separator in RewriteTablePath [iceberg]

2025-02-06 Thread via GitHub
manuzhang commented on PR #12173: URL: https://github.com/apache/iceberg/pull/12173#issuecomment-2641888019 @RussellSpitzer @szehon-ho @nastra please help review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] software.amazon.awssdk.services.s3.model.S3Exception: The bucket you are attempting to access must be addressed using the specified endpoint. [iceberg]

2025-02-06 Thread via GitHub
timoha commented on issue #11997: URL: https://github.com/apache/iceberg/issues/11997#issuecomment-2641847579 Running into the same problem when trying to access an iceberg table in `us-east-1` bucket from `us-east-2` account. We are using Glue for our catalog which also lives in `us-east-2

[PR] Filter rows directly from pa.RecordBatch [iceberg-python]

2025-02-06 Thread via GitHub
gabeiglio opened a new pull request, #1621: URL: https://github.com/apache/iceberg-python/pull/1621 This [PR](https://github.com/apache/arrow/issues/39220) from Apache Arrow was merged to allow to filter with a boolean expression directly on `pa.RecordBatch`. I believe pyiceberg is c

Re: [PR] Core: Fix RewriteTablePath Incremental Replication [iceberg]

2025-02-06 Thread via GitHub
dramaticlly commented on code in PR #12172: URL: https://github.com/apache/iceberg/pull/12172#discussion_r1945867503 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteTablePathsAction.java: ## @@ -235,6 +235,51 @@ public void testStartVersion() throws

Re: [PR] Core: Fix RewriteTablePath Incremental Replication [iceberg]

2025-02-06 Thread via GitHub
dramaticlly commented on PR #12172: URL: https://github.com/apache/iceberg/pull/12172#issuecomment-2641625230 So if I understand the problem correctly, today we filter the manifest based on snapshot when rewrites manifest list for incremental rewrite. snapshot based filtering shall only app

Re: [PR] Docs: Minor improvements to Spark Procedures docs [iceberg]

2025-02-06 Thread via GitHub
szehon-ho merged PR #12190: URL: https://github.com/apache/iceberg/pull/12190 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Docs: Add rewrite-table-path in spark procedure [iceberg]

2025-02-06 Thread via GitHub
flyrain commented on code in PR #12115: URL: https://github.com/apache/iceberg/pull/12115#discussion_r1945855015 ## docs/docs/spark-procedures.md: ## @@ -972,4 +972,100 @@ CALL catalog_name.system.compute_table_stats(table => 'my_table', snapshot_id => Collect statistics of th

Re: [PR] Docs: Minor improvements to Spark Procedures docs [iceberg]

2025-02-06 Thread via GitHub
szehon-ho commented on PR #12190: URL: https://github.com/apache/iceberg/pull/12190#issuecomment-2641636703 Thanks, good catch @smaheshwar-pltr -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] Docs: Minor improvements to Spark Procedures docs [iceberg]

2025-02-06 Thread via GitHub
smaheshwar-pltr commented on PR #12190: URL: https://github.com/apache/iceberg/pull/12190#issuecomment-2641640400 Thank you for reviewing 😄 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-06 Thread via GitHub
abrunner94 commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2641591748 Looking forward to this in the upcoming release! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Core: Fix RewriteTablePath Incremental Replication [iceberg]

2025-02-06 Thread via GitHub
dramaticlly commented on PR #12172: URL: https://github.com/apache/iceberg/pull/12172#issuecomment-2641532788 I did some look on this, I think right now the only strict delta manifest-list/manifests/data files are rewritten if `start_version` is provided. So for incremental rewrite and copy

[PR] support source watermark for flink sql windows [iceberg]

2025-02-06 Thread via GitHub
swapna267 opened a new pull request, #12191: URL: https://github.com/apache/iceberg/pull/12191 Iceberg Source to support Source Watermark, so it can be used in Flink WINDOW functions. https://github.com/apache/flink/blob/release-1.18/flink-table/flink-table-common/src/main/java/org/apache/

Re: [PR] Core: Try create Iceberg metadata table for Jdbc catalog in initialization [iceberg]

2025-02-06 Thread via GitHub
github-actions[bot] commented on PR #11427: URL: https://github.com/apache/iceberg/pull/11427#issuecomment-2641465243 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] backport #11301(rowconverter) to Flink 1.19 and 1.18 [iceberg]

2025-02-06 Thread via GitHub
github-actions[bot] commented on PR #11826: URL: https://github.com/apache/iceberg/pull/11826#issuecomment-2641465537 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Support changelog scan for table with delete files [iceberg]

2025-02-06 Thread via GitHub
github-actions[bot] commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2641465050 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Kafka Connect: Add table to topics mapping property [iceberg]

2025-02-06 Thread via GitHub
github-actions[bot] commented on PR #10422: URL: https://github.com/apache/iceberg/pull/10422#issuecomment-2641464688 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Docs: Add rewrite-table-path in spark procedure [iceberg]

2025-02-06 Thread via GitHub
dramaticlly commented on code in PR #12115: URL: https://github.com/apache/iceberg/pull/12115#discussion_r1945680164 ## docs/docs/spark-procedures.md: ## @@ -976,18 +976,18 @@ CALL catalog_name.system.compute_table_stats(table => 'my_table', snapshot_id => ## Table Replicati

Re: [PR] Docs: Add rewrite-table-path in spark procedure [iceberg]

2025-02-06 Thread via GitHub
dramaticlly commented on code in PR #12115: URL: https://github.com/apache/iceberg/pull/12115#discussion_r1945679456 ## docs/docs/spark-procedures.md: ## @@ -972,4 +972,101 @@ CALL catalog_name.system.compute_table_stats(table => 'my_table', snapshot_id => Collect statistics o

Re: [PR] spec: Remove `source-ids` for `V{1,2}` tables [iceberg]

2025-02-06 Thread via GitHub
szehon-ho commented on PR #12161: URL: https://github.com/apache/iceberg/pull/12161#issuecomment-2641373672 Thanks @Fokko initially was hoping that we could support it for v1/v2 tables but understand the issues you are pointing out. I guess we can patiently wait v3 tables then! -- This

Re: [PR] Docs: Add rewrite-table-path in spark procedure [iceberg]

2025-02-06 Thread via GitHub
szehon-ho commented on code in PR #12115: URL: https://github.com/apache/iceberg/pull/12115#discussion_r1945624171 ## docs/docs/spark-procedures.md: ## @@ -976,18 +976,18 @@ CALL catalog_name.system.compute_table_stats(table => 'my_table', snapshot_id => ## Table Replication

Re: [PR] Docs: Minor improvements to Spark Procedures docs [iceberg]

2025-02-06 Thread via GitHub
smaheshwar-pltr commented on code in PR #12190: URL: https://github.com/apache/iceberg/pull/12190#discussion_r1945626960 ## docs/docs/spark-procedures.md: ## @@ -853,7 +853,7 @@ CALL spark_catalog.system.create_changelog_view( table => 'db.tbl', options => map('start-snaps

[PR] Docs: Minor improvements to Spark Procedures docs [iceberg]

2025-02-06 Thread via GitHub
smaheshwar-pltr opened a new pull request, #12190: URL: https://github.com/apache/iceberg/pull/12190 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

Re: [PR] Add support for `write.data.path` [iceberg-python]

2025-02-06 Thread via GitHub
smaheshwar-pltr commented on code in PR #1611: URL: https://github.com/apache/iceberg-python/pull/1611#discussion_r1945618221 ## mkdocs/docs/configuration.md: ## @@ -54,18 +54,19 @@ Iceberg tables support table properties to configure table behavior. ### Write options -| K

Re: [PR] Parquet, Arrow: Refactor vectorized reader [iceberg]

2025-02-06 Thread via GitHub
flyrain commented on PR #9772: URL: https://github.com/apache/iceberg/pull/9772#issuecomment-2641355085 Hi @wgtmac , can we resume this PR? I can help with review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

[PR] Add support for view related operations [iceberg-go]

2025-02-06 Thread via GitHub
dttung2905 opened a new pull request, #290: URL: https://github.com/apache/iceberg-go/pull/290 From the [REST catalog API](https://raw.githubusercontent.com/apache/iceberg/refs/heads/main/open-api/rest-catalog-open-api.yaml), we can support a few more operation related to views -- This

Re: [PR] Update LICENSE/NOTICE for spark-runtime 3.3 and 3.4 [iceberg]

2025-02-06 Thread via GitHub
rdblue merged PR #12189: URL: https://github.com/apache/iceberg/pull/12189 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Update LICENSE/NOTICE in flink-runtime jar files [iceberg]

2025-02-06 Thread via GitHub
rdblue merged PR #12188: URL: https://github.com/apache/iceberg/pull/12188 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Update LICENSE/NOTICE in flink-runtime jar files [iceberg]

2025-02-06 Thread via GitHub
rdblue commented on PR #12188: URL: https://github.com/apache/iceberg/pull/12188#issuecomment-2641332325 Thanks for double checking the 1.20 changes, @amogh-jahagirdar! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

[PR] Build: Bump sqlalchemy from 2.0.37 to 2.0.38 [iceberg-python]

2025-02-06 Thread via GitHub
dependabot[bot] opened a new pull request, #1620: URL: https://github.com/apache/iceberg-python/pull/1620 Bumps [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) from 2.0.37 to 2.0.38. Release notes Sourced from https://github.com/sqlalchemy/sqlalchemy/releases";>sqlalchemy's

Re: [PR] Remove deprecated `WRITE_METADATA_LOCATION` and `WRITE_FOLDER_STORAGE_LOCATION` [iceberg]

2025-02-06 Thread via GitHub
dramaticlly commented on code in PR #12174: URL: https://github.com/apache/iceberg/pull/12174#discussion_r1945589569 ## core/src/main/java/org/apache/iceberg/LocationProviders.java: ## @@ -137,13 +134,7 @@ static class ObjectStoreLocationProvider implements LocationProvider {

Re: [PR] Remove deprecated `WRITE_METADATA_LOCATION` and `WRITE_FOLDER_STORAGE_LOCATION` [iceberg]

2025-02-06 Thread via GitHub
dramaticlly commented on code in PR #12174: URL: https://github.com/apache/iceberg/pull/12174#discussion_r1945589569 ## core/src/main/java/org/apache/iceberg/LocationProviders.java: ## @@ -137,13 +134,7 @@ static class ObjectStoreLocationProvider implements LocationProvider {

Re: [PR] Remove deprecated `WRITE_METADATA_LOCATION` and `WRITE_FOLDER_STORAGE_LOCATION` [iceberg]

2025-02-06 Thread via GitHub
dramaticlly commented on code in PR #12174: URL: https://github.com/apache/iceberg/pull/12174#discussion_r1945589569 ## core/src/main/java/org/apache/iceberg/LocationProviders.java: ## @@ -137,13 +134,7 @@ static class ObjectStoreLocationProvider implements LocationProvider {

Re: [PR] Remove deprecated `WRITE_METADATA_LOCATION` and `WRITE_FOLDER_STORAGE_LOCATION` [iceberg]

2025-02-06 Thread via GitHub
dramaticlly commented on code in PR #12174: URL: https://github.com/apache/iceberg/pull/12174#discussion_r1945589569 ## core/src/main/java/org/apache/iceberg/LocationProviders.java: ## @@ -137,13 +134,7 @@ static class ObjectStoreLocationProvider implements LocationProvider {

Re: [PR] Remove deprecated `WRITE_METADATA_LOCATION` and `WRITE_FOLDER_STORAGE_LOCATION` [iceberg]

2025-02-06 Thread via GitHub
dramaticlly commented on code in PR #12174: URL: https://github.com/apache/iceberg/pull/12174#discussion_r1945589569 ## core/src/main/java/org/apache/iceberg/LocationProviders.java: ## @@ -137,13 +134,7 @@ static class ObjectStoreLocationProvider implements LocationProvider {

Re: [PR] Remove deprecated `WRITE_METADATA_LOCATION` and `WRITE_FOLDER_STORAGE_LOCATION` [iceberg]

2025-02-06 Thread via GitHub
dramaticlly commented on code in PR #12174: URL: https://github.com/apache/iceberg/pull/12174#discussion_r1945589569 ## core/src/main/java/org/apache/iceberg/LocationProviders.java: ## @@ -137,13 +134,7 @@ static class ObjectStoreLocationProvider implements LocationProvider {

Re: [I] Support metadata compaction [iceberg-python]

2025-02-06 Thread via GitHub
kevinjqliu commented on issue #270: URL: https://github.com/apache/iceberg-python/issues/270#issuecomment-2641296013 sure thing @amitgilad3 Based on the conversation above, it looks like some of the components are already implemented -- This is an automated message from the Apache Gi

Re: [PR] Added support for Polars DataFrame and LazyFarame [iceberg-python]

2025-02-06 Thread via GitHub
yigal-rozenberg commented on PR #1614: URL: https://github.com/apache/iceberg-python/pull/1614#issuecomment-2641279700 love the idea! Will research in how to implement. In the mean time, I believe this specific change request is straight forward, and allow both DataFrame, and LazyFrame

Re: [I] Delete orphan files [iceberg-python]

2025-02-06 Thread via GitHub
kevinjqliu commented on issue #1200: URL: https://github.com/apache/iceberg-python/issues/1200#issuecomment-2641289104 > Extract Metadata-Tracked Files we might want to use all_files and all_metadata_files. `files` only gets the data files for the current snapshot -- This is an au

Re: [PR] Add support for `write.data.path` [iceberg-python]

2025-02-06 Thread via GitHub
kevinjqliu commented on code in PR #1611: URL: https://github.com/apache/iceberg-python/pull/1611#discussion_r1945565345 ## mkdocs/docs/configuration.md: ## @@ -239,9 +240,9 @@ When several files are stored under the same prefix, cloud object stores such as resulting in slowdo

Re: [PR] Core, Spark: Exclude non live content file in RewriteTablePathUtil [iceberg]

2025-02-06 Thread via GitHub
dramaticlly commented on code in PR #12006: URL: https://github.com/apache/iceberg/pull/12006#discussion_r1945554585 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteTablePathsAction.java: ## @@ -940,16 +1024,20 @@ protected void checkFileNum(

Re: [PR] Docs: Add rewrite-table-path in spark procedure [iceberg]

2025-02-06 Thread via GitHub
dramaticlly commented on code in PR #12115: URL: https://github.com/apache/iceberg/pull/12115#discussion_r1945547732 ## docs/docs/spark-procedures.md: ## @@ -972,4 +972,91 @@ CALL catalog_name.system.compute_table_stats(table => 'my_table', snapshot_id => Collect statistics of

Re: [PR] Core, Spark: Exclude non live content file in RewriteTablePathUtil [iceberg]

2025-02-06 Thread via GitHub
szehon-ho commented on code in PR #12006: URL: https://github.com/apache/iceberg/pull/12006#discussion_r1945524012 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteTablePathsAction.java: ## @@ -940,16 +1024,20 @@ protected void checkFileNum(

Re: [PR] Core, Spark: Exclude non live content file in RewriteTablePathUtil [iceberg]

2025-02-06 Thread via GitHub
szehon-ho commented on code in PR #12006: URL: https://github.com/apache/iceberg/pull/12006#discussion_r1945524012 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteTablePathsAction.java: ## @@ -940,16 +1024,20 @@ protected void checkFileNum(

Re: [PR] Core, Spark: Exclude non live content file in RewriteTablePathUtil [iceberg]

2025-02-06 Thread via GitHub
dramaticlly commented on code in PR #12006: URL: https://github.com/apache/iceberg/pull/12006#discussion_r1945480537 ## core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java: ## @@ -354,7 +354,10 @@ private static RewriteResult writeDataFileEntry( DataFile newData

Re: [PR] Added support for Polars DataFrame and LazyFarame [iceberg-python]

2025-02-06 Thread via GitHub
corleyma commented on PR #1614: URL: https://github.com/apache/iceberg-python/pull/1614#issuecomment-2641089912 Separately, rather than adding more library-specific conversion code, it might make sense for pyiceberg to start leveraging the [PyCapsule protocol](https://arrow.apache.org/docs/

Re: [PR] Update LICENSE/NOTICE in flink-runtime jar files [iceberg]

2025-02-06 Thread via GitHub
amogh-jahagirdar commented on code in PR #12188: URL: https://github.com/apache/iceberg/pull/12188#discussion_r1945416902 ## flink/v1.20/flink-runtime/LICENSE: ## @@ -267,14 +267,6 @@ License: http://www.apache.org/licenses/LICENSE-2.0 ---

Re: [PR] Added support for Polars DataFrame and LazyFarame [iceberg-python]

2025-02-06 Thread via GitHub
corleyma commented on PR #1614: URL: https://github.com/apache/iceberg-python/pull/1614#issuecomment-2641053258 > Polars 'scan_iceberg' uses PyIceberg to create the LazyFrame: > https://github.com/pola-rs/polars/blob/9359ed576d972dce257346fcd62c8857f3d23277/py-polars/polars/io/iceberg.py#

[PR] Update LICENSE/NOTICE for spark-runtime 3.3 and 3.4 [iceberg]

2025-02-06 Thread via GitHub
jbonofre opened a new pull request, #12189: URL: https://github.com/apache/iceberg/pull/12189 I checked the spark-runtime 3.3/3.4/3.5 jar files and they are all equivalent. So I "backport" `LICENSE` and `NOTICE` from spark-runtime 3.5 to 3.3/3.4. -- This is an automated message fro

[PR] Update LICENSE/NOTICE in flink-runtime jar files [iceberg]

2025-02-06 Thread via GitHub
jbonofre opened a new pull request, #12188: URL: https://github.com/apache/iceberg/pull/12188 This PR: - remove airlift slice from `LICENSE` (I don't see it in any flink-runtime jar) - add Codahale Metrics (present in all flink-runtime jars) - I check flink-runtime 1.18 and 1.19 and

Re: [PR] Fix NOTICE and LICENSE in the flink-runtime jar [iceberg]

2025-02-06 Thread via GitHub
jbonofre commented on code in PR #12145: URL: https://github.com/apache/iceberg/pull/12145#discussion_r1945390830 ## flink/v1.20/flink-runtime/LICENSE: ## @@ -508,3 +466,63 @@ This binary artifact contains failsafe. Copyright: Jonathan Halterman and friends Home page: https://

Re: [PR] [infra] add testpypi nightly build [iceberg-python]

2025-02-06 Thread via GitHub
kevinjqliu commented on PR #1601: URL: https://github.com/apache/iceberg-python/pull/1601#issuecomment-2640903507 oh wait @Fokko i think thats a false alarm. i was looking at a previous run which had the RC in the version. i yanked that one and now its back to normal https://test.pypi.org/p

Re: [PR] [infra] add testpypi nightly build [iceberg-python]

2025-02-06 Thread via GitHub
kevinjqliu commented on PR #1601: URL: https://github.com/apache/iceberg-python/pull/1601#issuecomment-2640893364 Thanks! I couldnt find anything about the dev -> rc conversion. I just added trusted publisher pyiceberg on testpypi ![Screenshot 2025-02-06 at 12 09 32  PM](https://gi

Re: [PR] [infra] add testpypi nightly build [iceberg-python]

2025-02-06 Thread via GitHub
Fokko commented on PR #1601: URL: https://github.com/apache/iceberg-python/pull/1601#issuecomment-2640880595 @kevinjqliu I just bumped you to owner 👍 Let me check why it converts `rc` into `dev` for the artifacts -- This is an automated message from the Apache Git Service. To respo

Re: [PR] [infra] add testpypi nightly build [iceberg-python]

2025-02-06 Thread via GitHub
kevinjqliu commented on PR #1601: URL: https://github.com/apache/iceberg-python/pull/1601#issuecomment-2640876441 oh i see, only "owner" can make this change ([from pypi: `What collaborator roles are available for a project on PyPI?`](https://pypi.org/help/)) This is what it looks lik

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-06 Thread via GitHub
mattmartin14 commented on code in PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#discussion_r1945302934 ## tests/table/test_upsert.py: ## @@ -0,0 +1,327 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] Custom fileio docs [iceberg-python]

2025-02-06 Thread via GitHub
summermousa-vendia commented on PR #1238: URL: https://github.com/apache/iceberg-python/pull/1238#issuecomment-2640750240 @kevinjqliu , @sikehish if this isn't being actively worked, are there any objections to me making the final adjustments? -- This is an automated message from the Apac

Re: [I] There is a bug with the new version of boto used by s3fs that prevents writes to non-AWS S3 buckets [iceberg-python]

2025-02-06 Thread via GitHub
Fokko commented on issue #1546: URL: https://github.com/apache/iceberg-python/issues/1546#issuecomment-2640775452 @ryanovas That's a good point. I'm fine with a flag at the catalog as well. We have a contributing guide [here](https://py.iceberg.apache.org/contributing/). We want to d

Re: [I] There is a bug with the new version of boto used by s3fs that prevents writes to non-AWS S3 buckets [iceberg-python]

2025-02-06 Thread via GitHub
ryanovas commented on issue #1546: URL: https://github.com/apache/iceberg-python/issues/1546#issuecomment-2640749333 I might suggest we allow for a way to set a flag maybe when we init a catalog? or something similar as the issue is only present for s3-compatible storage and AWS peeps prob

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-06 Thread via GitHub
mattmartin14 commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2640746041 @Fokko @tscottcoombes1 @kevinjqliu - another round of updates completed. As of now, i think these are the only ouststanding items: - the TOML lock file for poetry might

Re: [I] There is a bug with the new version of boto used by s3fs that prevents writes to non-AWS S3 buckets [iceberg-python]

2025-02-06 Thread via GitHub
Fokko commented on issue #1546: URL: https://github.com/apache/iceberg-python/issues/1546#issuecomment-2640740192 Thanks for keeping track of this @ryanovas. I think we can do an if-else based on the `s3fs` version that's installed. In the case of `≥1.36`, then we should pass in this param

Re: [I] Add support for encrypted/protected data type in iceberg table [iceberg-python]

2025-02-06 Thread via GitHub
Fokko commented on issue #1582: URL: https://github.com/apache/iceberg-python/issues/1582#issuecomment-2640733293 Thanks for the additional context @yigal-rozenberg. I find this very interesting. Also, I know certain companies that use this pattern for GDPR, where they erase the decryption

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-06 Thread via GitHub
mattmartin14 commented on code in PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#discussion_r1945259484 ## pyiceberg/table/__init__.py: ## @@ -1064,6 +1067,78 @@ def name_mapping(self) -> Optional[NameMapping]: """Return the table's field-id NameMappi

Re: [I] Reading Latest Metadata Json File of Iceberg Table [iceberg-python]

2025-02-06 Thread via GitHub
Fokko closed issue #1585: Reading Latest Metadata Json File of Iceberg Table URL: https://github.com/apache/iceberg-python/issues/1585 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [I] Reading Latest Metadata Json File of Iceberg Table [iceberg-python]

2025-02-06 Thread via GitHub
Fokko commented on issue #1585: URL: https://github.com/apache/iceberg-python/issues/1585#issuecomment-2640727316 @heman026 Thanks for raising this issue. The easiest way to do this is just to call `py_table.metadata_location`: https://github.com/user-attachments/assets/4ee6fbd8-4ba3

Re: [PR] Auth Manager API part 4: RESTClient, HTTPClient [iceberg]

2025-02-06 Thread via GitHub
adutra commented on code in PR #11992: URL: https://github.com/apache/iceberg/pull/11992#discussion_r1945236123 ## aws/src/main/java/org/apache/iceberg/aws/s3/VendedCredentialsProvider.java: ## @@ -75,10 +76,13 @@ private RESTClient httpClient() { if (null == client) {

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-06 Thread via GitHub
mattmartin14 commented on code in PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#discussion_r1945235685 ## tests/table/test_upsert.py: ## @@ -0,0 +1,327 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [I] There is a bug with the new version of boto used by s3fs that prevents writes to non-AWS S3 buckets [iceberg-python]

2025-02-06 Thread via GitHub
ryanovas commented on issue #1546: URL: https://github.com/apache/iceberg-python/issues/1546#issuecomment-2640686798 In the s3fs issue it looks like there's a config we can pass? I believe it was suggested already by @maarek but confirmed in this comment on the issue: https://github.com/fs

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-06 Thread via GitHub
mattmartin14 commented on code in PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#discussion_r1945220594 ## pyiceberg/table/__init__.py: ## @@ -1064,6 +1066,97 @@ def name_mapping(self) -> Optional[NameMapping]: """Return the table's field-id NameMappi

[PR] [infra] nightly pypi build for `pyiceberg_core` [iceberg-rust]

2025-02-06 Thread via GitHub
kevinjqliu opened a new pull request, #948: URL: https://github.com/apache/iceberg-rust/pull/948 Test run on my fork https://github.com/kevinjqliu/iceberg-rust/actions/runs/13183943217 Verified sdist and wheel library versions locally -- This is an automated message from the Apache Gi

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-06 Thread via GitHub
mattmartin14 commented on code in PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#discussion_r1945207485 ## pyiceberg/table/upsert_util.py: ## @@ -0,0 +1,158 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

Re: [PR] chore: use shared containers for integration tests [iceberg-rust]

2025-02-06 Thread via GitHub
ZENOTME commented on PR #924: URL: https://github.com/apache/iceberg-rust/pull/924#issuecomment-2640582262 > Hey @ZENOTME @Fokko, can you also take a look at this proposal? Thanks! @gruuya It's a great job to improve our ci. the idea and code change LGTM. -- This is an automated me

Re: [PR] chore: use shared containers for integration tests [iceberg-rust]

2025-02-06 Thread via GitHub
ZENOTME commented on code in PR #924: URL: https://github.com/apache/iceberg-rust/pull/924#discussion_r1945180516 ## crates/integration_tests/tests/shared_tests/mod.rs: ## @@ -0,0 +1,65 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lic

Re: [PR] Auth Manager API part 5: SigV4 Auth Manager [iceberg]

2025-02-06 Thread via GitHub
danielcweeks commented on PR #11995: URL: https://github.com/apache/iceberg/pull/11995#issuecomment-2640585014 Only one minor comment, but I'll wait for @nastra to look this over as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] Auth Manager API part 5: SigV4 Auth Manager [iceberg]

2025-02-06 Thread via GitHub
danielcweeks commented on code in PR #11995: URL: https://github.com/apache/iceberg/pull/11995#discussion_r1945167254 ## aws/src/main/java/org/apache/iceberg/aws/RESTSigV4AuthSession.java: ## @@ -0,0 +1,154 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Spec: Support geo type [iceberg]

2025-02-06 Thread via GitHub
rdblue commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1945163174 ## format/spec.md: ## @@ -468,7 +494,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ | Transform name| Description

Re: [PR] Fix NOTICE and LICENSE in the flink-runtime jar [iceberg]

2025-02-06 Thread via GitHub
rdblue merged PR #12145: URL: https://github.com/apache/iceberg/pull/12145 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Auth Manager API part 4: RESTClient, HTTPClient [iceberg]

2025-02-06 Thread via GitHub
danielcweeks merged PR #11992: URL: https://github.com/apache/iceberg/pull/11992 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceb

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-06 Thread via GitHub
mattmartin14 commented on code in PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#discussion_r1945153598 ## pyiceberg/table/__init__.py: ## @@ -1064,6 +1067,78 @@ def name_mapping(self) -> Optional[NameMapping]: """Return the table's field-id NameMappi

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-06 Thread via GitHub
mattmartin14 commented on code in PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#discussion_r1945153041 ## pyiceberg/table/__init__.py: ## @@ -1064,6 +1067,78 @@ def name_mapping(self) -> Optional[NameMapping]: """Return the table's field-id NameMappi

Re: [PR] Auth Manager API part 4: RESTClient, HTTPClient [iceberg]

2025-02-06 Thread via GitHub
danielcweeks commented on code in PR #11992: URL: https://github.com/apache/iceberg/pull/11992#discussion_r1945150977 ## aws/src/main/java/org/apache/iceberg/aws/s3/VendedCredentialsProvider.java: ## @@ -75,10 +76,13 @@ private RESTClient httpClient() { if (null == client)

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-06 Thread via GitHub
mattmartin14 commented on code in PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#discussion_r1945145462 ## pyiceberg/table/__init__.py: ## @@ -1064,6 +1066,97 @@ def name_mapping(self) -> Optional[NameMapping]: """Return the table's field-id NameMappi

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-06 Thread via GitHub
mattmartin14 commented on code in PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#discussion_r1945145894 ## pyiceberg/table/__init__.py: ## @@ -1064,6 +1066,97 @@ def name_mapping(self) -> Optional[NameMapping]: """Return the table's field-id NameMappi

Re: [PR] Fix NOTICE and LICENSE in the flink-runtime jar [iceberg]

2025-02-06 Thread via GitHub
jbonofre commented on code in PR #12145: URL: https://github.com/apache/iceberg/pull/12145#discussion_r1945145867 ## flink/v1.20/flink-runtime/NOTICE: ## @@ -63,29 +50,339 @@ NOTICE file: -Thi

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-06 Thread via GitHub
mattmartin14 commented on code in PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#discussion_r1945144251 ## pyiceberg/table/upsert_util.py: ## @@ -0,0 +1,154 @@ + +# Licensed to the Apache Software Foundation (ASF) under one Review Comment: fixed -- Thi

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-06 Thread via GitHub
mattmartin14 commented on code in PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#discussion_r1945143597 ## pyiceberg/table/__init__.py: ## @@ -1064,6 +1066,97 @@ def name_mapping(self) -> Optional[NameMapping]: """Return the table's field-id NameMappi

Re: [PR] Fix NOTICE and LICENSE in the flink-runtime jar [iceberg]

2025-02-06 Thread via GitHub
jbonofre commented on code in PR #12145: URL: https://github.com/apache/iceberg/pull/12145#discussion_r1945137159 ## flink/v1.20/flink-runtime/NOTICE: ## @@ -63,29 +50,339 @@ NOTICE file: -Thi

Re: [PR] Fix NOTICE and LICENSE in the flink-runtime jar [iceberg]

2025-02-06 Thread via GitHub
rdblue commented on code in PR #12145: URL: https://github.com/apache/iceberg/pull/12145#discussion_r1945120622 ## flink/v1.20/flink-runtime/NOTICE: ## @@ -63,29 +50,339 @@ NOTICE file: -This

  1   2   >