Re: [PR] Add iceberg_arrow library [iceberg-cpp]

2025-01-06 Thread via GitHub
wgtmac commented on code in PR #6: URL: https://github.com/apache/iceberg-cpp/pull/6#discussion_r1904813642 ## cmake_modules/BuildUtils.cmake: ## @@ -201,17 +202,26 @@ function(ADD_ICEBERG_LIB LIB_NAME) PUBLIC "$") endif() -install(TARGET

Re: [I] [BUG] pyiceberg hanging on multiprocessing [iceberg-python]

2025-01-06 Thread via GitHub
frankliee commented on issue #1488: URL: https://github.com/apache/iceberg-python/issues/1488#issuecomment-2574263503 @kevinjqliu I use pystack to get stack of child process, it shows that `pyarrow.FileSystem` causes the hanging. By the way, our env is not easy to upgrade pyiceberg t

Re: [PR] Add iceberg_arrow library [iceberg-cpp]

2025-01-06 Thread via GitHub
wgtmac commented on code in PR #6: URL: https://github.com/apache/iceberg-cpp/pull/6#discussion_r1904830435 ## cmake_modules/BuildUtils.cmake: ## @@ -201,17 +202,26 @@ function(ADD_ICEBERG_LIB LIB_NAME) PUBLIC "$") endif() -install(TARGET

Re: [PR] Add iceberg_arrow library [iceberg-cpp]

2025-01-06 Thread via GitHub
kou commented on code in PR #6: URL: https://github.com/apache/iceberg-cpp/pull/6#discussion_r1904828898 ## cmake_modules/BuildUtils.cmake: ## @@ -201,17 +202,26 @@ function(ADD_ICEBERG_LIB LIB_NAME) PUBLIC "$") endif() -install(TARGETS $

Re: [PR] Add iceberg_arrow library [iceberg-cpp]

2025-01-06 Thread via GitHub
kou commented on code in PR #6: URL: https://github.com/apache/iceberg-cpp/pull/6#discussion_r1904828726 ## cmake_modules/BuildUtils.cmake: ## @@ -201,17 +202,26 @@ function(ADD_ICEBERG_LIB LIB_NAME) PUBLIC "$") endif() -install(TARGETS $

Re: [PR] Add iceberg_arrow library [iceberg-cpp]

2025-01-06 Thread via GitHub
kou commented on code in PR #6: URL: https://github.com/apache/iceberg-cpp/pull/6#discussion_r1904829176 ## cmake_modules/BuildUtils.cmake: ## @@ -201,17 +202,26 @@ function(ADD_ICEBERG_LIB LIB_NAME) PUBLIC "$") endif() -install(TARGETS $

[PR] feat(catalog): Add Catalog Registry [iceberg-go]

2025-01-06 Thread via GitHub
zeroshade opened a new pull request, #244: URL: https://github.com/apache/iceberg-go/pull/244 Allow flexibility for handling catalogs and catalog implementations by adding a Catalog Registry to make it easy to add custom catalog implementations. The Registry allows registering an iden

[I] [Bug] Error in overwrite(): pyarrow.lib.ArrowInvalid: offset overflow with large dataset (~3M rows, ~1GB) [iceberg-python]

2025-01-06 Thread via GitHub
sundaresanr opened a new issue, #1491: URL: https://github.com/apache/iceberg-python/issues/1491 ### Apache Iceberg version 0.8.1 (latest release) ### Please describe the bug 🐞 Encountered the following error while calling overwrite() on a dataset with over 3 million row

Re: [I] Writing an arrow table with date64 unsupported [iceberg-python]

2025-01-06 Thread via GitHub
github-actions[bot] closed issue #830: Writing an arrow table with date64 unsupported URL: https://github.com/apache/iceberg-python/issues/830 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [I] Writing an arrow table with date64 unsupported [iceberg-python]

2025-01-06 Thread via GitHub
github-actions[bot] commented on issue #830: URL: https://github.com/apache/iceberg-python/issues/830#issuecomment-2574156186 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apac

Re: [I] Implement Human OAuth2 Flows for OAuth2Manager [iceberg]

2025-01-06 Thread via GitHub
github-actions[bot] commented on issue #10677: URL: https://github.com/apache/iceberg/issues/10677#issuecomment-2574157406 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

2025-01-06 Thread via GitHub
guykhazma commented on PR #11615: URL: https://github.com/apache/iceberg/pull/11615#issuecomment-2574163828 @huaxingao @RussellSpitzer friendly remainder, can you please review this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Add iceberg_arrow library [iceberg-cpp]

2025-01-06 Thread via GitHub
kou commented on code in PR #6: URL: https://github.com/apache/iceberg-cpp/pull/6#discussion_r1904773431 ## cmake_modules/BuildUtils.cmake: ## @@ -201,17 +202,26 @@ function(ADD_ICEBERG_LIB LIB_NAME) PUBLIC "$") endif() -install(TARGETS $

Re: [I] [Bug] Cannot use PyIceberg with multiple FS [iceberg-python]

2025-01-06 Thread via GitHub
jiakai-li commented on issue #1041: URL: https://github.com/apache/iceberg-python/issues/1041#issuecomment-2574308740 @kevinjqliu I guess we can close this issue now? At the meantime, I'm keen to work on the `write.data.path` and `write.metadata.path` if that's something we want to enable

Re: [PR] bump version to 0.8.1 [iceberg-python]

2025-01-06 Thread via GitHub
Fokko commented on PR #1489: URL: https://github.com/apache/iceberg-python/pull/1489#issuecomment-2573620664 Thanks for addressing this. We bumped the version on the 0.8.x branch. How about moving this to 0.9.0 right away since that would be the next version released from the main branch.

Re: [PR] Change dot notation in add column documentation to tuple [iceberg-python]

2025-01-06 Thread via GitHub
kevinjqliu commented on code in PR #1433: URL: https://github.com/apache/iceberg-python/pull/1433#discussion_r1904467272 ## mkdocs/docs/api.md: ## @@ -951,8 +951,10 @@ Using `add_column` you can add a column, without having to worry about the field with table.update_schema() a

[PR] Bump moto from 5.0.25 to 5.0.26 [iceberg-python]

2025-01-06 Thread via GitHub
dependabot[bot] opened a new pull request, #1490: URL: https://github.com/apache/iceberg-python/pull/1490 Bumps [moto](https://github.com/getmoto/moto) from 5.0.25 to 5.0.26. Changelog Sourced from https://github.com/getmoto/moto/blob/master/CHANGELOG.md";>moto's changelog.

Re: [PR] Remove unneeded metadata read during update event generation [iceberg]

2025-01-06 Thread via GitHub
grantatspothero commented on code in PR #11829: URL: https://github.com/apache/iceberg/pull/11829#discussion_r1904687147 ## core/src/test/java/org/apache/iceberg/TestTables.java: ## @@ -255,8 +257,13 @@ void failCommits(int numFailures) { this.failCommits = numFailures;

Re: [PR] Remove unneeded metadata read during update event generation [iceberg]

2025-01-06 Thread via GitHub
grantatspothero commented on code in PR #11829: URL: https://github.com/apache/iceberg/pull/11829#discussion_r1904687147 ## core/src/test/java/org/apache/iceberg/TestTables.java: ## @@ -255,8 +257,13 @@ void failCommits(int numFailures) { this.failCommits = numFailures;

Re: [PR] Remove unneeded metadata read during update event generation [iceberg]

2025-01-06 Thread via GitHub
grantatspothero commented on code in PR #11829: URL: https://github.com/apache/iceberg/pull/11829#discussion_r1904696125 ## core/src/test/java/org/apache/iceberg/TestTables.java: ## @@ -255,8 +257,13 @@ void failCommits(int numFailures) { this.failCommits = numFailures;

Re: [I] java.lang.IllegalArgumentException: requirement failed: length (-6235972) cannot be smaller than -1 [iceberg]

2025-01-06 Thread via GitHub
justdoitvimal commented on issue #9689: URL: https://github.com/apache/iceberg/issues/9689#issuecomment-2573677957 Do we final a conclusion as in what will exactly help in fixing this issue? I also ran into this issue after using the maintenance job for few months. I don't don't know the ex

Re: [PR] Use SupportsPrefixOperations for Remove OrphanFile Procedure [iceberg]

2025-01-06 Thread via GitHub
ismailsimsek commented on code in PR #11906: URL: https://github.com/apache/iceberg/pull/11906#discussion_r1903140118 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java: ## @@ -292,14 +293,37 @@ private Dataset validFileIdentDS()

Re: [PR] Remove unneeded metadata read during update event generation [iceberg]

2025-01-06 Thread via GitHub
grantatspothero commented on code in PR #11829: URL: https://github.com/apache/iceberg/pull/11829#discussion_r1904580753 ## core/src/test/java/org/apache/iceberg/util/TestReachableFileUtil.java: ## @@ -114,7 +114,10 @@ public void testMetadataFileLocationsWithMissingFiles() {

Re: [PR] Open-API: Fix compilation errors in generated Java classes due to mismatched return types [iceberg]

2025-01-06 Thread via GitHub
VladimirYushkevich commented on code in PR #11806: URL: https://github.com/apache/iceberg/pull/11806#discussion_r1904463285 ## open-api/rest-catalog-open-api.py: ## @@ -981,8 +966,33 @@ class ValueMap(BaseModel): ) +class ContentFile(BaseModel): +content: ContentEnu

Re: [I] Variant Data Type Support [iceberg]

2025-01-06 Thread via GitHub
aihuaxu commented on issue #10392: URL: https://github.com/apache/iceberg/issues/10392#issuecomment-2573670168 > @aihuaxu Will this also work for protobuf encoded columns? I have a dataset with event_bytes | event_name > > 101010100 | e1 101010100 | e2 That shouldn't work. Vari

Re: [PR] Spec: Support geo type [iceberg]

2025-01-06 Thread via GitHub
paleolimbot commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1904570442 ## format/spec.md: ## @@ -584,8 +589,8 @@ The schema of a manifest file is a struct called `manifest_entry` with the follo | _optional_ | _optional_ | _optional

Re: [PR] Core: add variant type support [iceberg]

2025-01-06 Thread via GitHub
aihuaxu commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1904474514 ## core/src/test/java/org/apache/iceberg/TestMetadataUpdateParser.java: ## @@ -52,6 +56,15 @@ public class TestMetadataUpdateParser { Types.NestedField.req

Re: [I] TypeError when `operation` field is missing in `summary`. [iceberg-python]

2025-01-06 Thread via GitHub
kevinjqliu commented on issue #1106: URL: https://github.com/apache/iceberg-python/issues/1106#issuecomment-2573699614 resolved by #1263 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [I] TypeError when `operation` field is missing in `summary`. [iceberg-python]

2025-01-06 Thread via GitHub
kevinjqliu closed issue #1106: TypeError when `operation` field is missing in `summary`. URL: https://github.com/apache/iceberg-python/issues/1106 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Use ExternalTypeInfo in Rowconverter code instead of deprecated TableSchema.getFieldTypes [iceberg]

2025-01-06 Thread via GitHub
stevenzwu merged PR #11838: URL: https://github.com/apache/iceberg/pull/11838 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] backport #11301(rowconverter) to Flink 1.19 and 1.18 [iceberg]

2025-01-06 Thread via GitHub
stevenzwu commented on PR #11826: URL: https://github.com/apache/iceberg/pull/11826#issuecomment-2573964980 I don't understand why this back port depends on PR #11838 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] Use compatible column name to set Parquet bloom filter [iceberg]

2025-01-06 Thread via GitHub
huaxingao commented on PR #11799: URL: https://github.com/apache/iceberg/pull/11799#issuecomment-2573977829 @RussellSpitzer > Looks like tests are not passing? I looked at the failed test again. The reason it failed is that the bloom filter is set on a field of the struct typ

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-01-06 Thread via GitHub
kevinjqliu commented on code in PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#discussion_r190406 ## tests/catalog/test_s3tables.py: ## @@ -0,0 +1,227 @@ +import pytest Review Comment: thats a good point, we have integration tests marked for `gcs` htt

Re: [PR] Hive: Optimize viewExists API in hive catalog [iceberg]

2025-01-06 Thread via GitHub
dramaticlly commented on code in PR #11813: URL: https://github.com/apache/iceberg/pull/11813#discussion_r1904501804 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCatalog.java: ## @@ -125,6 +125,51 @@ public void testHiveViewAndIcebergViewWithSameName() th

Re: [PR] Core: add variant builder implementation [iceberg]

2025-01-06 Thread via GitHub
aihuaxu commented on PR #11857: URL: https://github.com/apache/iceberg/pull/11857#issuecomment-2574053859 @rdblue, @RussellSpitzer Please help review. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-01-06 Thread via GitHub
HonahX commented on code in PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#discussion_r1904584229 ## pyiceberg/catalog/s3tables.py: ## @@ -0,0 +1,324 @@ +import re +from typing import TYPE_CHECKING, List, Optional, Set, Tuple, Union + +import boto3 + +from pyi

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-01-06 Thread via GitHub
HonahX commented on code in PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#discussion_r1904875003 ## pyiceberg/catalog/s3tables.py: ## @@ -0,0 +1,324 @@ +import re +from typing import TYPE_CHECKING, List, Optional, Set, Tuple, Union + +import boto3 + +from pyi

[I] Question about Decimal Type Limitations in Iceberg [iceberg]

2025-01-06 Thread via GitHub
rice668 opened a new issue, #11920: URL: https://github.com/apache/iceberg/issues/11920 ### Query engine Spark ### Question Hello Iceberg Community, https://github.com/user-attachments/assets/573a1a35-a3da-45a2-bb7e-10ec378dfd4c"; /> I would like to ask why

Re: [PR] Kafka Connect: Add table to topics mapping property [iceberg]

2025-01-06 Thread via GitHub
yornstei commented on PR #10422: URL: https://github.com/apache/iceberg/pull/10422#issuecomment-2574366593 This solution seems best to me; most explicit and least restrictive. I'll be using this in my fork. thanks @igorvoltaic. I also agree this isn't related to #11313 which is for dynam

Re: [PR] Kafka Connect: Add the configuration option to provide a transactional id prefix to use [iceberg]

2025-01-06 Thread via GitHub
bryanck commented on PR #11780: URL: https://github.com/apache/iceberg/pull/11780#issuecomment-2574371981 I'll take a look this week, thanks for your patience! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Kafka Connect: Add table to topics mapping property [iceberg]

2025-01-06 Thread via GitHub
bryanck commented on PR #10422: URL: https://github.com/apache/iceberg/pull/10422#issuecomment-2574373145 Thanks @yornstei, good to know you found this useful. I was wondering if you had an opinion on https://github.com/apache/iceberg/pull/11623? -- This is an automated message from the A

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-01-06 Thread via GitHub
HonahX commented on code in PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#discussion_r1904713991 ## pyiceberg/catalog/s3tables.py: ## @@ -0,0 +1,324 @@ +import re +from typing import TYPE_CHECKING, List, Optional, Set, Tuple, Union + +import boto3 + +from pyi

Re: [PR] Kafka Connect: Add table to topics mapping property [iceberg]

2025-01-06 Thread via GitHub
yornstei commented on PR #10422: URL: https://github.com/apache/iceberg/pull/10422#issuecomment-2574383989 > Thanks @yornstei, good to know you found this useful. I was wondering if you had an opinion on #11623? I took a look at that one too. From a configurable perspective, it seems

Re: [I] Update a TableSchema from a Schema [iceberg-rust]

2025-01-06 Thread via GitHub
Lordworms commented on issue #698: URL: https://github.com/apache/iceberg-rust/issues/698#issuecomment-2574388365 I would like to do this one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Use Snapshot's statistics file in SparkScan [iceberg]

2025-01-06 Thread via GitHub
jeesou commented on PR #11040: URL: https://github.com/apache/iceberg/pull/11040#issuecomment-2574390763 Hi @karuppayya @amogh-jahagirdar friendly reminder, please check the comments once. -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] Add iceberg_arrow library [iceberg-cpp]

2025-01-06 Thread via GitHub
wgtmac commented on code in PR #6: URL: https://github.com/apache/iceberg-cpp/pull/6#discussion_r1904261495 ## cmake_modules/BuildUtils.cmake: ## @@ -201,17 +202,26 @@ function(ADD_ICEBERG_LIB LIB_NAME) PUBLIC "$") endif() -install(TARGET

Re: [I] Support virtual addressing style in PyArrowFileIO [iceberg-python]

2025-01-06 Thread via GitHub
kevinjqliu closed issue #21: Support virtual addressing style in PyArrowFileIO URL: https://github.com/apache/iceberg-python/issues/21 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [I] Support virtual addressing style in PyArrowFileIO [iceberg-python]

2025-01-06 Thread via GitHub
kevinjqliu commented on issue #21: URL: https://github.com/apache/iceberg-python/issues/21#issuecomment-2573312668 closed by https://github.com/apache/iceberg-python/pull/1392/files#diff-8d5e63f2a87ead8cebe2fd8ac5dcf2198d229f01e16bb9e06e21f7277c328abdR377-R378 -- This is an automated mess

[PR] Spark 3.5: Implement RewriteTablePath [iceberg]

2025-01-06 Thread via GitHub
szehon-ho opened a new pull request, #11555: URL: https://github.com/apache/iceberg/pull/11555 This is the implementation for #10920 (an action to prepare metadata for an Iceberg table for DR copy) This has been used in production for awhile in our setup, although support for rewrite

Re: [PR] Spark 3.5: Implement RewriteTablePath [iceberg]

2025-01-06 Thread via GitHub
szehon-ho closed pull request #11555: Spark 3.5: Implement RewriteTablePath URL: https://github.com/apache/iceberg/pull/11555 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Change dot notation in add column documentation to tuple [iceberg-python]

2025-01-06 Thread via GitHub
kevinjqliu commented on PR #1433: URL: https://github.com/apache/iceberg-python/pull/1433#issuecomment-2573362375 im having trouble running the new statements in the docs, could you give it a try ? -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [PR] Change dot notation in add column documentation to tuple [iceberg-python]

2025-01-06 Thread via GitHub
jeppe-dos commented on PR #1433: URL: https://github.com/apache/iceberg-python/pull/1433#issuecomment-2573564249 > im having trouble running the new statements in the docs, could you give it a try ? The code doesn't work, as "confirmed_by" has been changed to "exchange". Exchange can

Re: [PR] Hive: Add Hive 4 support and remove Hive runtime [iceberg]

2025-01-06 Thread via GitHub
pvary commented on PR #11750: URL: https://github.com/apache/iceberg/pull/11750#issuecomment-2572593083 > Spark (and other modules) also depend on `TestHiveMetastore` from test modules of `iceberg-hive-metastore`. We cannot use old Hive dependency from Spark to run this class due to API cha

Re: [PR] Backport #11557 to FLink1.19 and 1.18 [iceberg]

2025-01-06 Thread via GitHub
pvary merged PR #11834: URL: https://github.com/apache/iceberg/pull/11834 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Backport #11557 to FLink1.19 and 1.18 [iceberg]

2025-01-06 Thread via GitHub
pvary commented on PR #11834: URL: https://github.com/apache/iceberg/pull/11834#issuecomment-2572785020 Merged to main. Thanks for the backport @huyuanfeng2018! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Hive: Optimize viewExists API in hive catalog [iceberg]

2025-01-06 Thread via GitHub
pvary commented on code in PR #11813: URL: https://github.com/apache/iceberg/pull/11813#discussion_r1903964817 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCatalog.java: ## @@ -125,6 +125,51 @@ public void testHiveViewAndIcebergViewWithSameName() throws T

Re: [PR] Hive: Optimize viewExists API in hive catalog [iceberg]

2025-01-06 Thread via GitHub
pvary commented on code in PR #11813: URL: https://github.com/apache/iceberg/pull/11813#discussion_r1903966574 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCatalog.java: ## @@ -125,6 +125,51 @@ public void testHiveViewAndIcebergViewWithSameName() throws T

Re: [PR] Hive: Optimize viewExists API in hive catalog [iceberg]

2025-01-06 Thread via GitHub
pvary commented on code in PR #11813: URL: https://github.com/apache/iceberg/pull/11813#discussion_r1903970848 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCatalog.java: ## @@ -125,6 +125,51 @@ public void testHiveViewAndIcebergViewWithSameName() throws T

Re: [PR] Count rows as a metadata only operation [iceberg-python]

2025-01-06 Thread via GitHub
tusharchou commented on PR #1388: URL: https://github.com/apache/iceberg-python/pull/1388#issuecomment-2572804832 Hi @Fokko @kevinjqliu @gli-chris-hao , I have implemented these suggestions with my best understanding. - [x] residual evaluator - [x] positional deletes - [x]

[I] Add ability to pickle a `Table` [iceberg-python]

2025-01-06 Thread via GitHub
Fokko opened a new issue, #513: URL: https://github.com/apache/iceberg-python/issues/513 ### Feature Request / Improvement This allows distribution of the Table object within Ray. -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [PR] Change dot notation in add column documentation to tuple [iceberg-python]

2025-01-06 Thread via GitHub
jeppe-dos commented on PR #1433: URL: https://github.com/apache/iceberg-python/pull/1433#issuecomment-2573318869 What if you create the struct first, and then add the nested field like so: ``` with table.update_schema() as update: update.add_column("retries", IntegerType(),

Re: [PR] Change dot notation in add column documentation to tuple [iceberg-python]

2025-01-06 Thread via GitHub
kevinjqliu commented on PR #1433: URL: https://github.com/apache/iceberg-python/pull/1433#issuecomment-2573325406 that works, but i think the first example should work too. We can track this in a separate issue. ``` >>> with table.update_schema() as update: ... update.a

Re: [PR] Spark 3.5: Implement RewriteTablePath [iceberg]

2025-01-06 Thread via GitHub
szehon-ho commented on code in PR #11555: URL: https://github.com/apache/iceberg/pull/11555#discussion_r1903293186 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java: ## @@ -0,0 +1,731 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-01-06 Thread via GitHub
felixscherz commented on code in PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#discussion_r1904397511 ## tests/catalog/test_s3tables.py: ## @@ -0,0 +1,173 @@ +import pytest + +from pyiceberg.catalog.s3tables import S3TableCatalog +from pyiceberg.exceptions im

Re: [PR] Spark 3.5: Implement RewriteTablePath [iceberg]

2025-01-06 Thread via GitHub
szehon-ho commented on code in PR #11555: URL: https://github.com/apache/iceberg/pull/11555#discussion_r1903293655 ## core/src/main/java/org/apache/iceberg/io/DeleteSchemaUtil.java: ## @@ -43,4 +43,15 @@ public static Schema pathPosSchema() { public static Schema posDeleteSch

Re: [PR] Spark 3.5: Implement RewriteTablePath [iceberg]

2025-01-06 Thread via GitHub
szehon-ho commented on code in PR #11555: URL: https://github.com/apache/iceberg/pull/11555#discussion_r1903293186 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java: ## @@ -0,0 +1,731 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] Spark 3.5: Implement RewriteTablePath [iceberg]

2025-01-06 Thread via GitHub
szehon-ho commented on code in PR #11555: URL: https://github.com/apache/iceberg/pull/11555#discussion_r1903293655 ## core/src/main/java/org/apache/iceberg/io/DeleteSchemaUtil.java: ## @@ -43,4 +43,15 @@ public static Schema pathPosSchema() { public static Schema posDeleteSch

Re: [PR] Spark 3.5: Implement RewriteTablePath [iceberg]

2025-01-06 Thread via GitHub
szehon-ho commented on code in PR #11555: URL: https://github.com/apache/iceberg/pull/11555#discussion_r1902646277 ## core/src/main/java/org/apache/iceberg/util/ContentFileUtil.java: ## @@ -60,28 +69,63 @@ public static CharSequence referencedDataFile(DeleteFile deleteFile) {

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-01-06 Thread via GitHub
felixscherz commented on code in PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#discussion_r1904401166 ## pyiceberg/catalog/s3tables.py: ## @@ -0,0 +1,322 @@ +import re +from typing import TYPE_CHECKING, List, Optional, Set, Tuple, Union + +import boto3 + +fro

Re: [PR] Fix read from multiple s3 regions [iceberg-python]

2025-01-06 Thread via GitHub
kevinjqliu commented on PR #1453: URL: https://github.com/apache/iceberg-python/pull/1453#issuecomment-2573265511 Thank you @jiakai-li for the contribution and @Fokko for the review :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Fix read from multiple s3 regions [iceberg-python]

2025-01-06 Thread via GitHub
kevinjqliu merged PR #1453: URL: https://github.com/apache/iceberg-python/pull/1453 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Change dot notation in add column documentation to tuple [iceberg-python]

2025-01-06 Thread via GitHub
kevinjqliu commented on code in PR #1433: URL: https://github.com/apache/iceberg-python/pull/1433#discussion_r1904280959 ## mkdocs/docs/api.md: ## @@ -961,20 +963,21 @@ Renaming a field in an Iceberg table is simple: ```python with table.update_schema() as update: update.

Re: [PR] Change dot notation in add column documentation to tuple [iceberg-python]

2025-01-06 Thread via GitHub
kevinjqliu commented on code in PR #1433: URL: https://github.com/apache/iceberg-python/pull/1433#discussion_r1904282046 ## mkdocs/docs/api.md: ## @@ -961,20 +963,21 @@ Renaming a field in an Iceberg table is simple: ```python with table.update_schema() as update: update.

Re: [PR] Change dot notation in add column documentation to tuple [iceberg-python]

2025-01-06 Thread via GitHub
kevinjqliu commented on code in PR #1433: URL: https://github.com/apache/iceberg-python/pull/1433#discussion_r1904280648 ## mkdocs/docs/api.md: ## @@ -1006,6 +1009,8 @@ Delete a field, careful this is a incompatible change (readers/writers might exp ```python with table.updat

Re: [PR] Change dot notation in add column documentation to tuple [iceberg-python]

2025-01-06 Thread via GitHub
jeppe-dos commented on PR #1433: URL: https://github.com/apache/iceberg-python/pull/1433#issuecomment-2573343676 > that works, but i think the first example should work too. We can track this in a separate issue. > > ``` > >>> with table.update_schema() as update: > ... upda

[PR] bump version to 0.8.1 [iceberg-python]

2025-01-06 Thread via GitHub
kevinjqliu opened a new pull request, #1489: URL: https://github.com/apache/iceberg-python/pull/1489 I noticed we never bumped the version in main branch to 0.8.1 Cherrypicked commit from the 0.8.x branch https://github.com/apache/iceberg-python/commit/a051584a3684392d2db6556449eb2

Re: [PR] Change dot notation in add column documentation to tuple [iceberg-python]

2025-01-06 Thread via GitHub
kevinjqliu commented on PR #1433: URL: https://github.com/apache/iceberg-python/pull/1433#issuecomment-2573349822 @jeppe-dos that would be great! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-01-06 Thread via GitHub
felixscherz commented on code in PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#discussion_r1904369776 ## pyiceberg/catalog/s3tables.py: ## @@ -0,0 +1,322 @@ +import re +from typing import TYPE_CHECKING, List, Optional, Set, Tuple, Union + +import boto3 + +fro

Re: [PR] feat(datafusion): Expose DataFusion statistics on an IcebergTableScan [iceberg-rust]

2025-01-06 Thread via GitHub
gruuya commented on code in PR #880: URL: https://github.com/apache/iceberg-rust/pull/880#discussion_r1904197220 ## crates/integrations/datafusion/src/table/mod.rs: ## @@ -130,10 +131,14 @@ impl TableProvider for IcebergTableProvider { filters: &[Expr], _limit:

Re: [PR] API: Support removeUnusedSpecs in ExpireSnapshots [iceberg]

2025-01-06 Thread via GitHub
advancedxy commented on PR #10755: URL: https://github.com/apache/iceberg/pull/10755#issuecomment-2573229194 @rdblue @danielcweeks @amogh-jahagirdar would you mind to take a look at this again? -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [I] [BUG] pyiceberg hanging on multiprocessing [iceberg-python]

2025-01-06 Thread via GitHub
kevinjqliu commented on issue #1488: URL: https://github.com/apache/iceberg-python/issues/1488#issuecomment-2573279503 Hi @frankliee thanks for reporting this issue. I noticed you're using version 0.7.1, the latest version is 0.8.1. Could you retry with the latest version? The issue

Re: [PR] Core: Unimplement Map from CharSequenceMap to obey contract [iceberg]

2025-01-06 Thread via GitHub
findepi commented on PR #11704: URL: https://github.com/apache/iceberg/pull/11704#issuecomment-2573411391 CharSequenceMap is not a sound Map implementation, it would be nice if someone could spend couple minutes and review this PR. -- This is an automated message from the Apache Git Servi

Re: [PR] Spark 3.5: Implement RewriteTablePath [iceberg]

2025-01-06 Thread via GitHub
flyrain commented on code in PR #11555: URL: https://github.com/apache/iceberg/pull/11555#discussion_r1904432648 ## core/src/main/java/org/apache/iceberg/io/DeleteSchemaUtil.java: ## @@ -43,4 +43,15 @@ public static Schema pathPosSchema() { public static Schema posDeleteSchem

Re: [PR] API: Support removeUnusedSpecs in ExpireSnapshots [iceberg]

2025-01-06 Thread via GitHub
amogh-jahagirdar commented on code in PR #10755: URL: https://github.com/apache/iceberg/pull/10755#discussion_r1904445943 ## core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java: ## @@ -1284,6 +1284,62 @@ public void testUpdateTableSpecThenRevert() { assertThat(t

Re: [PR] Open-API: Fix compilation errors in generated Java classes due to mismatched return types [iceberg]

2025-01-06 Thread via GitHub
VladimirYushkevich commented on code in PR #11806: URL: https://github.com/apache/iceberg/pull/11806#discussion_r1904456125 ## open-api/rest-catalog-open-api.py: ## @@ -981,8 +966,33 @@ class ValueMap(BaseModel): ) +class ContentFile(BaseModel): Review Comment: this

Re: [PR] Update maintenance.md [iceberg]

2025-01-06 Thread via GitHub
nastra commented on code in PR #11916: URL: https://github.com/apache/iceberg/pull/11916#discussion_r1905022184 ## docs/docs/maintenance.md: ## @@ -137,7 +137,7 @@ See the [`RewriteDataFiles` Javadoc](../../javadoc/{{ icebergVersion }}/org/apac ### Rewrite manifests -Icebe

Re: [PR] Update maintenance.md [iceberg]

2025-01-06 Thread via GitHub
nastra merged PR #11916: URL: https://github.com/apache/iceberg/pull/11916 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.29.43 to 2.29.45 [iceberg]

2025-01-06 Thread via GitHub
nastra merged PR #11910: URL: https://github.com/apache/iceberg/pull/11910 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Build: Bump org.xerial:sqlite-jdbc from 3.47.1.0 to 3.47.2.0 [iceberg]

2025-01-06 Thread via GitHub
nastra merged PR #11907: URL: https://github.com/apache/iceberg/pull/11907 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Build: Bump org.assertj:assertj-core from 3.27.0 to 3.27.2 [iceberg]

2025-01-06 Thread via GitHub
nastra commented on PR #11908: URL: https://github.com/apache/iceberg/pull/11908#issuecomment-2574587110 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] feat(puffin): Parse Puffin FileMetadata [iceberg-rust]

2025-01-06 Thread via GitHub
liurenjie1024 commented on code in PR #765: URL: https://github.com/apache/iceberg-rust/pull/765#discussion_r1903911917 ## crates/iceberg/src/puffin/metadata.rs: ## @@ -0,0 +1,777 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license a

Re: [PR] GCP: Add Iceberg Catalog for GCP BigQuery Metastore [iceberg]

2025-01-06 Thread via GitHub
vrishin-bolt commented on PR #11039: URL: https://github.com/apache/iceberg/pull/11039#issuecomment-2574498242 > @hesham-medhat @rdblue - could you pls give an update on this PR? It seems it would massively simplify the Iceberg table management in GCP for non-spark usecases. > > than

Re: [PR] Split metadata tables into separate modules [iceberg-rust]

2025-01-06 Thread via GitHub
liurenjie1024 commented on code in PR #872: URL: https://github.com/apache/iceberg-rust/pull/872#discussion_r1904969065 ## crates/iceberg/src/inspect/metadata_table.rs: ## @@ -0,0 +1,99 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lic

Re: [PR] Split metadata tables into separate modules [iceberg-rust]

2025-01-06 Thread via GitHub
liurenjie1024 commented on code in PR #872: URL: https://github.com/apache/iceberg-rust/pull/872#discussion_r1904966540 ## crates/iceberg/src/inspect/snapshots.rs: ## @@ -0,0 +1,183 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license

[PR] API: Align CharSequenceSet impl with Data/DeleteFileSet [iceberg]

2025-01-06 Thread via GitHub
nastra opened a new pull request, #11322: URL: https://github.com/apache/iceberg/pull/11322 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

Re: [PR] Metadata table scans as streams [iceberg-rust]

2025-01-06 Thread via GitHub
liurenjie1024 commented on PR #870: URL: https://github.com/apache/iceberg-rust/pull/870#issuecomment-2574495817 Thanks @rshkv for this pr, exactly what I mean! We should rebase this pr after #872 ? -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-01-06 Thread via GitHub
felixscherz commented on code in PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#discussion_r1905038020 ## pyiceberg/catalog/s3tables.py: ## @@ -0,0 +1,324 @@ +import re +from typing import TYPE_CHECKING, List, Optional, Set, Tuple, Union + +import boto3 + +fro

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-01-06 Thread via GitHub
felixscherz commented on code in PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#discussion_r1905038693 ## pyiceberg/catalog/s3tables.py: ## @@ -0,0 +1,324 @@ +import re +from typing import TYPE_CHECKING, List, Optional, Set, Tuple, Union + +import boto3 + +fro

Re: [PR] feat: support S3 Table Buckets with S3TablesCatalog [iceberg-python]

2025-01-06 Thread via GitHub
HonahX commented on code in PR #1429: URL: https://github.com/apache/iceberg-python/pull/1429#discussion_r1904909293 ## pyiceberg/catalog/s3tables.py: ## @@ -0,0 +1,324 @@ +import re +from typing import TYPE_CHECKING, List, Optional, Set, Tuple, Union + +import boto3 + +from pyi

Re: [PR] Change dot notation in add column documentation to tuple [iceberg-python]

2025-01-06 Thread via GitHub
jeppe-dos commented on PR #1433: URL: https://github.com/apache/iceberg-python/pull/1433#issuecomment-2572871762 The struct is now created first in the add column section. I have also changed from dot to tuple in move and rename column. -- This is an automated message from the Apache Git

  1   2   >