Re: [PR] CORE: return false when view exists endpoint isn't supported [iceberg]

2025-02-13 Thread via GitHub
nastra commented on code in PR #12259: URL: https://github.com/apache/iceberg/pull/12259#discussion_r1955661232 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -1239,7 +1239,9 @@ public List listViews(SessionContext context, Namespace namespa @

Re: [PR] S3: Disable strong integrity checksums [iceberg]

2025-02-13 Thread via GitHub
ebyhr commented on code in PR #12264: URL: https://github.com/apache/iceberg/pull/12264#discussion_r1955642546 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3RequestUtil.java: ## @@ -149,4 +151,10 @@ static void configurePermission( Function aclSetter) { aclSetter.

Re: [PR] S3: Disable strong integrity checksums [iceberg]

2025-02-13 Thread via GitHub
ebyhr commented on code in PR #12264: URL: https://github.com/apache/iceberg/pull/12264#discussion_r1955642546 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3RequestUtil.java: ## @@ -149,4 +151,10 @@ static void configurePermission( Function aclSetter) { aclSetter.

[I] Link LEARN MORE vom https://iceberg.apache.org/about/ runs into Not Found [iceberg]

2025-02-13 Thread via GitHub
alexandertodorovic opened a new issue, #12265: URL: https://github.com/apache/iceberg/issues/12265 Dear All, The LEARN MORE button on the [Apache Iceberg About](https://iceberg.apache.org/about/) page leads to a "Not Found" error: _Not Found The requested URL was not found o

Re: [PR] backport #11301(rowconverter) to Flink 1.19 and 1.18 [iceberg]

2025-02-13 Thread via GitHub
pvary commented on PR #11826: URL: https://github.com/apache/iceberg/pull/11826#issuecomment-2658364228 What is the state of this backport @abharath9? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] Spark streaming (merge into) iceberg table concurrent write with compaction job [iceberg]

2025-02-13 Thread via GitHub
singhpk234 commented on issue #12187: URL: https://github.com/apache/iceberg/issues/12187#issuecomment-2658363561 We should look into the combinations of A2 (remove old snapshot, rewrite manifest , compaction (binpack and sometimes z-order), rewrite position delete files, delete orphan file

Re: [PR] Implementation of version metadata table for view [iceberg]

2025-02-13 Thread via GitHub
huan233usc commented on PR #12014: URL: https://github.com/apache/iceberg/pull/12014#issuecomment-2658363474 Thank you both for the review, will update this pr accordingly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] S3: Disable strong integrity checksums [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar commented on code in PR #12264: URL: https://github.com/apache/iceberg/pull/12264#discussion_r1955600722 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3RequestUtil.java: ## @@ -149,4 +151,10 @@ static void configurePermission( Function aclSetter) {

Re: [PR] Spark: Structured Streaming read limit support follow-up [iceberg]

2025-02-13 Thread via GitHub
singhpk234 commented on code in PR #12260: URL: https://github.com/apache/iceberg/pull/12260#discussion_r1955584774 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java: ## @@ -309,6 +312,49 @@ private static StreamingOffset determineStar

Re: [PR] S3: Disable strong integrity checksums [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar commented on code in PR #12264: URL: https://github.com/apache/iceberg/pull/12264#discussion_r1955595821 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3RequestUtil.java: ## @@ -149,4 +151,10 @@ static void configurePermission( Function aclSetter) {

Re: [PR] S3: Disable strong integrity checksums [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar commented on code in PR #12264: URL: https://github.com/apache/iceberg/pull/12264#discussion_r1955595821 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3RequestUtil.java: ## @@ -149,4 +151,10 @@ static void configurePermission( Function aclSetter) {

Re: [PR] S3: Disable strong integrity checksums [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar commented on code in PR #12264: URL: https://github.com/apache/iceberg/pull/12264#discussion_r1955593603 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3RequestUtil.java: ## @@ -149,4 +151,10 @@ static void configurePermission( Function aclSetter) {

Re: [PR] Spark: Rewrite V2 deletes to V3 DVs [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar commented on code in PR #12250: URL: https://github.com/apache/iceberg/pull/12250#discussion_r1955585866 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeletesRewrite.java: ## @@ -213,45 +217,61 @@ static class PositionDeletesWrit

Re: [PR] Spark: Rewrite V2 deletes to V3 DVs [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar commented on code in PR #12250: URL: https://github.com/apache/iceberg/pull/12250#discussion_r1955585866 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeletesRewrite.java: ## @@ -213,45 +217,61 @@ static class PositionDeletesWrit

Re: [PR] Core: add variant type support [iceberg]

2025-02-13 Thread via GitHub
aihuaxu commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r192361 ## core/src/test/java/org/apache/iceberg/avro/TestSchemaConversions.java: ## @@ -370,4 +370,22 @@ public void testFieldDocsArePreserved() { Lists.newArrayLis

Re: [PR] Spark: Remove unused PruneColumnsWithReordering class [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar merged PR #12258: URL: https://github.com/apache/iceberg/pull/12258 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Spark: Remove unused PruneColumnsWithReordering class [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar commented on PR #12258: URL: https://github.com/apache/iceberg/pull/12258#issuecomment-2658231400 I'll go ahead and merge, thanks @Fokko @dramaticlly for reviewing! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [PR] add Status data structure [iceberg-cpp]

2025-02-13 Thread via GitHub
zhjwpku commented on PR #8: URL: https://github.com/apache/iceberg-cpp/pull/8#issuecomment-2658229119 This is obsoleted by #40, though we haven't decided to adopt expected or not, this PR can be closed. Thanks for all the inputs. -- This is an automated message from the Apache Git Service

Re: [PR] add Status data structure [iceberg-cpp]

2025-02-13 Thread via GitHub
zhjwpku closed pull request #8: add Status data structure URL: https://github.com/apache/iceberg-cpp/pull/8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-ma

[PR] Docker: Pin QEMU version temporarily [iceberg]

2025-02-13 Thread via GitHub
ebyhr opened a new pull request, #12262: URL: https://github.com/apache/iceberg/pull/12262 Fix failing job: https://github.com/apache/iceberg/actions/workflows/publish-iceberg-rest-fixture-docker.yml -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [PR] feat: support merge append action [iceberg-rust]

2025-02-13 Thread via GitHub
jonathanc-n commented on code in PR #902: URL: https://github.com/apache/iceberg-rust/pull/902#discussion_r1955391542 ## crates/integration_tests/tests/merge_append_test.rs: ## @@ -0,0 +1,183 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contribut

Re: [PR] Spark: Structured Streaming read limit support follow-up [iceberg]

2025-02-13 Thread via GitHub
wypoon commented on PR #12260: URL: https://github.com/apache/iceberg/pull/12260#issuecomment-2658021202 @singhpk234 @jackye1995 @RussellSpitzer this is a small fix; can you please review? -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] Spark: Structured Streaming read limit support follow-up [iceberg]

2025-02-13 Thread via GitHub
wypoon commented on code in PR #12260: URL: https://github.com/apache/iceberg/pull/12260#discussion_r1955407420 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreamingRead3.java: ## @@ -206,6 +220,18 @@ public void testReadStreamOnIcebergTableW

Re: [PR] Spark: Structured Streaming read limit support follow-up [iceberg]

2025-02-13 Thread via GitHub
wypoon commented on code in PR #12260: URL: https://github.com/apache/iceberg/pull/12260#discussion_r1955405894 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreamingRead3.java: ## @@ -154,8 +154,7 @@ public void testReadStreamOnIcebergTableWi

Re: [PR] Add support for `write.metadata.path` [iceberg-python]

2025-02-13 Thread via GitHub
kevinjqliu commented on code in PR #1642: URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1955405522 ## mkdocs/docs/configuration.md: ## @@ -203,12 +204,15 @@ PyIceberg uses [S3FileSystem](https://arrow.apache.org/docs/python/generated/pya ## Location Pro

Re: [PR] Add support for `write.metadata.path` [iceberg-python]

2025-02-13 Thread via GitHub
kevinjqliu commented on code in PR #1642: URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1955405522 ## mkdocs/docs/configuration.md: ## @@ -203,12 +204,15 @@ PyIceberg uses [S3FileSystem](https://arrow.apache.org/docs/python/generated/pya ## Location Pro

Re: [PR] Spark: Structured Streaming read limit support follow-up [iceberg]

2025-02-13 Thread via GitHub
wypoon commented on code in PR #12260: URL: https://github.com/apache/iceberg/pull/12260#discussion_r1955404930 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java: ## @@ -458,7 +502,7 @@ public ReadLimit getDefaultReadLimit() {

Re: [PR] Apply residuals when reading a table [iceberg-python]

2025-02-13 Thread via GitHub
corleyma commented on code in PR #1654: URL: https://github.com/apache/iceberg-python/pull/1654#discussion_r1955401755 ## pyiceberg/io/pyarrow.py: ## @@ -1342,9 +1342,8 @@ def _get_column_projection_values( def _task_to_record_batches( fs: FileSystem, task: FileScanTa

Re: [PR] Add support for `write.metadata.path` [iceberg-python]

2025-02-13 Thread via GitHub
kevinjqliu commented on code in PR #1642: URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1955395642 ## pyiceberg/table/locations.py: ## @@ -64,6 +71,35 @@ def new_data_location(self, data_file_name: str, partition_key: Optional[Partiti str: A f

Re: [PR] Add support for `write.metadata.path` [iceberg-python]

2025-02-13 Thread via GitHub
kevinjqliu commented on code in PR #1642: URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1955383770 ## pyiceberg/table/locations.py: ## @@ -64,6 +71,35 @@ def new_data_location(self, data_file_name: str, partition_key: Optional[Partiti str: A f

Re: [PR] Add support for `write.metadata.path` [iceberg-python]

2025-02-13 Thread via GitHub
kevinjqliu commented on code in PR #1642: URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1955380186 ## pyiceberg/table/locations.py: ## @@ -64,6 +71,35 @@ def new_data_location(self, data_file_name: str, partition_key: Optional[Partiti str: A f

Re: [PR] Hive: Add Hive 4 support and remove Hive runtime [iceberg]

2025-02-13 Thread via GitHub
github-actions[bot] commented on PR #11750: URL: https://github.com/apache/iceberg/pull/11750#issuecomment-2657973521 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [I] Docs: Add clear indicators for required fields in Spark syntax on CREATE TABLE. [iceberg]

2025-02-13 Thread via GitHub
github-actions[bot] commented on issue #9545: URL: https://github.com/apache/iceberg/issues/9545#issuecomment-265797 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Hive: Add Hive 4 support and remove Hive runtime [iceberg]

2025-02-13 Thread via GitHub
github-actions[bot] closed pull request #11750: Hive: Add Hive 4 support and remove Hive runtime URL: https://github.com/apache/iceberg/pull/11750 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Support bucket transform on multiple data columns [iceberg]

2025-02-13 Thread via GitHub
github-actions[bot] commented on issue #5626: URL: https://github.com/apache/iceberg/issues/5626#issuecomment-2657973302 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] backport #11301(rowconverter) to Flink 1.19 and 1.18 [iceberg]

2025-02-13 Thread via GitHub
github-actions[bot] closed pull request #11826: backport #11301(rowconverter) to Flink 1.19 and 1.18 URL: https://github.com/apache/iceberg/pull/11826 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] backport #11301(rowconverter) to Flink 1.19 and 1.18 [iceberg]

2025-02-13 Thread via GitHub
github-actions[bot] commented on PR #11826: URL: https://github.com/apache/iceberg/pull/11826#issuecomment-2657973561 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Flink: Maintenance - RewriteDataFiles [iceberg]

2025-02-13 Thread via GitHub
github-actions[bot] commented on PR #11497: URL: https://github.com/apache/iceberg/pull/11497#issuecomment-2657973451 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Core: Fix divide by zero when adjust split size [iceberg]

2025-02-13 Thread via GitHub
hantangwangd commented on PR #12201: URL: https://github.com/apache/iceberg/pull/12201#issuecomment-2657960091 @RussellSpitzer my pleasure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Add table upsert support [iceberg-python]

2025-02-13 Thread via GitHub
kevinjqliu commented on PR #1660: URL: https://github.com/apache/iceberg-python/pull/1660#issuecomment-2657959865 Thanks everyone for getting this over the finish line! Upsert has been a long awaited feature. I'm excited to include this as part of the upcoming 0.9.0 release. This is

Re: [PR] Manifest list encryption [iceberg]

2025-02-13 Thread via GitHub
gumartinm commented on PR #7770: URL: https://github.com/apache/iceberg/pull/7770#issuecomment-2657946677 If you need help with this pull request @ggershinsky, perhaps I could help you. -- This is an automated message from the Apache Git Service. To respond to the message, please

[PR] Core: use ReachableFileCleanup when table has discontinuous snapshots [iceberg]

2025-02-13 Thread via GitHub
MavsLee opened a new pull request, #12261: URL: https://github.com/apache/iceberg/pull/12261 This change is to fix fixes issue: #12200 . by updating `RemoveSnapshots.cleanExpiredSnapshots()` to use `ReachableFileCleanup` strategy to clean up expired files when table has discontinuous sn

[PR] Added description of CLI usage in README [iceberg-go]

2025-02-13 Thread via GitHub
subkanthi opened a new pull request, #301: URL: https://github.com/apache/iceberg-go/pull/301 Added description of CLI usage in README similar to pyiceberg. https://py.iceberg.apache.org/cli/ -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] feat(catalog/rest): Add support for view related operations [iceberg-go]

2025-02-13 Thread via GitHub
dttung2905 commented on code in PR #290: URL: https://github.com/apache/iceberg-go/pull/290#discussion_r195513 ## catalog/rest/rest.go: ## @@ -989,3 +992,95 @@ func (r *Catalog) CheckTableExists(ctx context.Context, identifier table.Identif } return true, ni

Re: [PR] Add table upsert support [iceberg-python]

2025-02-13 Thread via GitHub
mattmartin14 commented on PR #1660: URL: https://github.com/apache/iceberg-python/pull/1660#issuecomment-2657806614 > Great work @mattmartin14 πŸ‘πŸ» πŸ‘πŸ» > > If I'm not mistaken this is your first PR merged in any open source project correct? > > Not a bad first feature! Mine was ad

Re: [PR] Implement update for `remove-snapshot-ref` action [iceberg-python]

2025-02-13 Thread via GitHub
grihabor commented on PR #1598: URL: https://github.com/apache/iceberg-python/pull/1598#issuecomment-2657770025 @kevinjqliu @Fokko Shall we merge this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Core,Api: Add overwrite option when register external table to catalog [iceberg]

2025-02-13 Thread via GitHub
dramaticlly commented on code in PR #12228: URL: https://github.com/apache/iceberg/pull/12228#discussion_r1955046373 ## api/src/main/java/org/apache/iceberg/catalog/Catalog.java: ## @@ -344,7 +344,23 @@ default void invalidateTable(TableIdentifier identifier) {} * @throws Al

Re: [PR] CORE: return false when view exists endpoint isn't supported [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #12259: URL: https://github.com/apache/iceberg/pull/12259#discussion_r1955238348 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -1239,7 +1239,9 @@ public List listViews(SessionContext context, Namespace namespa @

Re: [PR] Add table upsert support [iceberg-python]

2025-02-13 Thread via GitHub
bitsondatadev commented on PR #1660: URL: https://github.com/apache/iceberg-python/pull/1660#issuecomment-2657742066 Great work @mattmartin14 πŸ‘πŸ» πŸ‘πŸ» If I'm not mistaken this is your first PR merged in any open source project correct? Not a bad first feature! Mine was adding [ar

Re: [I] set tblproperties, spark action expireSnapshots is not work. [iceberg]

2025-02-13 Thread via GitHub
RussellSpitzer commented on issue #12078: URL: https://github.com/apache/iceberg/issues/12078#issuecomment-2657726189 Ah so you are setting a Table Property in Hive. But then running Expire Snapshots in Spark but Spark isn't using the table properties that you set. Can you check the u

Re: [PR] Core: add variant type support [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1955196818 ## api/src/main/java/org/apache/iceberg/types/TypeUtil.java: ## @@ -712,6 +712,10 @@ public T map(Types.MapType map, Supplier keyResult, Supplier valueResult)

Re: [PR] Core: add variant type support [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1955206561 ## api/src/main/java/org/apache/iceberg/types/TypeUtil.java: ## @@ -616,8 +616,8 @@ public T map(Types.MapType map, T keyResult, T valueResult) { return null;

Re: [PR] Add table upsert support [iceberg-python]

2025-02-13 Thread via GitHub
mattmartin14 commented on PR #1660: URL: https://github.com/apache/iceberg-python/pull/1660#issuecomment-2657682312 @Fokko @kevinjqliu - should I go ahead and close the old PR now? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] CORE: return false when view exists endpoint isn't supported [iceberg]

2025-02-13 Thread via GitHub
danielcweeks commented on PR #12259: URL: https://github.com/apache/iceberg/pull/12259#issuecomment-2657694224 (cc @ajreid21) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Core: add variant type support [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1955196818 ## api/src/main/java/org/apache/iceberg/types/TypeUtil.java: ## @@ -712,6 +712,10 @@ public T map(Types.MapType map, Supplier keyResult, Supplier valueResult)

Re: [PR] Core: add variant type support [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1955196818 ## api/src/main/java/org/apache/iceberg/types/TypeUtil.java: ## @@ -712,6 +712,10 @@ public T map(Types.MapType map, Supplier keyResult, Supplier valueResult)

[PR] CORE: return false when view exists endpoint isn't supported [iceberg]

2025-02-13 Thread via GitHub
danielcweeks opened a new pull request, #12259: URL: https://github.com/apache/iceberg/pull/12259 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [PR] Core: add variant type support [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1955196818 ## api/src/main/java/org/apache/iceberg/types/TypeUtil.java: ## @@ -712,6 +712,10 @@ public T map(Types.MapType map, Supplier keyResult, Supplier valueResult)

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-13 Thread via GitHub
mattmartin14 closed pull request #1534: Feature: MERGE/Upsert Support URL: https://github.com/apache/iceberg-python/pull/1534 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Feature: MERGE/Upsert Support [iceberg-python]

2025-02-13 Thread via GitHub
mattmartin14 commented on PR #1534: URL: https://github.com/apache/iceberg-python/pull/1534#issuecomment-2657686000 Final stage of this PR was moved to new [PR #1660](https://github.com/apache/iceberg-python/pull/1660) due to some infrastructure challenges on my end. Closing this one now.

Re: [PR] Add table upsert support [iceberg-python]

2025-02-13 Thread via GitHub
mattmartin14 commented on PR #1660: URL: https://github.com/apache/iceberg-python/pull/1660#issuecomment-2657667851 > This looks great! Thanks again @mattmartin14 > > Regarding @bitsondatadev's comment, when you do a squash and commit, you'll see multiple authors, example can be found

Re: [PR] Add table upsert support [iceberg-python]

2025-02-13 Thread via GitHub
Fokko commented on PR #1660: URL: https://github.com/apache/iceberg-python/pull/1660#issuecomment-2657684518 @mattmartin14 Yes, please go ahead. Thanks everyone for driving this, @mattmartin14 in particular! -- This is an automated message from the Apache Git Service. To respond to the me

[PR] Spark: Remove unused PruneColumnsWithReordering class [iceberg]

2025-02-13 Thread via GitHub
rdblue opened a new pull request, #12258: URL: https://github.com/apache/iceberg/pull/12258 This class isn't used. If I remember correctly, the purpose of `PruneColumnsWithReordering` was to produce a schema that matched the requested Spark schema before DSv2 could add a projection to reord

Re: [PR] Add table upsert support [iceberg-python]

2025-02-13 Thread via GitHub
Fokko merged PR #1660: URL: https://github.com/apache/iceberg-python/pull/1660 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Add support for `write.metadata.path` [iceberg-python]

2025-02-13 Thread via GitHub
smaheshwar-pltr commented on code in PR #1642: URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1955183564 ## mkdocs/docs/configuration.md: ## @@ -203,12 +204,15 @@ PyIceberg uses [S3FileSystem](https://arrow.apache.org/docs/python/generated/pya ## Locatio

Re: [PR] Add support for `write.metadata.path` [iceberg-python]

2025-02-13 Thread via GitHub
smaheshwar-pltr commented on code in PR #1642: URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1955183564 ## mkdocs/docs/configuration.md: ## @@ -203,12 +204,15 @@ PyIceberg uses [S3FileSystem](https://arrow.apache.org/docs/python/generated/pya ## Locatio

Re: [PR] Core: add variant type support [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1955155159 ## core/src/main/java/org/apache/iceberg/avro/TypeToSchema.java: ## @@ -187,6 +187,18 @@ public Schema map(Types.MapType map, Schema keySchema, Schema valueSchema) {

Re: [PR] Add table upsert support [iceberg-python]

2025-02-13 Thread via GitHub
bitsondatadev commented on PR #1660: URL: https://github.com/apache/iceberg-python/pull/1660#issuecomment-2657648101 Hey @kevinjqliu, just a note after all the changes are done, it may be best to manually squash the commits and make sure Matt is the author for that...not sure what GitHub wi

Re: [PR] Core: add variant type support [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1955170959 ## core/src/test/java/org/apache/iceberg/avro/TestSchemaConversions.java: ## @@ -370,4 +370,22 @@ public void testFieldDocsArePreserved() { Lists.newArrayList

Re: [PR] feat: Add existing parquet files [iceberg-rust]

2025-02-13 Thread via GitHub
jonathanc-n commented on code in PR #960: URL: https://github.com/apache/iceberg-rust/pull/960#discussion_r1955169524 ## crates/iceberg/src/transaction.rs: ## @@ -169,6 +175,172 @@ impl<'a> Transaction<'a> { catalog.update_table(table_commit).await } + +/// A

Re: [PR] Core: add variant type support [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1955156020 ## core/src/main/java/org/apache/iceberg/schema/SchemaWithPartnerVisitor.java: ## @@ -106,7 +106,8 @@ public static T visit( } return visitor.map(

Re: [PR] Core: add variant type support [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1955155426 ## core/src/main/java/org/apache/iceberg/schema/SchemaWithPartnerVisitor.java: ## @@ -106,7 +106,8 @@ public static T visit( } return visitor.map(

Re: [PR] Core: add variant type support [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1955154197 ## api/src/test/java/org/apache/iceberg/types/TestTypeUtil.java: ## @@ -645,4 +651,70 @@ public void testReassignOrRefreshIdsCaseInsensitive() { requi

Re: [PR] Core: add variant type support [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1955149743 ## api/src/test/java/org/apache/iceberg/types/TestTypeUtil.java: ## @@ -645,4 +651,70 @@ public void testReassignOrRefreshIdsCaseInsensitive() { requi

Re: [PR] Core: Add InternalData read and write builders [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #12060: URL: https://github.com/apache/iceberg/pull/12060#discussion_r1955144135 ## core/src/main/java/org/apache/iceberg/V2Metadata.java: ## @@ -278,37 +277,41 @@ static Types.StructType fileType(Types.StructType partitionType) { DataFil

Re: [PR] feat: Add `StrictMetricsEvaluator` [iceberg-rust]

2025-02-13 Thread via GitHub
jonathanc-n commented on PR #963: URL: https://github.com/apache/iceberg-rust/pull/963#issuecomment-2657587163 @Fokko Thanks for the review. I got a bit mixed up with the logic with only nulls vs may contain null πŸ˜†. Should be good now -- This is an automated message from the Apache Git Se

Re: [PR] Core: Fix divide by zero when adjust split size [iceberg]

2025-02-13 Thread via GitHub
RussellSpitzer commented on PR #12201: URL: https://github.com/apache/iceberg/pull/12201#issuecomment-2657564295 Thanks @hantangwangd for the PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] Core: Fix divide by zero when adjust split size [iceberg]

2025-02-13 Thread via GitHub
RussellSpitzer merged PR #12201: URL: https://github.com/apache/iceberg/pull/12201 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Core: Add InternalData read and write builders [iceberg]

2025-02-13 Thread via GitHub
rdblue merged PR #12060: URL: https://github.com/apache/iceberg/pull/12060 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Core: BugFix: PartitionStatsUtil#computeStats returns incomplete stats in case of partition evolution [iceberg]

2025-02-13 Thread via GitHub
deniskuzZ commented on code in PR #12137: URL: https://github.com/apache/iceberg/pull/12137#discussion_r1955044758 ## core/src/test/java/org/apache/iceberg/TestPartitionStatsUtil.java: ## @@ -370,16 +371,167 @@ public void testPartitionStatsWithSchemaEvolution() throws Exceptio

Re: [PR] update site to include iceberg summit link [iceberg]

2025-02-13 Thread via GitHub
RussellSpitzer merged PR #12256: URL: https://github.com/apache/iceberg/pull/12256 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] update site to include iceberg summit link [iceberg]

2025-02-13 Thread via GitHub
RussellSpitzer commented on PR #12256: URL: https://github.com/apache/iceberg/pull/12256#issuecomment-2657549185 Thanks @danicafine ! Thanks @Fokko for the review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Add support for `write.metadata.path` [iceberg-python]

2025-02-13 Thread via GitHub
geruh commented on code in PR #1642: URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1955110870 ## pyiceberg/table/locations.py: ## @@ -64,6 +71,35 @@ def new_data_location(self, data_file_name: str, partition_key: Optional[Partiti str: A fully-

Re: [PR] API, Core: Support default values in UpdateSchema [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on PR #12211: URL: https://github.com/apache/iceberg/pull/12211#issuecomment-2657547085 Merged. Thanks for reviewing, @Fokko, @danielcweeks, and @aokolnychyi! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] API, Core: Support default values in UpdateSchema [iceberg]

2025-02-13 Thread via GitHub
rdblue merged PR #12211: URL: https://github.com/apache/iceberg/pull/12211 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Core: Add InternalData read and write builders [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on PR #12060: URL: https://github.com/apache/iceberg/pull/12060#issuecomment-2657544978 Merging this now that the 1.8.0 vote has passed. Thanks for reviewing, everyone! -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [PR] feat: Add `StrictMetricsEvaluator` [iceberg-rust]

2025-02-13 Thread via GitHub
jonathanc-n commented on code in PR #963: URL: https://github.com/apache/iceberg-rust/pull/963#discussion_r1955105071 ## crates/iceberg/src/expr/visitors/strict_metrics_evaluator.rs: ## @@ -0,0 +1,1854 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] feat: Add `StrictMetricsEvaluator` [iceberg-rust]

2025-02-13 Thread via GitHub
jonathanc-n commented on code in PR #963: URL: https://github.com/apache/iceberg-rust/pull/963#discussion_r1955094766 ## crates/iceberg/src/expr/visitors/strict_metrics_evaluator.rs: ## @@ -0,0 +1,1854 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] Add table upsert support [iceberg-python]

2025-02-13 Thread via GitHub
mattmartin14 commented on PR #1660: URL: https://github.com/apache/iceberg-python/pull/1660#issuecomment-2657519239 > cc reviewers from the other PR (@Fokko / @corleyma / @tscottcoombes1 / @marcoaanogueira) and the original PR author @mattmartin14 Looks great to me. Thanks for getting

Re: [PR] Core: BugFix: PartitionStatsUtil#computeStats returns incomplete stats in case of partition evolution [iceberg]

2025-02-13 Thread via GitHub
deniskuzZ commented on code in PR #12137: URL: https://github.com/apache/iceberg/pull/12137#discussion_r1955044758 ## core/src/test/java/org/apache/iceberg/TestPartitionStatsUtil.java: ## @@ -370,16 +371,167 @@ public void testPartitionStatsWithSchemaEvolution() throws Exceptio

Re: [PR] Core: BugFix: PartitionStatsUtil#computeStats returns incomplete stats in case of partition evolution [iceberg]

2025-02-13 Thread via GitHub
deniskuzZ commented on code in PR #12137: URL: https://github.com/apache/iceberg/pull/12137#discussion_r1955044758 ## core/src/test/java/org/apache/iceberg/TestPartitionStatsUtil.java: ## @@ -370,16 +371,167 @@ public void testPartitionStatsWithSchemaEvolution() throws Exceptio

Re: [PR] Core: BugFix: PartitionStatsUtil#computeStats returns incomplete stats in case of partition evolution [iceberg]

2025-02-13 Thread via GitHub
deniskuzZ commented on code in PR #12137: URL: https://github.com/apache/iceberg/pull/12137#discussion_r1955067425 ## core/src/test/java/org/apache/iceberg/TestPartitionStatsUtil.java: ## @@ -370,16 +371,167 @@ public void testPartitionStatsWithSchemaEvolution() throws Exceptio

Re: [PR] Add Variant custom logical type for Avro [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #12238: URL: https://github.com/apache/iceberg/pull/12238#discussion_r1955086053 ## core/src/main/java/org/apache/iceberg/avro/AvroCustomOrderSchemaVisitor.java: ## @@ -37,6 +37,10 @@ public static T visit(Schema schema, AvroCustomOrderSchemaVisi

Re: [PR] Add Variant custom logical type for Avro [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #12238: URL: https://github.com/apache/iceberg/pull/12238#discussion_r1955080951 ## core/src/test/java/org/apache/iceberg/avro/TestSchemaConversions.java: ## @@ -76,7 +78,8 @@ public void testPrimitiveTypes() { Schema.createFixed("fixe

Re: [PR] Core: BugFix: PartitionStatsUtil#computeStats returns incomplete stats in case of partition evolution [iceberg]

2025-02-13 Thread via GitHub
deniskuzZ commented on code in PR #12137: URL: https://github.com/apache/iceberg/pull/12137#discussion_r1955044758 ## core/src/test/java/org/apache/iceberg/TestPartitionStatsUtil.java: ## @@ -370,16 +371,167 @@ public void testPartitionStatsWithSchemaEvolution() throws Exceptio

Re: [PR] Core: BugFix: PartitionStatsUtil#computeStats returns incomplete stats in case of partition evolution [iceberg]

2025-02-13 Thread via GitHub
deniskuzZ commented on code in PR #12137: URL: https://github.com/apache/iceberg/pull/12137#discussion_r1955044758 ## core/src/test/java/org/apache/iceberg/TestPartitionStatsUtil.java: ## @@ -370,16 +371,167 @@ public void testPartitionStatsWithSchemaEvolution() throws Exceptio

Re: [PR] update site to include iceberg summit link [iceberg]

2025-02-13 Thread via GitHub
Fokko commented on code in PR #12256: URL: https://github.com/apache/iceberg/pull/12256#discussion_r1955078240 ## site/overrides/home.html: ## @@ -36,9 +36,9 @@ Apache Icebergβ„’ The open table format for analytic datasets. -https:

Re: [PR] Add Variant custom logical type for Avro [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #12238: URL: https://github.com/apache/iceberg/pull/12238#discussion_r1955075860 ## core/src/test/java/org/apache/iceberg/avro/TestSchemaConversions.java: ## @@ -76,7 +78,8 @@ public void testPrimitiveTypes() { Schema.createFixed("fixe

Re: [PR] Core: BugFix: PartitionStatsUtil#computeStats returns incomplete stats in case of partition evolution [iceberg]

2025-02-13 Thread via GitHub
deniskuzZ commented on code in PR #12137: URL: https://github.com/apache/iceberg/pull/12137#discussion_r1955044758 ## core/src/test/java/org/apache/iceberg/TestPartitionStatsUtil.java: ## @@ -370,16 +371,167 @@ public void testPartitionStatsWithSchemaEvolution() throws Exceptio

Re: [PR] Core: BugFix: PartitionStatsUtil#computeStats returns incomplete stats in case of partition evolution [iceberg]

2025-02-13 Thread via GitHub
deniskuzZ commented on code in PR #12137: URL: https://github.com/apache/iceberg/pull/12137#discussion_r1955044758 ## core/src/test/java/org/apache/iceberg/TestPartitionStatsUtil.java: ## @@ -370,16 +371,167 @@ public void testPartitionStatsWithSchemaEvolution() throws Exceptio

  1   2   >