Re: [PR] API: Define RepairManifests action interface [iceberg]

2024-08-07 Thread via GitHub
szehon-ho commented on PR #10784: URL: https://github.com/apache/iceberg/pull/10784#issuecomment-2275063775 Makes sense @amogh-jahagirdar ! > @szehon-ho I actually had a question on the snapshot repair, based on the description the goal of that is to repair snapshot summary stats wh

Re: [PR] Bump mypy-boto3-glue from 1.34.143 to 1.34.156 [iceberg-python]

2024-08-07 Thread via GitHub
Fokko merged PR #1021: URL: https://github.com/apache/iceberg-python/pull/1021 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Bump deptry from 0.18.0 to 0.19.0 [iceberg-python]

2024-08-07 Thread via GitHub
Fokko merged PR #1022: URL: https://github.com/apache/iceberg-python/pull/1022 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] API: Define RepairManifests action interface [iceberg]

2024-08-07 Thread via GitHub
amogh-jahagirdar commented on PR #10784: URL: https://github.com/apache/iceberg/pull/10784#issuecomment-2274937702 @szehon-ho I think I agree that the use cases presented are legitimate repair cases however I'm not sure that we should be combining all of them into a single `RepairTable` pro

Re: [PR] Update fsspec.py to respect s3.signer.uri property [iceberg-python]

2024-08-07 Thread via GitHub
guitcastro commented on PR #741: URL: https://github.com/apache/iceberg-python/pull/741#issuecomment-2274893639 > @Fokko thanks a lot for your feedback - I added docs and the constant. The constant is a very good idea - I hope we will be able to use remote signing with FileIO as well eventu

Re: [PR] AWS: Implement SupportsRecoveryOperations for S3FileIO [iceberg]

2024-08-07 Thread via GitHub
amogh-jahagirdar commented on code in PR #10721: URL: https://github.com/apache/iceberg/pull/10721#discussion_r1708529268 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java: ## @@ -420,4 +427,38 @@ protected void finalize() throws Throwable { } } } + +

Re: [PR] AWS: Implement SupportsRecoveryOperations for S3FileIO [iceberg]

2024-08-07 Thread via GitHub
amogh-jahagirdar commented on code in PR #10721: URL: https://github.com/apache/iceberg/pull/10721#discussion_r1708527555 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java: ## @@ -420,4 +427,38 @@ protected void finalize() throws Throwable { } } } + +

Re: [I] Can't import pyiceberg.catalog [iceberg-python]

2024-08-07 Thread via GitHub
kevinjqliu closed issue #500: Can't import pyiceberg.catalog URL: https://github.com/apache/iceberg-python/issues/500 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

Re: [PR] Spark: Add CopyTable spark action [iceberg]

2024-08-07 Thread via GitHub
huaxingao commented on PR #10024: URL: https://github.com/apache/iceberg/pull/10024#issuecomment-2274801248 @laithalzyoud Thanks for the thumbs-up! Could you please confirm if you are planning to continue working on this PR, or would you like me to take over? I’m happy to help in any way ne

Re: [I] Can't import pyiceberg.catalog [iceberg-python]

2024-08-07 Thread via GitHub
kevinjqliu commented on issue #500: URL: https://github.com/apache/iceberg-python/issues/500#issuecomment-2274801220 Addressed in #526 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Exclude Python 3.9.7 due to import error in catalog module [iceberg-python]

2024-08-07 Thread via GitHub
kevinjqliu merged PR #526: URL: https://github.com/apache/iceberg-python/pull/526 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ice

Re: [PR] Exclude Python 3.9.7 due to import error in catalog module [iceberg-python]

2024-08-07 Thread via GitHub
ndrluis commented on code in PR #526: URL: https://github.com/apache/iceberg-python/pull/526#discussion_r1708404061 ## pyproject.toml: ## @@ -49,7 +49,7 @@ include = [ ] [tool.poetry.dependencies] -python = "^3.8" +python = ">=3.8,<3.9.7 || >=3.9.8,<4.0" Review Comment:

Re: [PR] AWS: Implement SupportsRecoveryOperations for S3FileIO [iceberg]

2024-08-07 Thread via GitHub
singhpk234 commented on code in PR #10721: URL: https://github.com/apache/iceberg/pull/10721#discussion_r1708266721 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java: ## @@ -420,4 +427,38 @@ protected void finalize() throws Throwable { } } } + + @Overr

Re: [PR] Spark Action to Analyze table [iceberg]

2024-08-07 Thread via GitHub
aokolnychyi commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1708329708 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/ComputeTableStatsSparkAction.java: ## @@ -0,0 +1,164 @@ +/* + * Licensed to the Apache Software

Re: [PR] AWS: Implement SupportsRecoveryOperations for S3FileIO [iceberg]

2024-08-07 Thread via GitHub
singhpk234 commented on code in PR #10721: URL: https://github.com/apache/iceberg/pull/10721#discussion_r1708266721 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java: ## @@ -420,4 +427,38 @@ protected void finalize() throws Throwable { } } } + + @Overr

Re: [PR] AWS: Implement SupportsRecoveryOperations for S3FileIO [iceberg]

2024-08-07 Thread via GitHub
singhpk234 commented on code in PR #10721: URL: https://github.com/apache/iceberg/pull/10721#discussion_r1708266721 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java: ## @@ -420,4 +427,38 @@ protected void finalize() throws Throwable { } } } + + @Overr

Re: [I] Overwrite with filter division by zero error [iceberg-python]

2024-08-07 Thread via GitHub
Minfante377 commented on issue #1020: URL: https://github.com/apache/iceberg-python/issues/1020#issuecomment-2274625025 Looks like the `ResolveError` is fixed in main. Thanks for taking the time to reproduce the error! -- This is an automated message from the Apache Git Service. To re

Re: [I] Overwrite with filter division by zero error [iceberg-python]

2024-08-07 Thread via GitHub
ndrluis commented on issue #1020: URL: https://github.com/apache/iceberg-python/issues/1020#issuecomment-2274610759 @Minfante377 I was able to reproduce the code. ```python from pyiceberg.catalog import load_catalog import pyarrow as pa from pyiceberg.schema import Schema

Re: [I] Support to optimize, analyze tables and expire snapshots, remove orphan files [iceberg-python]

2024-08-07 Thread via GitHub
github-actions[bot] commented on issue #31: URL: https://github.com/apache/iceberg-python/issues/31#issuecomment-2274587982 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occu

Re: [I] Overwrite with filter division by zero error [iceberg-python]

2024-08-07 Thread via GitHub
Minfante377 commented on issue #1020: URL: https://github.com/apache/iceberg-python/issues/1020#issuecomment-2274587694 I'll try to generate a code example to reproduce the bug with some dummy data. Let me also try with the main branch regarding the ResolveError. Thank you for the quick

Re: [PR] Checkstyle: Add Checkstyle Rule to prevent Map and Set [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4620: URL: https://github.com/apache/iceberg/pull/4620#issuecomment-2274584738 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Flink: support watermark and computed columns [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4625: URL: https://github.com/apache/iceberg/pull/4625#issuecomment-2274584787 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] [Spark][MR]: Reuse InputFilesDecryptor [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4595: URL: https://github.com/apache/iceberg/pull/4595#issuecomment-2274584527 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Orc : Fix inner struct field as partition (#4604) [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4599: URL: https://github.com/apache/iceberg/pull/4599#issuecomment-2274584568 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Spark: Add Spark procedures for snapshot branching and tagging [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4586: URL: https://github.com/apache/iceberg/pull/4586#issuecomment-2274584483 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Change Data Capture(CDC)[Draft] [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4539: URL: https://github.com/apache/iceberg/pull/4539#issuecomment-2274584401 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] API: Add default value core api and schema serialization [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4525: URL: https://github.com/apache/iceberg/pull/4525#issuecomment-2274584244 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] Spark overwrite issue when partition changes [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on issue #4510: URL: https://github.com/apache/iceberg/issues/4510#issuecomment-2274584159 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Core: Skipping manifest clean-up for all Error or Exception. [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4507: URL: https://github.com/apache/iceberg/pull/4507#issuecomment-2274584120 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Docs: update Spark Write doc for partitioned tables [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4463: URL: https://github.com/apache/iceberg/pull/4463#issuecomment-2274584064 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Static Analysis: Rewrite empty and inappropriate Javadocs [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4438: URL: https://github.com/apache/iceberg/pull/4438#issuecomment-2274584007 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Spark: support stream-results in DeleteOrphanFiles action [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4451: URL: https://github.com/apache/iceberg/pull/4451#issuecomment-2274584031 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] There is no 'key_metadata' field in manifest_file, so delete it [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4399: URL: https://github.com/apache/iceberg/pull/4399#issuecomment-2274583831 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Enable parallel_delete in spark action to speed up snapshot expire [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4340: URL: https://github.com/apache/iceberg/pull/4340#issuecomment-2274583780 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] [WIP] API/Core: View support [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4657: URL: https://github.com/apache/iceberg/pull/4657#issuecomment-2274584932 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Support non-optional union types for ORC [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4654: URL: https://github.com/apache/iceberg/pull/4654#issuecomment-2274584882 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Flink 1.14: Implement LookupTableSource for IcebergTableSource [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4649: URL: https://github.com/apache/iceberg/pull/4649#issuecomment-2274584836 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] API: Drop column of deleted partitioned field to Unbound partitionSpec [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4602: URL: https://github.com/apache/iceberg/pull/4602#issuecomment-2274584611 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Core:Optimize Error msg in validateDeletedManifests (fix #3466) [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4612: URL: https://github.com/apache/iceberg/pull/4612#issuecomment-2274584692 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Support insert overwrite table for hive-mr engine. [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4605: URL: https://github.com/apache/iceberg/pull/4605#issuecomment-2274584652 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Spark: Implement the architecture to read default values [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4547: URL: https://github.com/apache/iceberg/pull/4547#issuecomment-2274584450 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Docs: add Cloudera native docs section [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4536: URL: https://github.com/apache/iceberg/pull/4536#issuecomment-2274584357 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] iceberg v2 table changelog streaming read feature support [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4528: URL: https://github.com/apache/iceberg/pull/4528#issuecomment-2274584304 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Core: Rewrite v2 table files. [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4522: URL: https://github.com/apache/iceberg/pull/4522#issuecomment-2274584206 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Static Analysis: Supressions we know are false positive [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4437: URL: https://github.com/apache/iceberg/pull/4437#issuecomment-2274583975 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Flink - Fix UPSERT delete file metadata [Backport to Flink 1.12] [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4418: URL: https://github.com/apache/iceberg/pull/4418#issuecomment-2274583879 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] [Spark][Core]: Release memory when rewriting RewriteFileGroup finished [iceberg]

2024-08-07 Thread via GitHub
github-actions[bot] commented on PR #4427: URL: https://github.com/apache/iceberg/pull/4427#issuecomment-2274583934 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] Overwrite with filter division by zero error [iceberg-python]

2024-08-07 Thread via GitHub
ndrluis commented on issue #1020: URL: https://github.com/apache/iceberg-python/issues/1020#issuecomment-2274570906 @Minfante377 Thank you for your report. Could you provide a minimal code example for the `ZeroDivisionError` case? Regarding the `ResolveError`, something was recently

Re: [I] Overwrite with filter division by zero error [iceberg-python]

2024-08-07 Thread via GitHub
Minfante377 commented on issue #1020: URL: https://github.com/apache/iceberg-python/issues/1020#issuecomment-2274528625 I don't know if it's related or not but when I try to perform a table.scan with a similar filter I'm getting an odd error too: ``` return self.table.scan(row

[PR] Bump deptry from 0.18.0 to 0.19.0 [iceberg-python]

2024-08-07 Thread via GitHub
dependabot[bot] opened a new pull request, #1022: URL: https://github.com/apache/iceberg-python/pull/1022 Bumps [deptry](https://github.com/fpgmaas/deptry) from 0.18.0 to 0.19.0. Release notes Sourced from https://github.com/fpgmaas/deptry/releases";>deptry's releases. 0.19.

[PR] Bump mypy-boto3-glue from 1.34.143 to 1.34.156 [iceberg-python]

2024-08-07 Thread via GitHub
dependabot[bot] opened a new pull request, #1021: URL: https://github.com/apache/iceberg-python/pull/1021 Bumps [mypy-boto3-glue](https://github.com/youtype/mypy_boto3_builder) from 1.34.143 to 1.34.156. Commits See full diff in https://github.com/youtype/mypy_boto3_builder/com

[I] Overwrite with filter division by zero error [iceberg-python]

2024-08-07 Thread via GitHub
Minfante377 opened a new issue, #1020: URL: https://github.com/apache/iceberg-python/issues/1020 ### Apache Iceberg version 0.7.0 (latest release) ### Please describe the bug 🐞 When doing an override with an specific overwrite_filter sometimes I'm getting a division by z

Re: [I] Using pyiceberg HiveCatalog to connect to hive4 [iceberg-python]

2024-08-07 Thread via GitHub
kevinjqliu commented on issue #1019: URL: https://github.com/apache/iceberg-python/issues/1019#issuecomment-2274369979 >Ah I see, thanks! Using the standlone-metastore container did indeed work with pyiceberg. Awesome! > When I start the standalone metastore, I'm able to conne

Re: [I] Using pyiceberg HiveCatalog to connect to hive4 [iceberg-python]

2024-08-07 Thread via GitHub
rni-HMC commented on issue #1019: URL: https://github.com/apache/iceberg-python/issues/1019#issuecomment-2274354720 Ah I see, thanks! Using the standlone-metastore container did indeed work with pyiceberg. When I start the standalone metastore, I'm able to connect via pyiceberg. But

Re: [PR] Allow setting `write.parquet.row-group-limit` [iceberg-python]

2024-08-07 Thread via GitHub
sungwy commented on PR #1016: URL: https://github.com/apache/iceberg-python/pull/1016#issuecomment-2274320281 LGTM @Fokko - merging in the change from main to resolve the conflict on the doc -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-08-07 Thread via GitHub
stevenzwu commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1707846715 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/SinkUtil.java: ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-08-07 Thread via GitHub
stevenzwu commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1707845200 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java: ## @@ -0,0 +1,811 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
sl255051 commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707698118 ## api/src/main/java/org/apache/iceberg/types/TypeUtil.java: ## @@ -181,11 +181,34 @@ public static Map indexQuotedNameById( return indexer.byId(); } + /*

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
sl255051 commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707732421 ## api/src/main/java/org/apache/iceberg/types/TypeUtil.java: ## @@ -181,11 +181,34 @@ public static Map indexQuotedNameById( return indexer.byId(); } + /*

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-08-07 Thread via GitHub
stevenzwu commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1707839978 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java: ## @@ -0,0 +1,811 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-08-07 Thread via GitHub
stevenzwu commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1707833724 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java: ## @@ -0,0 +1,811 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

2024-08-07 Thread via GitHub
amogh-jahagirdar commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r170001 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1004,24 +1004,28 @@ private Builder setInitialFormatVersion(int newFormatVersion) {

Re: [PR] Deprecate Redundant Identifier Support in TableIdentifier, and row_filter [iceberg-python]

2024-08-07 Thread via GitHub
sungwy commented on code in PR #994: URL: https://github.com/apache/iceberg-python/pull/994#discussion_r1707772219 ## tests/integration/test_writes/test_writes.py: ## @@ -1295,3 +1295,18 @@ def test_rest_catalog_with_empty_catalog_name_append_data(session_catalog: Catal )

Re: [I] Using pyiceberg HiveCatalog to connect to hive4 [iceberg-python]

2024-08-07 Thread via GitHub
kevinjqliu commented on issue #1019: URL: https://github.com/apache/iceberg-python/issues/1019#issuecomment-2274225056 The word "Hive" here is overloaded; there are two distinct systems when talking about Hive. 1. Hive, the query engine 2. Hive Metastore [Hive and Iceberg Quic

Re: [PR] Use `VisitorWithPartner` for name-mapping [iceberg-python]

2024-08-07 Thread via GitHub
sungwy commented on code in PR #1014: URL: https://github.com/apache/iceberg-python/pull/1014#discussion_r1706931818 ## pyiceberg/table/name_mapping.py: ## @@ -74,6 +75,11 @@ class NameMapping(IcebergRootModel[List[MappedField]]): def _field_by_name(self) -> Dict[str, Mappe

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
sl255051 commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707763657 ## core/src/test/java/org/apache/iceberg/TestPartitionSpecBuilderCaseSensitivity.java: ## @@ -0,0 +1,993 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] Improve test_version_format() error message for version mismatches [iceberg-python]

2024-08-07 Thread via GitHub
kevinjqliu commented on PR #1015: URL: https://github.com/apache/iceberg-python/pull/1015#issuecomment-2274217123 can you try running `make install` first? did you follow these steps? https://py.iceberg.apache.org/contributing/ I'm not very familiar with Windows environment --

Re: [PR] AWS: Fix flaky TestS3RestSigner [iceberg]

2024-08-07 Thread via GitHub
amogh-jahagirdar commented on code in PR #10898: URL: https://github.com/apache/iceberg/pull/10898#discussion_r1707758990 ## aws/src/test/java/org/apache/iceberg/aws/s3/signer/TestS3RestSigner.java: ## @@ -107,7 +107,13 @@ public static void afterClass() throws Exception {

Re: [PR] Improve test_version_format() error message for version mismatches [iceberg-python]

2024-08-07 Thread via GitHub
laksh-krishna-sharma commented on PR #1015: URL: https://github.com/apache/iceberg-python/pull/1015#issuecomment-2274206349 ![image](https://github.com/user-attachments/assets/31ad757c-c7e9-4730-9d51-e854d2a1f9dd) Sir, please guide me about what I have to do to resolve this. -- This i

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
sl255051 commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707750094 ## core/src/test/java/org/apache/iceberg/TestPartitionSpecInfo.java: ## @@ -20,6 +20,7 @@ import static org.apache.iceberg.types.Types.NestedField.required; impo

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
sl255051 commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707750094 ## core/src/test/java/org/apache/iceberg/TestPartitionSpecInfo.java: ## @@ -20,6 +20,7 @@ import static org.apache.iceberg.types.Types.NestedField.required; impo

Re: [PR] Improve test_version_format() error message for version mismatches [iceberg-python]

2024-08-07 Thread via GitHub
laksh-krishna-sharma commented on PR #1015: URL: https://github.com/apache/iceberg-python/pull/1015#issuecomment-2274199717 Sure, I'll run make lint again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
sl255051 commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707743014 ## api/src/main/java/org/apache/iceberg/types/TypeUtil.java: ## @@ -181,11 +181,34 @@ public static Map indexQuotedNameById( return indexer.byId(); } + /*

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
sl255051 commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707743014 ## api/src/main/java/org/apache/iceberg/types/TypeUtil.java: ## @@ -181,11 +181,34 @@ public static Map indexQuotedNameById( return indexer.byId(); } + /*

Re: [PR] Allow setting `write.parquet.page-row-limit` [iceberg-python]

2024-08-07 Thread via GitHub
ndrluis commented on PR #1017: URL: https://github.com/apache/iceberg-python/pull/1017#issuecomment-2274196689 WDYT about add a test verifying that the configuration from `write.parquet.page-row-limit` is passed down through `write_batch_size`as expected? -- This is an automated message

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
sl255051 commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707740896 ## api/src/main/java/org/apache/iceberg/PartitionSpec.java: ## @@ -450,13 +459,21 @@ Builder identity(String sourceName, String targetName) { return this;

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
sl255051 commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707740448 ## api/src/main/java/org/apache/iceberg/types/TypeUtil.java: ## @@ -181,11 +181,34 @@ public static Map indexQuotedNameById( return indexer.byId(); } + /*

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
sl255051 commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707739853 ## core/src/test/java/org/apache/iceberg/TestPartitionSpecBuilderCaseSensitivity.java: ## @@ -0,0 +1,993 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
sl255051 commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707732421 ## api/src/main/java/org/apache/iceberg/types/TypeUtil.java: ## @@ -181,11 +181,34 @@ public static Map indexQuotedNameById( return indexer.byId(); } + /*

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-08-07 Thread via GitHub
pvary commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1707718356 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerLockFactory.java: ## @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
rdblue commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707712257 ## core/src/test/java/org/apache/iceberg/TestPartitionSpecBuilderCaseSensitivity.java: ## @@ -0,0 +1,993 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
rdblue commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707707143 ## core/src/test/java/org/apache/iceberg/TestPartitionSpecBuilderCaseSensitivity.java: ## @@ -0,0 +1,993 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
rdblue commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707706083 ## core/src/test/java/org/apache/iceberg/TestPartitionSpecBuilderCaseSensitivity.java: ## @@ -0,0 +1,993 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
rdblue commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707699728 ## core/src/test/java/org/apache/iceberg/TestPartitionSpecBuilderCaseSensitivity.java: ## @@ -0,0 +1,993 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
rdblue commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707698954 ## core/src/test/java/org/apache/iceberg/TestPartitionSpecBuilderCaseSensitivity.java: ## @@ -0,0 +1,993 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
sl255051 commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707698118 ## api/src/main/java/org/apache/iceberg/types/TypeUtil.java: ## @@ -181,11 +181,34 @@ public static Map indexQuotedNameById( return indexer.byId(); } + /*

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
rdblue commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707696368 ## core/src/test/java/org/apache/iceberg/TestPartitionSpecBuilderCaseSensitivity.java: ## @@ -0,0 +1,993 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
rdblue commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707690572 ## core/src/test/java/org/apache/iceberg/TestPartitionSpecInfo.java: ## @@ -20,6 +20,7 @@ import static org.apache.iceberg.types.Types.NestedField.required; import

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
rdblue commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707689251 ## api/src/test/java/org/apache/iceberg/TestSchemaCaseSensitivity.java: ## @@ -0,0 +1,68 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
rdblue commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707687316 ## api/src/test/java/org/apache/iceberg/TestSchemaCaseSensitivity.java: ## @@ -0,0 +1,68 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
rdblue commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707675526 ## api/src/main/java/org/apache/iceberg/types/TypeUtil.java: ## @@ -181,11 +181,34 @@ public static Map indexQuotedNameById( return indexer.byId(); } + /**

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
rdblue commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707681345 ## api/src/main/java/org/apache/iceberg/types/TypeUtil.java: ## @@ -181,11 +181,34 @@ public static Map indexQuotedNameById( return indexer.byId(); } + /**

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
rdblue commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707675526 ## api/src/main/java/org/apache/iceberg/types/TypeUtil.java: ## @@ -181,11 +181,34 @@ public static Map indexQuotedNameById( return indexer.byId(); } + /**

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
rdblue commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707662722 ## api/src/main/java/org/apache/iceberg/types/TypeUtil.java: ## @@ -181,11 +181,34 @@ public static Map indexQuotedNameById( return indexer.byId(); } + /**

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
rdblue commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707660458 ## api/src/main/java/org/apache/iceberg/types/TypeUtil.java: ## @@ -181,11 +181,34 @@ public static Map indexQuotedNameById( return indexer.byId(); } + /**

Re: [PR] #10668 - Support case-insensitivity for column names in PartitionSpec [iceberg]

2024-08-07 Thread via GitHub
rdblue commented on code in PR #10678: URL: https://github.com/apache/iceberg/pull/10678#discussion_r1707655607 ## api/src/main/java/org/apache/iceberg/PartitionSpec.java: ## @@ -450,13 +459,21 @@ Builder identity(String sourceName, String targetName) { return this;

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

2024-08-07 Thread via GitHub
leangjonathan commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1707652177 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -1451,50 +1457,67 @@ public void testCreateV2MetadataThroughTableProperty() { .d

[I] Using pyiceberg HiveCatalog to connect to hive4 [iceberg-python]

2024-08-07 Thread via GitHub
rni-HMC opened a new issue, #1019: URL: https://github.com/apache/iceberg-python/issues/1019 ### Question ### Query engine HIVE ### Question I've followed the [Hive and Iceberg Quickstart](https://iceberg.apache.org/hive-quickstart/#docker-images), and have the h

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

2024-08-07 Thread via GitHub
leangjonathan commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1707621872 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1020,8 +1020,12 @@ public Builder upgradeFormatVersion(int newFormatVersion) { ret

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

2024-08-07 Thread via GitHub
leangjonathan commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1707621872 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1020,8 +1020,12 @@ public Builder upgradeFormatVersion(int newFormatVersion) { ret

  1   2   >