Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.54.2 to 1.54.3 [iceberg-go]

2024-05-26 Thread via GitHub
Fokko merged PR #87: URL: https://github.com/apache/iceberg-go/pull/87 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache

Re: [PR] feat: Adding literals [iceberg-go]

2024-05-26 Thread via GitHub
Fokko commented on PR #76: URL: https://github.com/apache/iceberg-go/pull/76#issuecomment-2132772683 @zeroshade Thanks! Can you fix the conflicts? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Implement caching of manifest-files [iceberg-python]

2024-05-26 Thread via GitHub
chinmay-bhat commented on issue #595: URL: https://github.com/apache/iceberg-python/issues/595#issuecomment-2132772245 Thank you for clarifying! Here's how I imagine manifests() would look like :) ``` @lru_cache() def manifests(self, manifest_location: str) -> List[ManifestFi

Re: [I] [bug]OversizedAllocationException when query data with Spark [iceberg]

2024-05-26 Thread via GitHub
zhangpenggh commented on issue #9820: URL: https://github.com/apache/iceberg/issues/9820#issuecomment-2132707438 > Set table property:`read.parquet.vectorization.batch-size` to a lower value, or just disable VECTORIZATION reader by setting `read.parquet.vectorization.enabled`=false should s

Re: [I] [bug]OversizedAllocationException when query data with Spark [iceberg]

2024-05-26 Thread via GitHub
zhangpenggh closed issue #9820: [bug]OversizedAllocationException when query data with Spark URL: https://github.com/apache/iceberg/issues/9820 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [I] Introduce a parameter to control whether the flink writer is linked with the previous operator [iceberg]

2024-05-26 Thread via GitHub
pvary commented on issue #10371: URL: https://github.com/apache/iceberg/issues/10371#issuecomment-2132677428 @huyuanfeng2018: Please link the discussion here, so it it's easy to follow the discussion. Thanks, Peter -- This is an automated message from the Apache Git Service. To respon

Re: [PR] Spark Action to Analyze table [iceberg]

2024-05-26 Thread via GitHub
jeesou commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1615485750 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/AnalyzeTableSparkAction.java: ## @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

2024-05-26 Thread via GitHub
catkins commented on PR #10151: URL: https://github.com/apache/iceberg/pull/10151#issuecomment-2132549187 This would be a great addition, I got a bit caught out in development this week when setting up an iceberg sink on my local docker flink setup, and being confused why the s3 config from

Re: [I] The decimal data type is transformed after the data is inserted. [iceberg-python]

2024-05-26 Thread via GitHub
ndrluis commented on issue #751: URL: https://github.com/apache/iceberg-python/issues/751#issuecomment-2132521441 The problem is not with append; it is something occurring when we call model_copy(deep=True) on the metadata object. When I was debugging, I couldn't find where the update

Re: [PR] Spark 3.5: Only traverse ancestors of current snapshot when building changelog scan [iceberg]

2024-05-26 Thread via GitHub
flyrain commented on code in PR #10252: URL: https://github.com/apache/iceberg/pull/10252#discussion_r1615317449 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestChangelogTable.java: ## @@ -291,6 +291,59 @@ public void testMetadataColumns() {

Re: [I] Implement caching of manifest-files [iceberg-python]

2024-05-26 Thread via GitHub
Fokko commented on issue #595: URL: https://github.com/apache/iceberg-python/issues/595#issuecomment-2132381356 @chinmay-bhat I don't think it is as easy as that. We should ensure that the `manifest_list` path is part of the cache. We could share the cache between calls, since if you do sub

Re: [PR] AWS: Retain Glue Catalog column comment after updating Iceberg table [iceberg]

2024-05-26 Thread via GitHub
lawofcycles commented on PR #10276: URL: https://github.com/apache/iceberg/pull/10276#issuecomment-2132371867 @geruh Thank you for your review! I have pushed the changes addressing your review comments. Could you please take another look when you have a chance? -- This is an automat

Re: [I] Implement caching of manifest-files [iceberg-python]

2024-05-26 Thread via GitHub
chinmay-bhat commented on issue #595: URL: https://github.com/apache/iceberg-python/issues/595#issuecomment-2132370830 Thanks @Fokko for the quick response. > based on the path to the metadata to cache it I'm not clear on this. Are you saying we can simply add `lru_cache` to `de

Re: [I] Implement caching of manifest-files [iceberg-python]

2024-05-26 Thread via GitHub
Fokko commented on issue #595: URL: https://github.com/apache/iceberg-python/issues/595#issuecomment-2132364651 @chinmay-bhat I think we can generalize this quite easily, since from the spec: > Once written, data and metadata files are immutable until they are deleted. I think

Re: [I] Implement caching of manifest-files [iceberg-python]

2024-05-26 Thread via GitHub
chinmay-bhat commented on issue #595: URL: https://github.com/apache/iceberg-python/issues/595#issuecomment-2132350280 Hi, can we increase the scope of this issue to cache/store all_manifests, data_manifests & delete_manifests? Or do I create a new issue for this? This feature would be usef

Re: [I] Introduce a parameter to control whether the flink writer is linked with the previous operator [iceberg]

2024-05-26 Thread via GitHub
huyuanfeng2018 commented on issue #10371: URL: https://github.com/apache/iceberg/issues/10371#issuecomment-2132190924 > @huyuanfeng2018: Did we try to add this feature to Flink? What was the response of the community there? Seems suboptimal for me to add this feature to every connector inde

Re: [I] Introduce a parameter to control whether the flink writer is linked with the previous operator [iceberg]

2024-05-26 Thread via GitHub
pvary commented on issue #10371: URL: https://github.com/apache/iceberg/issues/10371#issuecomment-2132166526 @huyuanfeng2018: Did we try to add this feature to Flink? What was the response of the community there? Seems suboptimal for me to add this feature to every connector independently..

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.27.15 to 1.27.16 [iceberg-go]

2024-05-26 Thread via GitHub
dependabot[bot] opened a new pull request, #90: URL: https://github.com/apache/iceberg-go/pull/90 Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.27.15 to 1.27.16. Commits https://github.com/aws/aws-sdk-go-v2/commit/8abec4c1a7b9326c05fe

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.15 to 1.17.16 [iceberg-go]

2024-05-26 Thread via GitHub
dependabot[bot] opened a new pull request, #89: URL: https://github.com/apache/iceberg-go/pull/89 Bumps [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) from 1.17.15 to 1.17.16. Commits https://github.com/aws/aws-sdk-go-v2/commit/8abec4c1a7b9326

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/glue from 1.80.3 to 1.81.1 [iceberg-go]

2024-05-26 Thread via GitHub
dependabot[bot] opened a new pull request, #88: URL: https://github.com/apache/iceberg-go/pull/88 Bumps [github.com/aws/aws-sdk-go-v2/service/glue](https://github.com/aws/aws-sdk-go-v2) from 1.80.3 to 1.81.1. Commits https://github.com/aws/aws-sdk-go-v2/commit/8abec4c1a7b9326c

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.54.2 to 1.54.3 [iceberg-go]

2024-05-26 Thread via GitHub
dependabot[bot] opened a new pull request, #87: URL: https://github.com/apache/iceberg-go/pull/87 Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.54.2 to 1.54.3. Commits https://github.com/aws/aws-sdk-go-v2/commit/8abec4c1a7b9326c05