[PR] Core: Add LoadCredentialsResponse class/parser [iceberg]

2024-10-16 Thread via GitHub
nastra opened a new pull request, #11339: URL: https://github.com/apache/iceberg/pull/11339 This implements the spec changes from https://github.com/apache/iceberg/pull/11281 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] OpenAPI: Add endpoint for refreshing vended credentials [iceberg]

2024-10-16 Thread via GitHub
nastra commented on code in PR #11281: URL: https://github.com/apache/iceberg/pull/11281#discussion_r1804199315 ## open-api/rest-catalog-open-api.yaml: ## @@ -3103,6 +3141,32 @@ components: uuid: type: string +Credential: + type: object + requ

Re: [PR] Remove repeated 'for' [iceberg]

2024-10-16 Thread via GitHub
nastra merged PR #11338: URL: https://github.com/apache/iceberg/pull/11338 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.37 to 1.17.41 [iceberg-go]

2024-10-16 Thread via GitHub
dependabot[bot] closed pull request #165: build(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.37 to 1.17.41 URL: https://github.com/apache/iceberg-go/pull/165 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Config File Handling [iceberg-go]

2024-10-16 Thread via GitHub
nastra commented on PR #156: URL: https://github.com/apache/iceberg-go/pull/156#issuecomment-2418645089 @alex-kar can you please rebase and fix the merge conflict? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.37 to 1.17.41 [iceberg-go]

2024-10-16 Thread via GitHub
dependabot[bot] commented on PR #165: URL: https://github.com/apache/iceberg-go/pull/165#issuecomment-2418644026 Looks like github.com/aws/aws-sdk-go-v2/credentials is up-to-date now, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to t

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.27.39 to 1.28.0 [iceberg-go]

2024-10-16 Thread via GitHub
nastra merged PR #178: URL: https://github.com/apache/iceberg-go/pull/178 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [I] Spark:read iceberg table data error [iceberg]

2024-10-16 Thread via GitHub
nastra commented on issue #11336: URL: https://github.com/apache/iceberg/issues/11336#issuecomment-2418641364 @beyond-up can you share the full stack trace please? Usually there's some more info in other parts of the stack trace that show what went wrong -- This is an automated message fr

Re: [PR] feat(catalog/glue): add support for glue catalog namespace operations [iceberg-go]

2024-10-16 Thread via GitHub
nastra commented on code in PR #173: URL: https://github.com/apache/iceberg-go/pull/173#discussion_r1804185400 ## catalog/glue.go: ## @@ -30,22 +30,40 @@ import ( "github.com/aws/aws-sdk-go-v2/service/glue/types" ) -const glueTypeIceberg = "ICEBERG" +const ( + g

Re: [PR] feat(catalog/glue): add support for glue catalog namespace operations [iceberg-go]

2024-10-16 Thread via GitHub
nastra commented on code in PR #173: URL: https://github.com/apache/iceberg-go/pull/173#discussion_r1804182610 ## catalog/glue.go: ## @@ -180,8 +267,155 @@ func (c *GlueCatalog) ListNamespaces(ctx context.Context, parent table.Identifie return icebergNamespaces, nil }

Re: [PR] feat(catalog/glue): add support for glue catalog namespace operations [iceberg-go]

2024-10-16 Thread via GitHub
nastra commented on code in PR #173: URL: https://github.com/apache/iceberg-go/pull/173#discussion_r1804180423 ## catalog/glue.go: ## @@ -122,33 +149,93 @@ func (c *GlueCatalog) LoadTable(ctx context.Context, identifier table.Identifier return icebergTable, nil } -fu

Re: [PR] feat(catalog/glue): add support for glue catalog namespace operations [iceberg-go]

2024-10-16 Thread via GitHub
nastra commented on code in PR #173: URL: https://github.com/apache/iceberg-go/pull/173#discussion_r1804177804 ## catalog/glue.go: ## @@ -122,33 +149,93 @@ func (c *GlueCatalog) LoadTable(ctx context.Context, identifier table.Identifier return icebergTable, nil } -fu

Re: [PR] feat(catalog/glue): add support for glue catalog namespace operations [iceberg-go]

2024-10-16 Thread via GitHub
nastra commented on code in PR #173: URL: https://github.com/apache/iceberg-go/pull/173#discussion_r1804175030 ## catalog/glue.go: ## @@ -122,33 +149,93 @@ func (c *GlueCatalog) LoadTable(ctx context.Context, identifier table.Identifier return icebergTable, nil } -fu

Re: [I] Document table properties [iceberg-python]

2024-10-16 Thread via GitHub
sikehish commented on issue #1231: URL: https://github.com/apache/iceberg-python/issues/1231#issuecomment-2418621291 @kevinjqliu Hi, I've created a PR #1232 . Let me know if any changes are to be made. Thank you for the oppurtunity :) -- This is an automated message from the Apac

[PR] docs/configuration.md: Documented table properties (#1231) [iceberg-python]

2024-10-16 Thread via GitHub
sikehish opened a new pull request, #1232: URL: https://github.com/apache/iceberg-python/pull/1232 This PR is for #1231. ### **Changes** - Added missing table properties to the `configuration.md` file, including: - `write.target-file-size-bytes` - `write.parquet.row-grou

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.27.39 to 1.27.43 [iceberg-go]

2024-10-16 Thread via GitHub
dependabot[bot] closed pull request #167: build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.27.39 to 1.27.43 URL: https://github.com/apache/iceberg-go/pull/167 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.27.39 to 1.27.43 [iceberg-go]

2024-10-16 Thread via GitHub
dependabot[bot] commented on PR #167: URL: https://github.com/apache/iceberg-go/pull/167#issuecomment-2418602984 Superseded by #178. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.27.39 to 1.28.0 [iceberg-go]

2024-10-16 Thread via GitHub
dependabot[bot] opened a new pull request, #178: URL: https://github.com/apache/iceberg-go/pull/178 Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.27.39 to 1.28.0. Commits https://github.com/aws/aws-sdk-go-v2/commit/54f11c0ac651fbec8d6

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.61.2 to 1.65.3 [iceberg-go]

2024-10-16 Thread via GitHub
nastra merged PR #170: URL: https://github.com/apache/iceberg-go/pull/170 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Core: Allow servers to express supported endpoints via endpoint field in ConfigResponse [iceberg]

2024-10-16 Thread via GitHub
nastra commented on PR #10929: URL: https://github.com/apache/iceberg/pull/10929#issuecomment-2418591566 @danielhumanmod this will be shipped with Iceberg 1.7.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [I] Failed to load FileIO implementation: org.apache.iceberg.azure.adlsv2.ADLSFileIO [iceberg]

2024-10-16 Thread via GitHub
ismaelhamed closed issue #11330: Failed to load FileIO implementation: org.apache.iceberg.azure.adlsv2.ADLSFileIO URL: https://github.com/apache/iceberg/issues/11330 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] Failed to load FileIO implementation: org.apache.iceberg.azure.adlsv2.ADLSFileIO [iceberg]

2024-10-16 Thread via GitHub
ismaelhamed commented on issue #11330: URL: https://github.com/apache/iceberg/issues/11330#issuecomment-2418590399 Indeed, thanks @nastra ! Closing now, as this is definitely not a bug. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Spark: Add RewriteTablePath action interface [iceberg]

2024-10-16 Thread via GitHub
szehon-ho commented on code in PR #10920: URL: https://github.com/apache/iceberg/pull/10920#discussion_r1804149603 ## api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java: ## @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

Re: [PR] Spark: Add RewriteTablePath action interface [iceberg]

2024-10-16 Thread via GitHub
szehon-ho commented on code in PR #10920: URL: https://github.com/apache/iceberg/pull/10920#discussion_r1804149603 ## api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java: ## @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

Re: [PR] SnapshotExpireAction get snapshot to be expired [iceberg]

2024-10-16 Thread via GitHub
stevie9868 closed pull request #11337: SnapshotExpireAction get snapshot to be expired URL: https://github.com/apache/iceberg/pull/11337 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Added Unit Test for Flink-Spark Equality Delete Write [iceberg]

2024-10-16 Thread via GitHub
adelly13 closed pull request #11328: Added Unit Test for Flink-Spark Equality Delete Write URL: https://github.com/apache/iceberg/pull/11328 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [I] Document table properties [iceberg-python]

2024-10-16 Thread via GitHub
sikehish commented on issue #1231: URL: https://github.com/apache/iceberg-python/issues/1231#issuecomment-2418462613 > @sikehish assigned! Thank you, I'm working on it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [PR] Arrow: Fix indexing in Parquet dictionary encoded values readers [iceberg]

2024-10-16 Thread via GitHub
amogh-jahagirdar commented on PR #11247: URL: https://github.com/apache/iceberg/pull/11247#issuecomment-2418462602 @wypoon Sure, I will take a look tomorrow! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] Core: Allow servers to express supported endpoints via endpoint field in ConfigResponse [iceberg]

2024-10-16 Thread via GitHub
danielhumanmod commented on PR #10929: URL: https://github.com/apache/iceberg/pull/10929#issuecomment-2418440252 Hi, may I know which version of Iceberg will contains this commit? I’m working on a feature in Apache Polaris to support endpoints in /v1/config. Understanding the release timeli

[I] Spark:read iceberg table data error [iceberg]

2024-10-16 Thread via GitHub
beyond-up opened a new issue, #11336: URL: https://github.com/apache/iceberg/issues/11336 ### Apache Iceberg version 1.5.2 ### Query engine Spark ### Please describe the bug šŸž use iceberg-spark-runtime-3.3_2.12-1.5.2.jar java.lang.NullPointerException

Re: [PR] Kafka Connect: Add kerberos authentication option [iceberg]

2024-10-16 Thread via GitHub
Dawnpool commented on PR #10173: URL: https://github.com/apache/iceberg/pull/10173#issuecomment-2418414725 @bryanck Flink itself has options for Kerberos authentication, not the Flink Iceberg sink. Here is the reference [link](https://nightlies.apache.org/flink/flink-docs-master/docs/de

Re: [PR] Handling NO Coordinator Scenario and Data Loss in the current Design [iceberg]

2024-10-16 Thread via GitHub
kumarpritam863 commented on PR #11298: URL: https://github.com/apache/iceberg/pull/11298#issuecomment-2418328906 Hi Bryan, This PR represents a consolidated view of the following two PRs, which overlap with this one: 1. https://github.com/apache/iceberg/pull/11288 2. https:/

Re: [PR] Workers gets stuck as there is no-coordinator for emitting Start_Commit request in Incremental Cooperative Rebalancing[ICR] Mode [iceberg]

2024-10-16 Thread via GitHub
kumarpritam863 commented on PR #11288: URL: https://github.com/apache/iceberg/pull/11288#issuecomment-2418324422 Hi Bryan, Thank you for the review and for your valuable input. I'd like to clarify the behavior in the different rebalancing modes: In Eager mode, during any rebala

Re: [PR] Config for deciding whether to use Iceberg Time type [iceberg]

2024-10-16 Thread via GitHub
kumarpritam863 commented on PR #11174: URL: https://github.com/apache/iceberg/pull/11174#issuecomment-2418322573 Hi Bryan, Thank you for your response. I completely agree with your view that the logic for Single Message Transforms (SMTs) should not be part of any sink. However, in th

Re: [PR] Added support for evolving the partition of the table [iceberg]

2024-10-16 Thread via GitHub
kumarpritam863 commented on PR #11275: URL: https://github.com/apache/iceberg/pull/11275#issuecomment-2418306851 Hi Bryan, Thank you for your insightful review. I agree with your observations. We are already ensuring that the table is created with the necessary schemas and partitions

Re: [PR] Handling NO Coordinator Scenario and Data Loss in the current Design [iceberg]

2024-10-16 Thread via GitHub
kumarpritam863 commented on PR #11298: URL: https://github.com/apache/iceberg/pull/11298#issuecomment-2418301248 Thanks @bryanck for the review. Sure I will consolidate these in one PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Workers gets stuck as there is no-coordinator for emitting Start_Commit request in Incremental Cooperative Rebalancing[ICR] Mode [iceberg]

2024-10-16 Thread via GitHub
kumarpritam863 commented on PR #11288: URL: https://github.com/apache/iceberg/pull/11288#issuecomment-2418300052 Thanks @bryanck for the review. In the **Eager** mode, since on any rebalance all the tasks first leave all the partitions that they have and then they rejoin hence in this case

Re: [PR] Spark: Add RewriteTablePath action interface [iceberg]

2024-10-16 Thread via GitHub
flyrain commented on code in PR #10920: URL: https://github.com/apache/iceberg/pull/10920#discussion_r1803960871 ## api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java: ## @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Spark: Add RewriteTablePath action interface [iceberg]

2024-10-16 Thread via GitHub
flyrain commented on code in PR #10920: URL: https://github.com/apache/iceberg/pull/10920#discussion_r1803960871 ## api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java: ## @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [I] TypeError when `operation` field is missing in `summary`. [iceberg-python]

2024-10-16 Thread via GitHub
buremba commented on issue #1106: URL: https://github.com/apache/iceberg-python/issues/1106#issuecomment-2418203863 That sounds like a good plan @kevinjqliu ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [I] Connect to multiple Azure accounts [iceberg-python]

2024-10-16 Thread via GitHub
github-actions[bot] commented on issue #636: URL: https://github.com/apache/iceberg-python/issues/636#issuecomment-2418202240 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity oc

Re: [I] Truncate does not support long widths [iceberg]

2024-10-16 Thread via GitHub
github-actions[bot] commented on issue #9716: URL: https://github.com/apache/iceberg/issues/9716#issuecomment-2418196523 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] truncate partition transform width greater than long errors out [iceberg]

2024-10-16 Thread via GitHub
github-actions[bot] commented on issue #9715: URL: https://github.com/apache/iceberg/issues/9715#issuecomment-2418196507 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Date partition transforms are underspecified [iceberg]

2024-10-16 Thread via GitHub
github-actions[bot] commented on issue #9714: URL: https://github.com/apache/iceberg/issues/9714#issuecomment-2418196486 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Extend check-nullability parameter scope to allow writing optional list elements and map values to required elements and values [iceberg]

2024-10-16 Thread via GitHub
github-actions[bot] closed issue #9091: Extend check-nullability parameter scope to allow writing optional list elements and map values to required elements and values URL: https://github.com/apache/iceberg/issues/9091 -- This is an automated message from the Apache Git Service. To respond t

Re: [I] Failed to create namespace using spark sql based on iceberg hadoop catalog (rest catalog) [iceberg]

2024-10-16 Thread via GitHub
github-actions[bot] closed issue #9072: Failed to create namespace using spark sql based on iceberg hadoop catalog (rest catalog) URL: https://github.com/apache/iceberg/issues/9072 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [I] Extend check-nullability parameter scope to allow writing optional list elements and map values to required elements and values [iceberg]

2024-10-16 Thread via GitHub
github-actions[bot] commented on issue #9091: URL: https://github.com/apache/iceberg/issues/9091#issuecomment-2418196056 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Failed to create namespace using spark sql based on iceberg hadoop catalog (rest catalog) [iceberg]

2024-10-16 Thread via GitHub
github-actions[bot] commented on issue #9072: URL: https://github.com/apache/iceberg/issues/9072#issuecomment-2418196036 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Iceberg: Partition-Level Tagging Support [iceberg]

2024-10-16 Thread via GitHub
github-actions[bot] closed issue #9060: Iceberg: Partition-Level Tagging Support URL: https://github.com/apache/iceberg/issues/9060 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [I] Error generating Go code from rest-catalog-open-api.yaml [iceberg]

2024-10-16 Thread via GitHub
github-actions[bot] closed issue #9070: Error generating Go code from rest-catalog-open-api.yaml URL: https://github.com/apache/iceberg/issues/9070 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [I] Error generating Go code from rest-catalog-open-api.yaml [iceberg]

2024-10-16 Thread via GitHub
github-actions[bot] commented on issue #9070: URL: https://github.com/apache/iceberg/issues/9070#issuecomment-2418196021 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Iceberg: Partition-Level Tagging Support [iceberg]

2024-10-16 Thread via GitHub
github-actions[bot] commented on issue #9060: URL: https://github.com/apache/iceberg/issues/9060#issuecomment-2418195999 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [PR] Support wasb[s] paths in ADLSFileIO [iceberg]

2024-10-16 Thread via GitHub
Jordano-Dremio commented on PR #11294: URL: https://github.com/apache/iceberg/pull/11294#issuecomment-2418162913 Hi @mrcnc, @RussellSpitzer . To confirm this PR solution to 10127, any incoming iceberg tables stored in azure will interpret the metadata/data with the `abfss` prefix + `.dfs.`

Re: [PR] Spec: Adds Row Lineage [iceberg]

2024-10-16 Thread via GitHub
aokolnychyi commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r1803912103 ## format/spec.md: ## @@ -298,16 +298,101 @@ Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALU The set of metadata columns is:

Re: [PR] Spec: Adds Row Lineage [iceberg]

2024-10-16 Thread via GitHub
aokolnychyi commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r1803911588 ## format/spec.md: ## @@ -298,16 +298,101 @@ Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALU The set of metadata columns is:

Re: [PR] Spec: Adds Row Lineage [iceberg]

2024-10-16 Thread via GitHub
aokolnychyi commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r1803911003 ## format/spec.md: ## @@ -298,16 +298,101 @@ Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALU The set of metadata columns is:

Re: [PR] Spec: Adds Row Lineage [iceberg]

2024-10-16 Thread via GitHub
aokolnychyi commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r1803911003 ## format/spec.md: ## @@ -298,16 +298,101 @@ Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALU The set of metadata columns is:

Re: [PR] Spec: Adds Row Lineage [iceberg]

2024-10-16 Thread via GitHub
aokolnychyi commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r1803908468 ## format/spec.md: ## @@ -298,16 +298,101 @@ Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALU The set of metadata columns is:

Re: [PR] Spec: Adds Row Lineage [iceberg]

2024-10-16 Thread via GitHub
aokolnychyi commented on code in PR #11130: URL: https://github.com/apache/iceberg/pull/11130#discussion_r1803908468 ## format/spec.md: ## @@ -298,16 +298,101 @@ Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALU The set of metadata columns is:

Re: [PR] IO Implementation using Go CDK [iceberg-go]

2024-10-16 Thread via GitHub
dwilson1988 commented on PR #176: URL: https://github.com/apache/iceberg-go/pull/176#issuecomment-2418127366 Okay, played around a bit and here's where my head is at. The main reason I'd like to isolated the creation of a `*blob.Bucket` is I've found that the particular implementation

Re: [PR] AWS: Fix S3InputStream retry policy [iceberg]

2024-10-16 Thread via GitHub
edgarRd commented on PR #11335: URL: https://github.com/apache/iceberg/pull/11335#issuecomment-2418102961 @amogh-jahagirdar I've address the comment. PTAL - Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] add checkClassName regex func [iceberg]

2024-10-16 Thread via GitHub
bryanck merged PR #11303: URL: https://github.com/apache/iceberg/pull/11303 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] AWS: Fix S3InputStream retry policy [iceberg]

2024-10-16 Thread via GitHub
edgarRd commented on code in PR #11335: URL: https://github.com/apache/iceberg/pull/11335#discussion_r1803884005 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java: ## @@ -66,10 +70,14 @@ class S3InputStream extends SeekableInputStream implements RangeReadable {

Re: [PR] AWS: Fix S3InputStream retry policy [iceberg]

2024-10-16 Thread via GitHub
edgarRd commented on code in PR #11335: URL: https://github.com/apache/iceberg/pull/11335#discussion_r1803882204 ## aws/src/test/java/org/apache/iceberg/aws/s3/TestFlakyS3InputStream.java: ## @@ -49,10 +51,29 @@ public class TestFlakyS3InputStream extends TestS3InputStream {

Re: [PR] AWS: Fix S3InputStream retry policy [iceberg]

2024-10-16 Thread via GitHub
amogh-jahagirdar commented on code in PR #11335: URL: https://github.com/apache/iceberg/pull/11335#discussion_r1803860896 ## aws/src/test/java/org/apache/iceberg/aws/s3/TestFlakyS3InputStream.java: ## @@ -49,10 +51,29 @@ public class TestFlakyS3InputStream extends TestS3Input

Re: [PR] AWS: Fix S3InputStream retry policy [iceberg]

2024-10-16 Thread via GitHub
amogh-jahagirdar commented on code in PR #11335: URL: https://github.com/apache/iceberg/pull/11335#discussion_r1803860896 ## aws/src/test/java/org/apache/iceberg/aws/s3/TestFlakyS3InputStream.java: ## @@ -49,10 +51,29 @@ public class TestFlakyS3InputStream extends TestS3Input

Re: [PR] Support changelog scan for table with delete files [iceberg]

2024-10-16 Thread via GitHub
aokolnychyi commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2418069515 What I wrote above is more complicated, we need to understand if that complexity will be justified. I am not sure yet. -- This is an automated message from the Apache Git Service.

Re: [PR] AWS: Fix S3InputStream retry policy [iceberg]

2024-10-16 Thread via GitHub
amogh-jahagirdar commented on code in PR #11335: URL: https://github.com/apache/iceberg/pull/11335#discussion_r1803860090 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3InputStream.java: ## @@ -66,10 +70,14 @@ class S3InputStream extends SeekableInputStream implements RangeRe

Re: [PR] Updating SparkScan to only read Apache DataSketches [iceberg]

2024-10-16 Thread via GitHub
RussellSpitzer commented on PR #11035: URL: https://github.com/apache/iceberg/pull/11035#issuecomment-2418066570 Thanks @jeesou for the PR, @aokolnychyi , @karuppayya , @huaxingao , @guykhazma all for reviewing. -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Support changelog scan for table with delete files [iceberg]

2024-10-16 Thread via GitHub
aokolnychyi commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2418068303 I went through some of my old notes, which we should discuss. We have the following tasks right now: - `AddedRowsScanTask` (added data file + deletes that happened within th

Re: [I] Stale column stats getting reported when reading puffin files generated by Presto with Spark engine [iceberg]

2024-10-16 Thread via GitHub
RussellSpitzer closed issue #11034: Stale column stats getting reported when reading puffin files generated by Presto with Spark engine URL: https://github.com/apache/iceberg/issues/11034 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Updating SparkScan to only read Apache DataSketches [iceberg]

2024-10-16 Thread via GitHub
RussellSpitzer merged PR #11035: URL: https://github.com/apache/iceberg/pull/11035 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] IO Implementation using Go CDK [iceberg-go]

2024-10-16 Thread via GitHub
dwilson1988 commented on PR #176: URL: https://github.com/apache/iceberg-go/pull/176#issuecomment-2418063681 Understood! I'll dig into your last question and get back to you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] IO Implementation using Go CDK [iceberg-go]

2024-10-16 Thread via GitHub
loicalleyne commented on PR #176: URL: https://github.com/apache/iceberg-go/pull/176#issuecomment-2418061193 My goal today was just to "get something on paper" to move this forward since the other PR has been stalled since July, I used the other PR as a starting point so I mostly followed t

Re: [PR] AWS: Introduce opt-in S3LocationProvider which is optimized for S3 performance [iceberg]

2024-10-16 Thread via GitHub
ookumuso commented on PR #2: URL: https://github.com/apache/iceberg/pull/2#issuecomment-2418059649 > @ookumuso a couple small remaining comments: > > 1. It looks like we're only using 20 bits, not 24 like in the description > > 2. The aws.md docs don't reflect t

Re: [PR] Arrow: add support for null vectors [iceberg]

2024-10-16 Thread via GitHub
RussellSpitzer commented on code in PR #10953: URL: https://github.com/apache/iceberg/pull/10953#discussion_r1803852642 ## arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorHolder.java: ## @@ -140,12 +141,21 @@ public static class ConstantVectorHolder extends Vector

Re: [PR] IO Implementation using Go CDK [iceberg-go]

2024-10-16 Thread via GitHub
dwilson1988 commented on PR #176: URL: https://github.com/apache/iceberg-go/pull/176#issuecomment-2418039810 @loicalleyne - happy to take a look. We use this internally in some of our software with Parquet and implemented a ReaderAt. I'll do a more thorough review when I get a chance, but

Re: [PR] Arrow: add support for null vectors [iceberg]

2024-10-16 Thread via GitHub
RussellSpitzer commented on code in PR #10953: URL: https://github.com/apache/iceberg/pull/10953#discussion_r1803845635 ## arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorHolder.java: ## @@ -140,12 +141,21 @@ public static class ConstantVectorHolder extends Vector

Re: [PR] IO Implementation using Go CDK [iceberg-go]

2024-10-16 Thread via GitHub
loicalleyne commented on PR #176: URL: https://github.com/apache/iceberg-go/pull/176#issuecomment-2418031722 @dwilson1988 I saw your note about wanting to work on the CDK features, if you're able to provide some feedback that would be great. -- This is an automated message from the Apache

Re: [PR] AWS: Introduce opt-in S3LocationProvider which is optimized for S3 performance [iceberg]

2024-10-16 Thread via GitHub
danielcweeks commented on PR #2: URL: https://github.com/apache/iceberg/pull/2#issuecomment-2418015319 @ookumuso a couple small remaining comments: 1. It looks like we're only using 20 bits, not 24 like in the description 2. The aws.md docs don't reflect the updated pathing

Re: [PR] Spark: support rewrite on specified target branch [iceberg]

2024-10-16 Thread via GitHub
jackye1995 commented on PR #8797: URL: https://github.com/apache/iceberg/pull/8797#issuecomment-2417996073 also @amitgilad3 @zinking -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Spark: support rewrite on specified target branch [iceberg]

2024-10-16 Thread via GitHub
jackye1995 commented on PR #8797: URL: https://github.com/apache/iceberg/pull/8797#issuecomment-2417995218 looks like this did not get much attention, @amit-cloudinary I did another pass, let me know if you can make the updates -- This is an automated message from the Apache Git Service.

Re: [PR] AWS: Fix S3inputstream retry policy [iceberg]

2024-10-16 Thread via GitHub
edgarRd commented on PR #11335: URL: https://github.com/apache/iceberg/pull/11335#issuecomment-2417977467 @amogh-jahagirdar @jackye1995 @danielcweeks PTAL. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] AWS: Fix S3inputstream retry policy [iceberg]

2024-10-16 Thread via GitHub
edgarRd commented on PR #11335: URL: https://github.com/apache/iceberg/pull/11335#issuecomment-2417977679 cc @Parth-Brahmbhatt -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

[PR] AWS: Fix S3inputstream retry policy [iceberg]

2024-10-16 Thread via GitHub
edgarRd opened a new pull request, #11335: URL: https://github.com/apache/iceberg/pull/11335 The retry policy for `S3InputStream` reads introduced in https://github.com/apache/iceberg/commit/c0d73f4ef5c16401bdfd62e1745faf2fbbf62177 is not actually re-opening the stream on each retry attempt

Re: [I] Remove old metadata files [iceberg-python]

2024-10-16 Thread via GitHub
sungwy commented on issue #1199: URL: https://github.com/apache/iceberg-python/issues/1199#issuecomment-2417968905 Awesome! Just assigned this to you @kaushiksrini -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Support changelog scan for table with delete files [iceberg]

2024-10-16 Thread via GitHub
wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1803801850 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -63,33 +61,39 @@ protected CloseableIterable doPlanFiles( return CloseableIter

Re: [PR] Spark: Add RewriteTablePath action interface [iceberg]

2024-10-16 Thread via GitHub
szehon-ho commented on code in PR #10920: URL: https://github.com/apache/iceberg/pull/10920#discussion_r1803788034 ## api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java: ## @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

Re: [PR] Spark: Add RewriteTablePath action interface [iceberg]

2024-10-16 Thread via GitHub
szehon-ho commented on code in PR #10920: URL: https://github.com/apache/iceberg/pull/10920#discussion_r1803788034 ## api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java: ## @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

Re: [PR] Spark: support rewrite on specified target branch [iceberg]

2024-10-16 Thread via GitHub
jackye1995 commented on code in PR #8797: URL: https://github.com/apache/iceberg/pull/8797#discussion_r1803785511 ## api/src/main/java/org/apache/iceberg/actions/RewriteDataFiles.java: ## @@ -171,6 +171,16 @@ default RewriteDataFiles zOrder(String... columns) { */ Rewrite

Re: [PR] Spark: support rewrite on specified target branch [iceberg]

2024-10-16 Thread via GitHub
jackye1995 commented on code in PR #8797: URL: https://github.com/apache/iceberg/pull/8797#discussion_r180389 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteDataFilesProcedure.java: ## @@ -134,6 +138,17 @@ private RewriteDataFiles checkAndApplyF

Re: [PR] Spark: support rewrite on specified target branch [iceberg]

2024-10-16 Thread via GitHub
jackye1995 commented on code in PR #8797: URL: https://github.com/apache/iceberg/pull/8797#discussion_r1803775302 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteDataFilesProcedure.java: ## @@ -134,6 +138,17 @@ private RewriteDataFiles checkAndApplyF

Re: [PR] Spark: support rewrite on specified target branch [iceberg]

2024-10-16 Thread via GitHub
jackye1995 commented on code in PR #8797: URL: https://github.com/apache/iceberg/pull/8797#discussion_r180389 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteDataFilesProcedure.java: ## @@ -134,6 +138,17 @@ private RewriteDataFiles checkAndApplyF

Re: [PR] Spark: support rewrite on specified target branch [iceberg]

2024-10-16 Thread via GitHub
jackye1995 commented on code in PR #8797: URL: https://github.com/apache/iceberg/pull/8797#discussion_r1803777014 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/Spark3Util.java: ## @@ -254,6 +258,15 @@ private static void apply(UpdateSchema pendingUpdate, TableChang

Re: [PR] Spark: support rewrite on specified target branch [iceberg]

2024-10-16 Thread via GitHub
jackye1995 commented on code in PR #8797: URL: https://github.com/apache/iceberg/pull/8797#discussion_r1803775302 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteDataFilesProcedure.java: ## @@ -134,6 +138,17 @@ private RewriteDataFiles checkAndApplyF

Re: [PR] Aliyun: Remove spring-boot dependency [iceberg]

2024-10-16 Thread via GitHub
jbonofre commented on code in PR #11291: URL: https://github.com/apache/iceberg/pull/11291#discussion_r1803771650 ## aliyun/src/test/java/org/apache/iceberg/aliyun/oss/mock/AliyunOSSMockLocalStore.java: ## @@ -121,8 +116,9 @@ void deleteBucket(String bucketName) throws IOExcepti

Re: [PR] Spark: support rewrite on specified target branch [iceberg]

2024-10-16 Thread via GitHub
jackye1995 commented on code in PR #8797: URL: https://github.com/apache/iceberg/pull/8797#discussion_r1803766282 ## api/src/main/java/org/apache/iceberg/actions/RewriteDataFiles.java: ## @@ -171,6 +171,16 @@ default RewriteDataFiles zOrder(String... columns) { */ Rewrite

[PR] IO Implementation using Go CDK [iceberg-go]

2024-10-16 Thread via GitHub
loicalleyne opened a new pull request, #176: URL: https://github.com/apache/iceberg-go/pull/176 Extends PR [#111](https://github.com/apache/iceberg-go/pull/111) Implements [#92](https://github.com/apache/iceberg-go/issues/92). The Go CDK has well-maintained implementations for accessi

Re: [I] Remove old metadata files [iceberg-python]

2024-10-16 Thread via GitHub
kaushiksrini commented on issue #1199: URL: https://github.com/apache/iceberg-python/issues/1199#issuecomment-2417893890 hey @sungwy, i'd be interested in implementing this feature! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [PR] Feature: Write to branches [iceberg-python]

2024-10-16 Thread via GitHub
vinjai commented on PR #941: URL: https://github.com/apache/iceberg-python/pull/941#issuecomment-2417891570 Fixed another bug. Writes with same name but different ref types Please review whenever you get some time. -- This is an automated message from the Apache Git Service. To respo

  1   2   >