Re: [I] Iceberg Materialized Views [iceberg]

2024-11-07 Thread via GitHub
JanKaul commented on issue #10043: URL: https://github.com/apache/iceberg/issues/10043#issuecomment-2463977652 I've created a PR in #11041 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Core: Support DVs in DeleteLoader [iceberg]

2024-11-07 Thread via GitHub
aokolnychyi commented on code in PR #11481: URL: https://github.com/apache/iceberg/pull/11481#discussion_r1833847228 ## data/src/main/java/org/apache/iceberg/data/BaseDeleteLoader.java: ## @@ -259,4 +284,46 @@ private long estimateEqDeletesSize(DeleteFile deleteFile, Schema pro

[I] When write.object-storage.enabled=true, it is difficult to gather information for individual partition of partitioned tables [iceberg]

2024-11-07 Thread via GitHub
borderlayout opened a new issue, #11488: URL: https://github.com/apache/iceberg/issues/11488 ### Feature Request / Improvement Hi all: When using Amazon S3 object storage with Iceberg, there can be a throttling issue for the same path. By setting the parameter write.object-sto

Re: [PR] Core: Support DVs in DeleteLoader [iceberg]

2024-11-07 Thread via GitHub
aokolnychyi commented on code in PR #11486: URL: https://github.com/apache/iceberg/pull/11486#discussion_r1833801076 ## data/src/main/java/org/apache/iceberg/data/DeleteLoader.java: ## @@ -42,4 +42,14 @@ public interface DeleteLoader { * @return a position delete index for t

Re: [PR] Core: Support DVs in DeleteLoader [iceberg]

2024-11-07 Thread via GitHub
aokolnychyi closed pull request #11486: Core: Support DVs in DeleteLoader URL: https://github.com/apache/iceberg/pull/11486 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] Docs: Fix format of verifying release candidate with Flink [iceberg]

2024-11-07 Thread via GitHub
pvary commented on PR #11487: URL: https://github.com/apache/iceberg/pull/11487#issuecomment-2463887940 Thanks @manuzhang for the fix -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Docs: Fix format of verifying release candidate with Flink [iceberg]

2024-11-07 Thread via GitHub
pvary merged PR #11487: URL: https://github.com/apache/iceberg/pull/11487 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Flink: Port #11144 to v1.19 [iceberg]

2024-11-07 Thread via GitHub
pvary commented on PR #11473: URL: https://github.com/apache/iceberg/pull/11473#issuecomment-2463881089 Thanks for the review @Fokko and @stevenzwu -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Flink: Port #11144 to v1.19 [iceberg]

2024-11-07 Thread via GitHub
pvary merged PR #11473: URL: https://github.com/apache/iceberg/pull/11473 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Flink-1.19: Fix the file offset mismatch when Flink reader first seek… [iceberg]

2024-11-07 Thread via GitHub
pvary commented on PR #10567: URL: https://github.com/apache/iceberg/pull/10567#issuecomment-2463879935 @stevenzwu: any idea for testing? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [I] Flink: Maintenance - DeleteOrphanFiles [iceberg]

2024-11-07 Thread via GitHub
pvary commented on issue #10306: URL: https://github.com/apache/iceberg/issues/10306#issuecomment-2463878408 Needed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [I] Flink: Maintenance - RewriteManifestFiles [iceberg]

2024-11-07 Thread via GitHub
pvary commented on issue #10305: URL: https://github.com/apache/iceberg/issues/10305#issuecomment-2463877853 Needed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [I] Flink: Maintenance - CommitConverter [iceberg]

2024-11-07 Thread via GitHub
pvary commented on issue #10302: URL: https://github.com/apache/iceberg/issues/10302#issuecomment-2463876090 We need this. Cc @rodmeneses -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [I] Flink: Maintenance - RewriteDataFiles [iceberg]

2024-11-07 Thread via GitHub
pvary commented on issue #10303: URL: https://github.com/apache/iceberg/issues/10303#issuecomment-2463876797 In progress -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] Serialization of the org.apache.iceberg.io.WriteResult class. [iceberg]

2024-11-07 Thread via GitHub
pvary commented on issue #10710: URL: https://github.com/apache/iceberg/issues/10710#issuecomment-2463865540 I think we could add the typeinformation when defining the streams -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] Spark: support hilbert curve when rewrite [iceberg]

2024-11-07 Thread via GitHub
raghavg7796 commented on PR #5824: URL: https://github.com/apache/iceberg/pull/5824#issuecomment-2463839754 Hey! I was interested in hilbert curve based sorting with Iceberg, however seems like this PR was abandoned. Is there any plan to support hilbert curve sorting in Iceberg? -- This

Re: [PR] Core: Make metrics reporter serializable (alternative impl) [iceberg]

2024-11-07 Thread via GitHub
manuzhang commented on PR #8032: URL: https://github.com/apache/iceberg/pull/8032#issuecomment-2463801021 @nastra are you still working on this? I want to make `BaseTable#metricsReporter` public in https://github.com/apache/iceberg/pull/11340/ as well. -- This is an automated message fro

[PR] Introduce `assign_fresh_ids` and allow skipping fresh assignment of IDs on table creation [iceberg-python]

2024-11-07 Thread via GitHub
sungwy opened a new pull request, #1304: URL: https://github.com/apache/iceberg-python/pull/1304 Implements: https://github.com/apache/iceberg-python/issues/1284 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Spark 3.5: Display write metrics on SQL UI [iceberg]

2024-11-07 Thread via GitHub
manuzhang commented on code in PR #11340: URL: https://github.com/apache/iceberg/pull/11340#discussion_r1833716157 ## spark/v3.5/spark/src/main/scala/org/apache/spark/sql/MetricsUtils.scala: ## @@ -0,0 +1,39 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Core: Fix caching table with metadata table names [iceberg]

2024-11-07 Thread via GitHub
manuzhang commented on PR #11123: URL: https://github.com/apache/iceberg/pull/11123#issuecomment-2463773057 @nastra @ajantha-bhat @RussellSpitzer Please take a look when you find time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] User ID information in Iceberg Table's snapshot [iceberg]

2024-11-07 Thread via GitHub
ArijitSinghEDA commented on issue #11474: URL: https://github.com/apache/iceberg/issues/11474#issuecomment-2463754891 Thank you @RussellSpitzer! I saw the PR made by @szehon-ho It is what I was looking for, but is it only limited to `INSERT` commands, or does it work with `UPDATE` an

Re: [PR] AWS: Enable RetryMode for AWS KMS client [iceberg]

2024-11-07 Thread via GitHub
hsiang-c commented on PR #11420: URL: https://github.com/apache/iceberg/pull/11420#issuecomment-2463714791 Thanks for your feedback @danielcweeks Please take a look again. If you still have time, I have another PR for you https://github.com/apache/iceberg/pull/11321 -- This is an a

Re: [PR] Remove deprecated `datetime` functions [iceberg-python]

2024-11-07 Thread via GitHub
kevinjqliu commented on PR #1134: URL: https://github.com/apache/iceberg-python/pull/1134#issuecomment-2463666573 https://github.com/search?q=repo%3Aapache%2Ficeberg-python%20utcfromtimestamp&type=code there's a few instances of `utcfromtimestamp` -- This is an automated message from t

Re: [PR] Spark 3.5: Iceberg parser should passthrough unsupported procedure to delegate [iceberg]

2024-11-07 Thread via GitHub
pan3793 commented on code in PR #11480: URL: https://github.com/apache/iceberg/pull/11480#discussion_r1833624554 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSparkSqlExtensionsParser.scala: ## @@ -151,6 +155,11 @@ class Ice

Re: [PR] Spark 3.5: Iceberg parser should passthrough unsupported procedure to delegate [iceberg]

2024-11-07 Thread via GitHub
pan3793 commented on code in PR #11480: URL: https://github.com/apache/iceberg/pull/11480#discussion_r1833619653 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java: ## @@ -68,11 +68,29 @@ public static void stopSpark() {

Re: [PR] Docs: Fix verifying release candidate with Spark and Flink [iceberg]

2024-11-07 Thread via GitHub
manuzhang commented on code in PR #11461: URL: https://github.com/apache/iceberg/pull/11461#discussion_r1833564722 ## site/docs/how-to-release.md: ## @@ -435,13 +435,13 @@ spark-shell \ To verify using Flink, start a Flink SQL Client with the following command: ```bash -wget

[PR] Docs: Fix format of verifying release candidate with Flink [iceberg]

2024-11-07 Thread via GitHub
manuzhang opened a new pull request, #11487: URL: https://github.com/apache/iceberg/pull/11487 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [I] Document `advisory-partition-size` configuration for Spark's Adaptive Query Execution [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] closed issue #10051: Document `advisory-partition-size` configuration for Spark's Adaptive Query Execution URL: https://github.com/apache/iceberg/issues/10051 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] Core: improve DefaultErrorHandler message for unhandled codes [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] commented on PR #10640: URL: https://github.com/apache/iceberg/pull/10640#issuecomment-2463470558 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Fixed an incorrect example [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] commented on PR #10627: URL: https://github.com/apache/iceberg/pull/10627#issuecomment-2463470501 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Relax client.credentials-provider requirements [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] commented on PR #10604: URL: https://github.com/apache/iceberg/pull/10604#issuecomment-2463470442 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Let `./gradlew clean` clean everything [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] commented on PR #10601: URL: https://github.com/apache/iceberg/pull/10601#issuecomment-2463470420 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Speed up Spark + Flink unit test execution [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] commented on PR #10581: URL: https://github.com/apache/iceberg/pull/10581#issuecomment-2463470396 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Support create multiple element ns together for nessie [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] commented on PR #10630: URL: https://github.com/apache/iceberg/pull/10630#issuecomment-2463470526 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Drop table purge issue for parquet tables with SparkSessionCatalog [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] closed pull request #10158: Drop table purge issue for parquet tables with SparkSessionCatalog URL: https://github.com/apache/iceberg/pull/10158 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] Metadata file is not getting created when Iceberg table is created using Hive with catalog as GlueCatalog [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] commented on issue #10025: URL: https://github.com/apache/iceberg/issues/10025#issuecomment-2463469924 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] closed pull request #10151: Flink: FlinkFileIO implementation URL: https://github.com/apache/iceberg/pull/10151 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [I] Add capability to support an external index for additional file filtering [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] commented on issue #10083: URL: https://github.com/apache/iceberg/issues/10083#issuecomment-2463470029 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Weird behavior struct fields in Spark entries metadata table [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] closed issue #10044: Weird behavior struct fields in Spark entries metadata table URL: https://github.com/apache/iceberg/issues/10044 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] Add capability to support an external index for additional file filtering [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] closed issue #10083: Add capability to support an external index for additional file filtering URL: https://github.com/apache/iceberg/issues/10083 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [I] OR condition does not leverage all parquet metadata (metrics, dictionary, bloom filter) causing inefficient queries [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] commented on issue #10029: URL: https://github.com/apache/iceberg/issues/10029#issuecomment-2463469950 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [PR] Ability to build for all Scala versions [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] commented on PR #10606: URL: https://github.com/apache/iceberg/pull/10606#issuecomment-2463470466 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] Weird behavior struct fields in Spark entries metadata table [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] commented on issue #10044: URL: https://github.com/apache/iceberg/issues/10044#issuecomment-2463469979 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [PR] Core: Fix possible proken in `Tasks.Builder.runSingleThreaded` [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] commented on PR #10613: URL: https://github.com/apache/iceberg/pull/10613#issuecomment-2463470480 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Spec: Add GCS and ADLS configuration to REST table load [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] commented on PR #10576: URL: https://github.com/apache/iceberg/pull/10576#issuecomment-2463470375 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] commented on PR #10151: URL: https://github.com/apache/iceberg/pull/10151#issuecomment-2463470121 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [I] iceberg does not allow selection of AWS profiles other than "default" [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] commented on issue #10085: URL: https://github.com/apache/iceberg/issues/10085#issuecomment-2463470055 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [PR] Drop table purge issue for parquet tables with SparkSessionCatalog [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] commented on PR #10158: URL: https://github.com/apache/iceberg/pull/10158#issuecomment-2463470137 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [I] iceberg does not allow selection of AWS profiles other than "default" [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] closed issue #10085: iceberg does not allow selection of AWS profiles other than "default" URL: https://github.com/apache/iceberg/issues/10085 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [I] Document `advisory-partition-size` configuration for Spark's Adaptive Query Execution [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] commented on issue #10051: URL: https://github.com/apache/iceberg/issues/10051#issuecomment-2463469994 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] OR condition does not leverage all parquet metadata (metrics, dictionary, bloom filter) causing inefficient queries [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] closed issue #10029: OR condition does not leverage all parquet metadata (metrics, dictionary, bloom filter) causing inefficient queries URL: https://github.com/apache/iceberg/issues/10029 -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [I] Metadata file is not getting created when Iceberg table is created using Hive with catalog as GlueCatalog [iceberg]

2024-11-07 Thread via GitHub
github-actions[bot] closed issue #10025: Metadata file is not getting created when Iceberg table is created using Hive with catalog as GlueCatalog URL: https://github.com/apache/iceberg/issues/10025 -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Add `view_exists` method to REST Catalog [iceberg-python]

2024-11-07 Thread via GitHub
shiv-io commented on PR #1242: URL: https://github.com/apache/iceberg-python/pull/1242#issuecomment-2463458916 @sungwy thanks for reviewing! Let me know if there's anything else. I noticed we don't yet have docs for `list_views` (#817) and `drop_views` (#820), btw. -- This is an automat

Re: [PR] Add `view_exists` method to REST Catalog [iceberg-python]

2024-11-07 Thread via GitHub
shiv-io commented on code in PR #1242: URL: https://github.com/apache/iceberg-python/pull/1242#discussion_r1833516291 ## tests/catalog/test_rest.py: ## @@ -450,6 +450,28 @@ def test_list_views_404(rest_mock: Mocker) -> None: assert "Namespace does not exist" in str(e.value)

Re: [PR] Remove deprecated `datetime` functions [iceberg-python]

2024-11-07 Thread via GitHub
hussein-awala commented on PR #1134: URL: https://github.com/apache/iceberg-python/pull/1134#issuecomment-2463392475 > Thanks for cleaning this up @hussein-awala Could you resolve the conflicts? Thanks! I did and looks like the usage of the deprecated method was removed by another PR

Re: [I] Support writing to a table with sort-order [iceberg-python]

2024-11-07 Thread via GitHub
vinjai commented on issue #271: URL: https://github.com/apache/iceberg-python/issues/271#issuecomment-2463354739 Hey @Fokko I'm not completely sure about the details of the Rust Extension implementation. Could you share any documents/proposals/issues that explain what we're aimi

Re: [PR] Feature: Write to branches [iceberg-python]

2024-11-07 Thread via GitHub
vinjai commented on PR #941: URL: https://github.com/apache/iceberg-python/pull/941#issuecomment-2463348171 Hey @kevinjqliu Did you get a chance to look at this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Core: Support DVs in DeleteLoader [iceberg]

2024-11-07 Thread via GitHub
danielcweeks commented on code in PR #11486: URL: https://github.com/apache/iceberg/pull/11486#discussion_r1833445653 ## data/src/main/java/org/apache/iceberg/data/DeleteLoader.java: ## @@ -42,4 +42,14 @@ public interface DeleteLoader { * @return a position delete index for

Re: [PR] Spark 3.5: Iceberg parser should passthrough unsupported procedure to delegate [iceberg]

2024-11-07 Thread via GitHub
huaxingao commented on code in PR #11480: URL: https://github.com/apache/iceberg/pull/11480#discussion_r1833437708 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSparkSqlExtensionsParser.scala: ## @@ -151,6 +155,11 @@ class I

Re: [PR] API: Add Variant data type [iceberg]

2024-11-07 Thread via GitHub
aihuaxu commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1833206108 ## api/src/main/java/org/apache/iceberg/transforms/Identity.java: ## @@ -38,6 +38,9 @@ class Identity implements Transform { */ @Deprecated public static I

Re: [PR] Core: Support DVs in DeleteLoader [iceberg]

2024-11-07 Thread via GitHub
aokolnychyi commented on PR #11481: URL: https://github.com/apache/iceberg/pull/11481#issuecomment-2463277936 @nastra @danielcweeks @jbonofre, what do you folks think about #11486 as an alternative implementation? Am I overcomplicating or is that one cleaner? -- This is an automated messa

Re: [PR] API: Add Variant data type [iceberg]

2024-11-07 Thread via GitHub
RussellSpitzer commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1833022565 ## api/src/main/java/org/apache/iceberg/transforms/Identity.java: ## @@ -38,6 +38,9 @@ class Identity implements Transform { */ @Deprecated public st

Re: [PR] API: Add Variant data type [iceberg]

2024-11-07 Thread via GitHub
aihuaxu commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1833199261 ## api/src/test/java/org/apache/iceberg/TestPartitionSpecValidation.java: ## @@ -312,4 +314,15 @@ public void testAddPartitionFieldsWithAndWithoutFieldIds() { a

Re: [I] Express REST server capabilities [iceberg]

2024-11-07 Thread via GitHub
danielcweeks commented on issue #10462: URL: https://github.com/apache/iceberg/issues/10462#issuecomment-2462985775 This has been addressed by #10928 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] Express REST server capabilities [iceberg]

2024-11-07 Thread via GitHub
danielcweeks closed issue #10462: Express REST server capabilities URL: https://github.com/apache/iceberg/issues/10462 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] REST: Docker file for Rest catalog adapter image [iceberg]

2024-11-07 Thread via GitHub
ajantha-bhat commented on code in PR #11283: URL: https://github.com/apache/iceberg/pull/11283#discussion_r1832629068 ## docker/iceberg-rest-adapter-image/README.md: ## @@ -0,0 +1,87 @@ + + +# Iceberg rest adapter image Review Comment: > A Technology Compatibility Kit (TCK)

Re: [PR] API: Add Variant data type [iceberg]

2024-11-07 Thread via GitHub
aihuaxu commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1833123566 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -412,6 +413,29 @@ public String toString() { } } + public static class VariantType extends

Re: [PR] Do not deprecate `botocore_session` [iceberg-python]

2024-11-07 Thread via GitHub
kevinjqliu merged PR #1300: URL: https://github.com/apache/iceberg-python/pull/1300 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [I] Do not deprecate Botocore Session in upcoming release (0.8) [iceberg-python]

2024-11-07 Thread via GitHub
kevinjqliu closed issue #1104: Do not deprecate Botocore Session in upcoming release (0.8) URL: https://github.com/apache/iceberg-python/issues/1104 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [I] Support dynamic overwrite [iceberg-python]

2024-11-07 Thread via GitHub
kevinjqliu commented on issue #1287: URL: https://github.com/apache/iceberg-python/issues/1287#issuecomment-2462831829 I see thanks for the explanation. When writing in fast append mode (DELETE+APPEND), it's possible to accidentally time travel to the DELETE snapshot and only see the delet

Re: [PR] AWS: Enable RetryMode for AWS KMS client [iceberg]

2024-11-07 Thread via GitHub
danielcweeks commented on PR #11420: URL: https://github.com/apache/iceberg/pull/11420#issuecomment-2462804351 > Is having too many options a concern to you? I'd like to learn more about your thoughts. I feel like the proliferation of options is not a good thing unless there's real v

Re: [PR] API: Add Variant data type [iceberg]

2024-11-07 Thread via GitHub
RussellSpitzer commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1833032148 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -1687,6 +1687,44 @@ public void testV3TimestampNanoTypeSupport() { 3); } +

Re: [PR] API: Add Variant data type [iceberg]

2024-11-07 Thread via GitHub
rdblue commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1832994460 ## api/src/main/java/org/apache/iceberg/transforms/Identity.java: ## @@ -38,6 +38,9 @@ class Identity implements Transform { */ @Deprecated public static Id

Re: [PR] API: Add Variant data type [iceberg]

2024-11-07 Thread via GitHub
RussellSpitzer commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1833032148 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -1687,6 +1687,44 @@ public void testV3TimestampNanoTypeSupport() { 3); } +

Re: [PR] Docs: Fix verifying release candidate with Spark and Flink [iceberg]

2024-11-07 Thread via GitHub
nastra merged PR #11461: URL: https://github.com/apache/iceberg/pull/11461 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] API: Add Variant data type [iceberg]

2024-11-07 Thread via GitHub
RussellSpitzer commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1833022565 ## api/src/main/java/org/apache/iceberg/transforms/Identity.java: ## @@ -38,6 +38,9 @@ class Identity implements Transform { */ @Deprecated public st

Re: [PR] Spec: Explicitly specify `operation` as a required field of Snapshot's `summary` field [iceberg]

2024-11-07 Thread via GitHub
RussellSpitzer commented on PR #11355: URL: https://github.com/apache/iceberg/pull/11355#issuecomment-2462688106 @sungwy Can you please rebase this? I think we are good to merge once the conflicts are resolved -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] API: Add Variant data type [iceberg]

2024-11-07 Thread via GitHub
rdblue commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1832990597 ## api/src/main/java/org/apache/iceberg/expressions/ExpressionUtil.java: ## @@ -562,7 +563,7 @@ private static String sanitize(Literal literal, long now, int today) {

Re: [PR] API: Add Variant data type [iceberg]

2024-11-07 Thread via GitHub
rdblue commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1833015207 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -1687,6 +1687,44 @@ public void testV3TimestampNanoTypeSupport() { 3); } + @Test

Re: [PR] REST: AuthManager API [iceberg]

2024-11-07 Thread via GitHub
adutra commented on PR #10753: URL: https://github.com/apache/iceberg/pull/10753#issuecomment-2462696503 I rebased this PR and squashed the commits because the commit history was getting big. I only retained 3 commits: 1. [26f5e5f](https://github.com/apache/iceberg/pull/10753/

Re: [PR] API: Add Variant data type [iceberg]

2024-11-07 Thread via GitHub
rdblue commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1833011700 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -412,6 +413,29 @@ public String toString() { } } + public static class VariantType extends P

Re: [PR] API: Add Variant data type [iceberg]

2024-11-07 Thread via GitHub
rdblue commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1833009142 ## api/src/test/java/org/apache/iceberg/TestPartitionSpecValidation.java: ## @@ -312,4 +314,15 @@ public void testAddPartitionFieldsWithAndWithoutFieldIds() { as

Re: [PR] API: Add Variant data type [iceberg]

2024-11-07 Thread via GitHub
rdblue commented on code in PR #11324: URL: https://github.com/apache/iceberg/pull/11324#discussion_r1833006470 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -412,6 +413,29 @@ public String toString() { } } + public static class VariantType extends P

Re: [I] What's the use of old metadata file, why not delete by default? [iceberg]

2024-11-07 Thread via GitHub
RussellSpitzer commented on issue #11206: URL: https://github.com/apache/iceberg/issues/11206#issuecomment-2462668198 Mostly for cases in which something has gone very very wrong. Snapshots don't take into account DML changes in schema or partition spec so it's impossible to fully revert st

Re: [I] Long-running Spark rewrite Files Action may lead to OutOfMemoryError [iceberg]

2024-11-07 Thread via GitHub
RussellSpitzer commented on issue #11277: URL: https://github.com/apache/iceberg/issues/11277#issuecomment-2462663665 Do you have any other details on this? Possibly a way to test this deterministically? I think if we had more information about the length of the job, the amount of work, and

Re: [I] [Spark] Identity partition on required column generates nullable partition tuple in manifest file [iceberg]

2024-11-07 Thread via GitHub
RussellSpitzer commented on issue #11300: URL: https://github.com/apache/iceberg/issues/11300#issuecomment-2462655068 I don't know if this is really a bug since there is no behavior change really associated with it. I believe this is a leftover from V1 where we were unable to ever remove pa

Re: [I] User ID information in Iceberg Table's snapshot [iceberg]

2024-11-07 Thread via GitHub
RussellSpitzer commented on issue #11474: URL: https://github.com/apache/iceberg/issues/11474#issuecomment-2462632980 There has been a long standing issue in OSS Spark to have an API for passing through write options during SparkSQL. I believe it is tackled in Spark 4.0.X I ThinK @szehon-ho

Re: [I] Arrow - Removed unused vectorized code readers [iceberg]

2024-11-07 Thread via GitHub
RussellSpitzer closed issue #11370: Arrow - Removed unused vectorized code readers URL: https://github.com/apache/iceberg/issues/11370 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [I] How does client use hadoopcatlog to read the iceberg table writen by hivecatalog? [iceberg]

2024-11-07 Thread via GitHub
RussellSpitzer commented on issue #11375: URL: https://github.com/apache/iceberg/issues/11375#issuecomment-2462641625 I strongly recommend not doing this, using different catalogs with the same table can produce inconsistent results. @manuzhang is correct though that theoretically it would

[PR] Core, Flink, Spark: Verify maintenance procedures with DVs [iceberg]

2024-11-07 Thread via GitHub
nastra opened a new pull request, #11485: URL: https://github.com/apache/iceberg/pull/11485 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

Re: [I] Spark aggreation by partition could use metadata files [iceberg]

2024-11-07 Thread via GitHub
RussellSpitzer commented on issue #11394: URL: https://github.com/apache/iceberg/issues/11394#issuecomment-2462637697 Spark does have pushdowns for this. I don't know what Spark version you are using but aggregate pushdown is implemented by @huaxingao -- This is an automated message from

Re: [I] Compute column stats incrementally [iceberg]

2024-11-07 Thread via GitHub
RussellSpitzer commented on issue #11475: URL: https://github.com/apache/iceberg/issues/11475#issuecomment-2462630598 Not yet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Docs: Fix verifying release candidate with Spark and Flink [iceberg]

2024-11-07 Thread via GitHub
manuzhang commented on code in PR #11461: URL: https://github.com/apache/iceberg/pull/11461#discussion_r1832947268 ## site/docs/how-to-release.md: ## @@ -435,13 +435,13 @@ spark-shell \ To verify using Flink, start a Flink SQL Client with the following command: ```bash -wget

Re: [PR] Docs: Fix verifying release candidate with Spark and Flink [iceberg]

2024-11-07 Thread via GitHub
nastra commented on code in PR #11461: URL: https://github.com/apache/iceberg/pull/11461#discussion_r1832940493 ## site/docs/how-to-release.md: ## @@ -435,13 +435,13 @@ spark-shell \ To verify using Flink, start a Flink SQL Client with the following command: ```bash -wget ${

Re: [PR] Spark 3.5: Iceberg parser should passthrough unsupported procedure to delegate [iceberg]

2024-11-07 Thread via GitHub
RussellSpitzer commented on code in PR #11480: URL: https://github.com/apache/iceberg/pull/11480#discussion_r1832876547 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSparkSqlExtensionsParser.scala: ## @@ -151,6 +155,11 @@ cl

Re: [PR] Spark 3.5: Iceberg parser should passthrough unsupported procedure to delegate [iceberg]

2024-11-07 Thread via GitHub
RussellSpitzer commented on code in PR #11480: URL: https://github.com/apache/iceberg/pull/11480#discussion_r1832879050 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java: ## @@ -68,11 +68,29 @@ public static void stopSp

Re: [I] PyIceberg Cookbook [iceberg-python]

2024-11-07 Thread via GitHub
francocalvo commented on issue #1201: URL: https://github.com/apache/iceberg-python/issues/1201#issuecomment-2462254614 Thank you for the prompt answer! > The writes work with pyarrow tables and dataframe. Im don't think you need to pass as string Yes, what I mean is when I n

Re: [I] MinIO + Spark + hive metadata + iceberg format [iceberg]

2024-11-07 Thread via GitHub
xhuyvn commented on issue #10222: URL: https://github.com/apache/iceberg/issues/10222#issuecomment-2462560292 > Hi, have you found the solution yet? i have the same problem when using hive 4.0 with minio: > > pyspark.errors.exceptions.captured.AnalysisException: org.apache.hadoop.hiv

Re: [PR] Core: Support DVs in DeleteLoader [iceberg]

2024-11-07 Thread via GitHub
jbonofre commented on code in PR #11481: URL: https://github.com/apache/iceberg/pull/11481#discussion_r1832918196 ## data/src/main/java/org/apache/iceberg/data/BaseDeleteLoader.java: ## @@ -259,4 +284,46 @@ private long estimateEqDeletesSize(DeleteFile deleteFile, Schema projec

Re: [PR] API: Removes Explicit Parameterization of Schema Tests [iceberg]

2024-11-07 Thread via GitHub
RussellSpitzer merged PR #11444: URL: https://github.com/apache/iceberg/pull/11444 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

  1   2   >