Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-16 Thread via GitHub
bennychow commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1604387531 ## format/materialized-view-spec.md: ## @@ -0,0 +1,132 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool t

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-16 Thread via GitHub
wmoustafa commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1604351916 ## format/materialized-view-spec.md: ## @@ -0,0 +1,132 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool t

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-16 Thread via GitHub
wmoustafa commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1604351916 ## format/materialized-view-spec.md: ## @@ -0,0 +1,132 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool t

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-16 Thread via GitHub
wmoustafa commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1604351916 ## format/materialized-view-spec.md: ## @@ -0,0 +1,132 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool t

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-16 Thread via GitHub
wmoustafa commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1604351916 ## format/materialized-view-spec.md: ## @@ -0,0 +1,132 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool t

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-16 Thread via GitHub
wmoustafa commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1604351916 ## format/materialized-view-spec.md: ## @@ -0,0 +1,132 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool t

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-16 Thread via GitHub
bennychow commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1604342841 ## format/materialized-view-spec.md: ## @@ -0,0 +1,132 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool t

[PR] Docs: update Apache Doris dead links [iceberg]

2024-05-16 Thread via GitHub
vinlee19 opened a new pull request, #10344: URL: https://github.com/apache/iceberg/pull/10344 "dev" refers to the documentation for the master branch of Apache Doris. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Core: Introduce AuthConfig [iceberg]

2024-05-16 Thread via GitHub
dramaticlly commented on code in PR #10161: URL: https://github.com/apache/iceberg/pull/10161#discussion_r1604193353 ## core/src/main/java/org/apache/iceberg/rest/auth/AuthConfig.java: ## @@ -0,0 +1,68 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-16 Thread via GitHub
wmoustafa commented on PR #10280: URL: https://github.com/apache/iceberg/pull/10280#issuecomment-2116364483 > What if we keep this MV spec page, but instead of describing the fields here, we make these pointer to the corresponding table and view spec sections? So this MV spec page contains

Re: [PR] Spark: Add SparkSQLProperty to control split-size [iceberg]

2024-05-16 Thread via GitHub
sumedhsakdeo commented on PR #10336: URL: https://github.com/apache/iceberg/pull/10336#issuecomment-2116335906 Thanks Shardul for taking a look. Appreciate your review Anton and Amogh. Also adding @wmoustafa! -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-16 Thread via GitHub
wmoustafa commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1604144985 ## format/materialized-view-spec.md: ## @@ -0,0 +1,132 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool t

[PR] Build: Bump boto3 from 1.34.69 to 1.34.106 [iceberg-python]

2024-05-16 Thread via GitHub
dependabot[bot] opened a new pull request, #749: URL: https://github.com/apache/iceberg-python/pull/749 Bumps [boto3](https://github.com/boto/boto3) from 1.34.69 to 1.34.106. Changelog Sourced from https://github.com/boto/boto3/blob/develop/CHANGELOG.rst";>boto3's changelog.

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-16 Thread via GitHub
bennychow commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1604061611 ## format/materialized-view-spec.md: ## @@ -0,0 +1,132 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool t

Re: [I] Aws Glue error for append data [iceberg-python]

2024-05-16 Thread via GitHub
apersilva commented on issue #738: URL: https://github.com/apache/iceberg-python/issues/738#issuecomment-2116238350 It´s work, thanks a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-16 Thread via GitHub
jackye1995 commented on PR #10280: URL: https://github.com/apache/iceberg/pull/10280#issuecomment-2116228474 I have a similar concern with @stevenzwu in the devlist: > With the agreed model of separate view and storage table, I am wondering if a separate materialized view spec page is

Re: [PR] Prevent StructLikeWrapper#equals from throwing an exception [iceberg]

2024-05-16 Thread via GitHub
alexjo2144 closed pull request #5157: Prevent StructLikeWrapper#equals from throwing an exception URL: https://github.com/apache/iceberg/pull/5157 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-16 Thread via GitHub
jackye1995 commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1604060355 ## format/materialized-view-spec.md: ## @@ -0,0 +1,132 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool

Re: [PR] [Spec] Add Iceberg Materialized View Spec [iceberg]

2024-05-16 Thread via GitHub
jackye1995 commented on code in PR #10280: URL: https://github.com/apache/iceberg/pull/10280#discussion_r1604060355 ## format/materialized-view-spec.md: ## @@ -0,0 +1,132 @@ + + +# Iceberg Materialized View Spec + +## Background and Motivation +Iceberg views are a powerful tool

Re: [I] Aws Glue error for append data [iceberg-python]

2024-05-16 Thread via GitHub
ndrluis commented on issue #738: URL: https://github.com/apache/iceberg-python/issues/738#issuecomment-2116167599 Sorry, I double-checked the Java implementation, and it's correct on the Python side. @apersilva, for your case, I believe you need to do something like this: ```

Re: [PR] Support getting a snapshot right before the given timestamp [iceberg-python]

2024-05-16 Thread via GitHub
syun64 commented on PR #748: URL: https://github.com/apache/iceberg-python/pull/748#issuecomment-2115989197 Hi @ndrluis thank you for flagging this! That PR went under my radar, and I'm excited to see a changelog scanning feature being implemented already on PyIceberg. As for the que

Re: [PR] Support getting a snapshot right before the given timestamp [iceberg-python]

2024-05-16 Thread via GitHub
ndrluis commented on PR #748: URL: https://github.com/apache/iceberg-python/pull/748#issuecomment-2115935825 Hello @chinmay-bhat, I noticed that you are implementing the ancestors_of method, and we have another pull request (#533) that is implementing the same behavior in another pla

Re: [PR] [FEAT]register table using iceberg metadata file via pyiceberg [iceberg-python]

2024-05-16 Thread via GitHub
MehulBatra commented on code in PR #711: URL: https://github.com/apache/iceberg-python/pull/711#discussion_r1603214209 ## tests/catalog/test_glue.py: ## @@ -848,3 +848,17 @@ def test_table_exists( assert test_catalog.table_exists(identifier) is True # Act and Assert fo

Re: [I] Spark: Support UUID partitioned tables [iceberg]

2024-05-16 Thread via GitHub
nastra closed issue #8247: Spark: Support UUID partitioned tables URL: https://github.com/apache/iceberg/issues/8247 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

Re: [PR] Spark: Fix issue when partitioning by UUID [iceberg]

2024-05-16 Thread via GitHub
nastra merged PR #8250: URL: https://github.com/apache/iceberg/pull/8250 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Support getting a snapshot right before the given timestamp [iceberg-python]

2024-05-16 Thread via GitHub
chinmay-bhat commented on code in PR #748: URL: https://github.com/apache/iceberg-python/pull/748#discussion_r1603682743 ## pyiceberg/table/metadata.py: ## @@ -230,6 +230,14 @@ def snapshot_by_id(self, snapshot_id: int) -> Optional[Snapshot]: """Get the snapshot by sna

Re: [PR] Spark: Fix issue when partitioning by UUID [iceberg]

2024-05-16 Thread via GitHub
nastra commented on PR #8250: URL: https://github.com/apache/iceberg/pull/8250#issuecomment-2115675915 thanks for the reviews @hililiwei, @singhpk234, @amogh-jahagirdar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Core: Remove unused ManifestGroup#filterManifests(Predicate) [iceberg]

2024-05-16 Thread via GitHub
amogh-jahagirdar merged PR #10339: URL: https://github.com/apache/iceberg/pull/10339 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Core: Remove unused ManifestGroup#filterManifests(Predicate) [iceberg]

2024-05-16 Thread via GitHub
amogh-jahagirdar commented on PR #10339: URL: https://github.com/apache/iceberg/pull/10339#issuecomment-2115673471 Thanks for the review! I'll go ahead and merge -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[PR] [WIP] Support In and notIn operators in ParquetFilters.ConvertFilterToParquet [iceberg]

2024-05-16 Thread via GitHub
sririshindra opened a new pull request, #10341: URL: https://github.com/apache/iceberg/pull/10341 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [PR] Spark: Fix issue when partitioning by UUID [iceberg]

2024-05-16 Thread via GitHub
nastra commented on code in PR #8250: URL: https://github.com/apache/iceberg/pull/8250#discussion_r1603639825 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/InternalRowWrapper.java: ## @@ -40,9 +44,17 @@ class InternalRowWrapper implements StructLike { priv

Re: [PR] Spark: Fix issue when partitioning by UUID [iceberg]

2024-05-16 Thread via GitHub
amogh-jahagirdar commented on code in PR #8250: URL: https://github.com/apache/iceberg/pull/8250#discussion_r1603587516 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/InternalRowWrapper.java: ## @@ -40,9 +44,17 @@ class InternalRowWrapper implements StructLike

Re: [PR] Url encode field names for partition paths [iceberg]

2024-05-16 Thread via GitHub
dimas-b commented on code in PR #10329: URL: https://github.com/apache/iceberg/pull/10329#discussion_r1603601992 ## api/src/main/java/org/apache/iceberg/PartitionSpec.java: ## @@ -189,7 +189,7 @@ public String partitionToPath(StructLike data) { if (i > 0) { sb.ap

Re: [PR] Support getting a snapshot right before the given timestamp [iceberg-python]

2024-05-16 Thread via GitHub
chinmay-bhat commented on code in PR #748: URL: https://github.com/apache/iceberg-python/pull/748#discussion_r1603586709 ## pyiceberg/table/metadata.py: ## @@ -230,6 +230,14 @@ def snapshot_by_id(self, snapshot_id: int) -> Optional[Snapshot]: """Get the snapshot by sna

Re: [PR] Build: Bump io.delta:delta-spark_2.12 from 3.1.0 to 3.2.0 [iceberg]

2024-05-16 Thread via GitHub
nastra merged PR #10320: URL: https://github.com/apache/iceberg/pull/10320 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Update mkdocs.yml - Fixes Apache Doris documentation link [iceberg]

2024-05-16 Thread via GitHub
nastra merged PR #10263: URL: https://github.com/apache/iceberg/pull/10263 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Update mkdocs.yml - Fixes Apache Doris documentation link [iceberg]

2024-05-16 Thread via GitHub
vinlee19 commented on code in PR #10263: URL: https://github.com/apache/iceberg/pull/10263#discussion_r1603408090 ## docs/mkdocs.yml: ## @@ -60,7 +60,7 @@ nav: - Amazon EMR: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-iceberg-use-cluster.html - Snowflake: http

[PR] Support getting a snapshot right before the given timestamp [iceberg-python]

2024-05-16 Thread via GitHub
chinmay-bhat opened a new pull request, #748: URL: https://github.com/apache/iceberg-python/pull/748 Bring support to retrieve a snapshot before a particular timestamp, which is needed to perform Spark functions like [`rollback_to_timestamp`](https://iceberg.apache.org/docs/1.5.1/spark-proc

Re: [I] How to set Spark conf to use Parquet and Iceberg tables using glue catalog without catalog name(spark_catalog)? [iceberg]

2024-05-16 Thread via GitHub
andreaschiappacasse commented on issue #7748: URL: https://github.com/apache/iceberg/issues/7748#issuecomment-2115161393 We are incurring in the same problem, @ryu3065 have you managed to find a solution? We would like to run a MERGE query that reads from Parquet tables and writes on an

Re: [PR] Spark 3.5: Fix the setting of equalAuthorities in RemoveOrphanFilesProcedure [iceberg]

2024-05-16 Thread via GitHub
hantangwangd commented on PR #10334: URL: https://github.com/apache/iceberg/pull/10334#issuecomment-2115089030 @nastra No problem, my pleasure! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Spark 3.5: Fix the setting of equalAuthorities in RemoveOrphanFilesProcedure [iceberg]

2024-05-16 Thread via GitHub
nastra commented on PR #10334: URL: https://github.com/apache/iceberg/pull/10334#issuecomment-2115071027 @hantangwangd could you also please backport this to Spark 3.3 / 3.4? Opening one PR where this is fixed for 3.3 + 3.4 should be fine. Thanks a lot -- This is an automated message from

Re: [PR] Build: Bump nessie from 0.81.1 to 0.82.0 [iceberg]

2024-05-16 Thread via GitHub
Fokko merged PR #10318: URL: https://github.com/apache/iceberg/pull/10318 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Spark 3.5: Fix the setting of equalAuthorities in RemoveOrphanFilesProcedure [iceberg]

2024-05-16 Thread via GitHub
nastra merged PR #10334: URL: https://github.com/apache/iceberg/pull/10334 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Spark 3.5: Fix the setting of equalAuthorities in RemoveOrphanFilesProcedure [iceberg]

2024-05-16 Thread via GitHub
nastra commented on code in PR #10334: URL: https://github.com/apache/iceberg/pull/10334#discussion_r1603217331 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveOrphanFilesProcedure.java: ## @@ -666,4 +666,87 @@ public void testRemoveOrp

Re: [PR] [FEAT]register table using iceberg metadata file via pyiceberg [iceberg-python]

2024-05-16 Thread via GitHub
MehulBatra commented on code in PR #711: URL: https://github.com/apache/iceberg-python/pull/711#discussion_r1603214209 ## tests/catalog/test_glue.py: ## @@ -848,3 +848,17 @@ def test_table_exists( assert test_catalog.table_exists(identifier) is True # Act and Assert fo

Re: [PR] [FEAT]register table using iceberg metadata file via pyiceberg [iceberg-python]

2024-05-16 Thread via GitHub
mehulbatraa commented on code in PR #711: URL: https://github.com/apache/iceberg-python/pull/711#discussion_r1603210840 ## tests/catalog/test_glue.py: ## @@ -848,3 +848,17 @@ def test_table_exists( assert test_catalog.table_exists(identifier) is True # Act and Assert f

Re: [PR] [FEAT]register table using iceberg metadata file via pyiceberg [iceberg-python]

2024-05-16 Thread via GitHub
mehulbatraa commented on code in PR #711: URL: https://github.com/apache/iceberg-python/pull/711#discussion_r1603210840 ## tests/catalog/test_glue.py: ## @@ -848,3 +848,17 @@ def test_table_exists( assert test_catalog.table_exists(identifier) is True # Act and Assert f

Re: [PR] Spark 3.5: Fix the setting of equalAuthorities in RemoveOrphanFilesProcedure [iceberg]

2024-05-16 Thread via GitHub
hantangwangd commented on code in PR #10334: URL: https://github.com/apache/iceberg/pull/10334#discussion_r1603153003 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveOrphanFilesProcedure.java: ## @@ -666,4 +666,87 @@ public void testRem

Re: [PR] Spark 3.5: Fix the setting of equalAuthorities in RemoveOrphanFilesProcedure [iceberg]

2024-05-16 Thread via GitHub
nastra commented on code in PR #10334: URL: https://github.com/apache/iceberg/pull/10334#discussion_r1603146737 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveOrphanFilesProcedure.java: ## @@ -666,4 +666,87 @@ public void testRemoveOrp

Re: [PR] Spark 3.5: Fix the setting of equalAuthorities in RemoveOrphanFilesProcedure [iceberg]

2024-05-16 Thread via GitHub
hantangwangd commented on code in PR #10334: URL: https://github.com/apache/iceberg/pull/10334#discussion_r160307 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveOrphanFilesProcedure.java: ## @@ -666,4 +666,87 @@ public void testRem

Re: [PR] REST: honor OAuth config sent by the server [iceberg]

2024-05-16 Thread via GitHub
adutra closed pull request #10256: REST: honor OAuth config sent by the server URL: https://github.com/apache/iceberg/pull/10256 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] REST: honor OAuth config sent by the server [iceberg]

2024-05-16 Thread via GitHub
adutra commented on PR #10256: URL: https://github.com/apache/iceberg/pull/10256#issuecomment-2114629113 Hi @rdblue, thank you for your detailed answer. I am really sorry that this PR, that I thought would be a fairly consensual one, eventually cracked open a can of worms that I did

Re: [I] Equality delete lost after compact data files [iceberg]

2024-05-16 Thread via GitHub
CodingJun commented on issue #10312: URL: https://github.com/apache/iceberg/issues/10312#issuecomment-2114253464 Do you know if this is a bug? @RussellSpitzer @aokolnychyi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u