Re: [PR] Spark: Add support for describing/showing views [iceberg]

2024-01-31 Thread via GitHub
nastra merged PR #9513: URL: https://github.com/apache/iceberg/pull/9513 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Spark 3.4: Rewrite identifier when using Subquery expressions in View [iceberg]

2024-01-31 Thread via GitHub
nastra merged PR #9594: URL: https://github.com/apache/iceberg/pull/9594 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [I] [Proposal] Iceberg Materialized View Spec [iceberg]

2024-01-31 Thread via GitHub
JanKaul commented on issue #6420: URL: https://github.com/apache/iceberg/issues/6420#issuecomment-1918780632 I've updated the Specification Draft. Please let me know if you think certain changes are in order. -- This is an automated message from the Apache Git Service. To respond to the m

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-31 Thread via GitHub
jbonofre commented on PR #9487: URL: https://github.com/apache/iceberg/pull/9487#issuecomment-1918818634 I updated the PR with: - a property triggering SQL table update if needed - update of previous version SQL schema to the new model - update the SQL statement to get Iceberg table,

Re: [PR] Spark: Support altering view properties [iceberg]

2024-01-31 Thread via GitHub
nastra commented on code in PR #9582: URL: https://github.com/apache/iceberg/pull/9582#discussion_r1472624137 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1362,6 +1362,120 @@ public void showCreateComplexView() { as

Re: [PR] Spark: Support altering view properties [iceberg]

2024-01-31 Thread via GitHub
nastra commented on code in PR #9582: URL: https://github.com/apache/iceberg/pull/9582#discussion_r1472630689 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1362,6 +1362,120 @@ public void showCreateComplexView() { as

Re: [PR] Spark: Support altering view properties [iceberg]

2024-01-31 Thread via GitHub
nastra commented on code in PR #9582: URL: https://github.com/apache/iceberg/pull/9582#discussion_r1472648183 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java: ## @@ -606,6 +607,29 @@ public View createView( @Override public View alterView(Ident

Re: [PR] Spark 3.4: Add support for describing/showing views [iceberg]

2024-01-31 Thread via GitHub
Fokko merged PR #9595: URL: https://github.com/apache/iceberg/pull/9595 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Spark 3.4: Add support for describing/showing views [iceberg]

2024-01-31 Thread via GitHub
Fokko commented on PR #9595: URL: https://github.com/apache/iceberg/pull/9595#issuecomment-1918956973 Great work @nastra Thanks for backporting this 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Flink: Don't fail to serialize IcebergSourceSplit when there is too many delete files [iceberg]

2024-01-31 Thread via GitHub
pvary commented on PR #9464: URL: https://github.com/apache/iceberg/pull/9464#issuecomment-1919072892 > Do you want me to proceed and update my PR @pvary accordingly? I would wait until the PR at least merged to the Flink main branch -- This is an automated message from the Apache G

Re: [PR] Flink: Added error handling and default logic for Flink version detection [iceberg]

2024-01-31 Thread via GitHub
pvary merged PR #9452: URL: https://github.com/apache/iceberg/pull/9452 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [I] [Flink] NullPointer when doing `FlinkEnvironmentContext.init` for Flink 1.17 and iceberg 1.3.0 [iceberg]

2024-01-31 Thread via GitHub
pvary closed issue #7879: [Flink] NullPointer when doing `FlinkEnvironmentContext.init` for Flink 1.17 and iceberg 1.3.0 URL: https://github.com/apache/iceberg/issues/7879 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Flink: Added error handling and default logic for Flink version detection [iceberg]

2024-01-31 Thread via GitHub
pvary commented on PR #9452: URL: https://github.com/apache/iceberg/pull/9452#issuecomment-1919078132 Thanks @gjacoby126 for the PR and the persistence until we decided on the right approach! -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] Flink: Implement enumerator metrics for pending splits, pending recor… [iceberg]

2024-01-31 Thread via GitHub
pvary commented on PR #9524: URL: https://github.com/apache/iceberg/pull/9524#issuecomment-1919086059 > @pvary this is only supported by 1.18 and a 1.17 impl would cause a runtime error. As mentioned in the PR description, I don't think backporting is necessary here Got it.. Thanks @

[I] [Flink] NullPointer when doing `FlinkEnvironmentContext.init` for Flink 1.17 and iceberg 1.3.0 [iceberg]

2024-01-31 Thread via GitHub
pawel-big-lebowski opened a new issue, #7879: URL: https://github.com/apache/iceberg/issues/7879 ### Apache Iceberg version 1.3.0 (latest release) ### Query engine Flink ### Please describe the bug 🐞 I am running Flink 1.17 and iceberg 1.3.0. I am trying to

Re: [I] [Flink] NullPointer when doing `FlinkEnvironmentContext.init` for Flink 1.17 and iceberg 1.3.0 [iceberg]

2024-01-31 Thread via GitHub
pvary closed issue #7879: [Flink] NullPointer when doing `FlinkEnvironmentContext.init` for Flink 1.17 and iceberg 1.3.0 URL: https://github.com/apache/iceberg/issues/7879 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

[PR] Spark: Bypass Spark's ViewCatalog API when replacing a view [iceberg]

2024-01-31 Thread via GitHub
nastra opened a new pull request, #9596: URL: https://github.com/apache/iceberg/pull/9596 Spark's `ViewCatalog` API doesn't have a `replace()` in 3.5 as it was only introduced later. Therefore we're bypassing Spark's `ViewCatalog` so that we can keep the view's history after executing a `CR

Re: [PR] Spark: Bypass Spark's ViewCatalog API when replacing a view [iceberg]

2024-01-31 Thread via GitHub
nastra commented on code in PR #9596: URL: https://github.com/apache/iceberg/pull/9596#discussion_r1472820936 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SupportsViewReplace.java: ## @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Flink: Added error handling and default logic for Flink version detection [iceberg]

2024-01-31 Thread via GitHub
gjacoby126 commented on PR #9452: URL: https://github.com/apache/iceberg/pull/9452#issuecomment-1919110607 Thanks for the reviews, @pvary @nastra and @stevenzwu! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] InMemory Catalog Implementation [iceberg-python]

2024-01-31 Thread via GitHub
syun64 commented on code in PR #289: URL: https://github.com/apache/iceberg-python/pull/289#discussion_r1472905954 ## tests/catalog/test_base.py: ## @@ -355,8 +162,6 @@ def test_create_table_pyarrow_schema(catalog: InMemoryCatalog, pyarrow_schema_si table = catalog.create_

Re: [PR] Spark 3.4: Add support for reading Iceberg views [iceberg]

2024-01-31 Thread via GitHub
nastra commented on code in PR #9422: URL: https://github.com/apache/iceberg/pull/9422#discussion_r1472925778 ## spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -0,0 +1,647 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] feat: add support for catalogs with glue implementation to start [iceberg-go]

2024-01-31 Thread via GitHub
nastra commented on code in PR #51: URL: https://github.com/apache/iceberg-go/pull/51#discussion_r1472931535 ## catalog/catalog.go: ## @@ -0,0 +1,65 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file

Re: [PR] Spark: IN clause on system function is not pushed down [iceberg]

2024-01-31 Thread via GitHub
nastra commented on PR #9192: URL: https://github.com/apache/iceberg/pull/9192#issuecomment-1919260424 @aokolnychyi could you take a look at this please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-01-31 Thread via GitHub
RussellSpitzer commented on PR #9546: URL: https://github.com/apache/iceberg/pull/9546#issuecomment-1919357852 Based on our discussions last night on slack, I believe that we are basically in the following situation ```java commit() { // Failure here means we throw CommitFailedE

Re: [PR] feat: add support for catalogs with glue implementation to start [iceberg-go]

2024-01-31 Thread via GitHub
nastra merged PR #51: URL: https://github.com/apache/iceberg-go/pull/51 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] feat: add support for catalogs with glue implementation to start [iceberg-go]

2024-01-31 Thread via GitHub
nastra commented on PR #51: URL: https://github.com/apache/iceberg-go/pull/51#issuecomment-1919372982 thanks for the reviews everyone and thanks to @wolfeidau for getting this done! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [PR] InMemory Catalog Implementation [iceberg-python]

2024-01-31 Thread via GitHub
syun64 commented on code in PR #289: URL: https://github.com/apache/iceberg-python/pull/289#discussion_r1473049416 ## tests/catalog/test_base.py: ## @@ -355,8 +162,6 @@ def test_create_table_pyarrow_schema(catalog: InMemoryCatalog, pyarrow_schema_si table = catalog.create_

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-01-31 Thread via GitHub
RussellSpitzer commented on PR #9546: URL: https://github.com/apache/iceberg/pull/9546#issuecomment-1919382964 Just realized there is a concurrency issue as well and we can't just check if n+1-metadata.json exists since another process may have made this file while we were throwing an excep

Re: [PR] Open-API: Add table updates for statistics [iceberg]

2024-01-31 Thread via GitHub
nastra commented on code in PR #9564: URL: https://github.com/apache/iceberg/pull/9564#discussion_r1473058765 ## open-api/rest-catalog-open-api.yaml: ## @@ -3158,6 +3193,55 @@ components: metadata: $ref: '#/components/schemas/TableMetadata' +StatisticsF

Re: [I] It's not possible to readStream from an Iceberg table as source when its snapshots expire [iceberg]

2024-01-31 Thread via GitHub
methiakshit-plutoflume commented on issue #9504: URL: https://github.com/apache/iceberg/issues/9504#issuecomment-1919419646 The way spark understand the batch of data is by taking a diff between two snapshots. So, it is important for you to keep the history of snapshots which makes you feel

Re: [PR] Write support [iceberg-python]

2024-01-31 Thread via GitHub
mkleinbort-ic commented on PR #41: URL: https://github.com/apache/iceberg-python/pull/41#issuecomment-1919466577 Is there an ETA for write functionality in the released version? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Convert Hugo versioned docs to mkdocs format [iceberg]

2024-01-31 Thread via GitHub
Fokko commented on code in PR #9591: URL: https://github.com/apache/iceberg/pull/9591#discussion_r1473065889 ## docs/docs/flink.md: ## @@ -28,7 +20,7 @@ menu: # Flink -Apache Iceberg supports both [Apache Flink](https://flink.apache.org/)'s DataStream API and Table API. Se

Re: [PR] Write support [iceberg-python]

2024-01-31 Thread via GitHub
EternalDeiwos commented on PR #41: URL: https://github.com/apache/iceberg-python/pull/41#issuecomment-1919475189 Check the attached milestone for progress. When those issues are resolved it will be ready for release. -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Write support [iceberg-python]

2024-01-31 Thread via GitHub
syun64 commented on PR #41: URL: https://github.com/apache/iceberg-python/pull/41#issuecomment-1919479530 Hi @mkleinbort-ic we've just started [voting](https://lists.apache.org/list.html?d...@iceberg.apache.org) on the first release candidate that incorporates this change -- This is an a

[PR] Retry with new Access Token on 419 response [iceberg-python]

2024-01-31 Thread via GitHub
anupam-saini opened a new pull request, #340: URL: https://github.com/apache/iceberg-python/pull/340 Closes #234 - Attempts to retry 2 more times if the initial call fails. - The retry is only triggered for `AuthorizationExpiredError` response. -- This is an automated message

Re: [PR] Rest Catalog: Add RESTful AppendFiles data operation [iceberg]

2024-01-31 Thread via GitHub
danielcweeks commented on code in PR #9292: URL: https://github.com/apache/iceberg/pull/9292#discussion_r1473142384 ## core/src/main/java/org/apache/iceberg/MetadataUpdate.java: ## @@ -490,4 +491,21 @@ public void applyTo(ViewMetadata.Builder viewMetadataBuilder) { viewM

Re: [PR] Spark: Support altering view properties [iceberg]

2024-01-31 Thread via GitHub
rdblue commented on code in PR #9582: URL: https://github.com/apache/iceberg/pull/9582#discussion_r1473167133 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1362,6 +1362,120 @@ public void showCreateComplexView() { as

Re: [PR] Spark: Support altering view properties [iceberg]

2024-01-31 Thread via GitHub
rdblue commented on code in PR #9582: URL: https://github.com/apache/iceberg/pull/9582#discussion_r1473170465 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1362,6 +1362,127 @@ public void showCreateComplexView() { as

Re: [PR] Spark: Support altering view properties [iceberg]

2024-01-31 Thread via GitHub
rdblue commented on code in PR #9582: URL: https://github.com/apache/iceberg/pull/9582#discussion_r1473170891 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1362,6 +1362,127 @@ public void showCreateComplexView() { as

Re: [PR] Spark: Support altering view properties [iceberg]

2024-01-31 Thread via GitHub
nastra commented on code in PR #9582: URL: https://github.com/apache/iceberg/pull/9582#discussion_r1472624137 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1362,6 +1362,120 @@ public void showCreateComplexView() { as

Re: [PR] Spark: Support altering view properties [iceberg]

2024-01-31 Thread via GitHub
rdblue commented on code in PR #9582: URL: https://github.com/apache/iceberg/pull/9582#discussion_r1473172442 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1362,6 +1362,127 @@ public void showCreateComplexView() { as

Re: [PR] Spark: Support altering view properties [iceberg]

2024-01-31 Thread via GitHub
rdblue commented on code in PR #9582: URL: https://github.com/apache/iceberg/pull/9582#discussion_r1473173269 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1362,6 +1362,127 @@ public void showCreateComplexView() { as

Re: [PR] Spark: Support altering view properties [iceberg]

2024-01-31 Thread via GitHub
rdblue commented on code in PR #9582: URL: https://github.com/apache/iceberg/pull/9582#discussion_r1473173626 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1362,6 +1362,127 @@ public void showCreateComplexView() { as

Re: [PR] Spark: Support altering view properties [iceberg]

2024-01-31 Thread via GitHub
rdblue commented on code in PR #9582: URL: https://github.com/apache/iceberg/pull/9582#discussion_r1473175246 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkView.java: ## @@ -36,7 +36,7 @@ public class SparkView implements org.apache.spark.sql.connector.

Re: [PR] Spark: Bypass Spark's ViewCatalog API when replacing a view [iceberg]

2024-01-31 Thread via GitHub
rdblue commented on code in PR #9596: URL: https://github.com/apache/iceberg/pull/9596#discussion_r1473176643 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SupportsReplacingViews.java: ## @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

Re: [PR] Spark: Bypass Spark's ViewCatalog API when replacing a view [iceberg]

2024-01-31 Thread via GitHub
rdblue commented on code in PR #9596: URL: https://github.com/apache/iceberg/pull/9596#discussion_r1473178368 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1362,6 +1362,21 @@ public void showCreateComplexView() { ass

Re: [PR] Spark: Bypass Spark's ViewCatalog API when replacing a view [iceberg]

2024-01-31 Thread via GitHub
rdblue commented on code in PR #9596: URL: https://github.com/apache/iceberg/pull/9596#discussion_r1473179743 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java: ## @@ -608,6 +608,53 @@ public View createView( "Creating a view is not supported

Re: [PR] Spark: Bypass Spark's ViewCatalog API when replacing a view [iceberg]

2024-01-31 Thread via GitHub
rdblue commented on code in PR #9596: URL: https://github.com/apache/iceberg/pull/9596#discussion_r1473180927 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java: ## @@ -608,6 +608,53 @@ public View createView( "Creating a view is not supported

Re: [I] Support partitioned writes [iceberg-python]

2024-01-31 Thread via GitHub
syun64 commented on issue #208: URL: https://github.com/apache/iceberg-python/issues/208#issuecomment-1919558503 The [Design Document](https://docs.google.com/document/d/1TLIzxKJilvhAq4JDoGMWMZdkRZXvcrG5YrxLvJ5UXkQ/edit#heading=h.f84o4qaemlga) on data file writes that was discussed during t

Re: [PR] Spark: Bypass Spark's ViewCatalog API when replacing a view [iceberg]

2024-01-31 Thread via GitHub
rdblue commented on code in PR #9596: URL: https://github.com/apache/iceberg/pull/9596#discussion_r1473187177 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java: ## @@ -608,6 +608,53 @@ public View createView( "Creating a view is not supported

Re: [PR] Spark: Bypass Spark's ViewCatalog API when replacing a view [iceberg]

2024-01-31 Thread via GitHub
rdblue commented on code in PR #9596: URL: https://github.com/apache/iceberg/pull/9596#discussion_r1473189034 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java: ## @@ -608,6 +608,53 @@ public View createView( "Creating a view is not supported

Re: [I] `update_{min,max}` fail if `val` is None [iceberg-python]

2024-01-31 Thread via GitHub
Fokko commented on issue #337: URL: https://github.com/apache/iceberg-python/issues/337#issuecomment-1919569515 Thanks for raising this @jonashaag. This seems like a serious issue. Do you have a stack trace or some lines to reproduce it? That would be very helpful. -- This is an automated

Re: [PR] API, Spark: Fix aggregation pushdown on struct fields [iceberg]

2024-01-31 Thread via GitHub
rdblue merged PR #9176: URL: https://github.com/apache/iceberg/pull/9176 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] API, Spark: Fix aggregation pushdown on struct fields [iceberg]

2024-01-31 Thread via GitHub
rdblue commented on PR #9176: URL: https://github.com/apache/iceberg/pull/9176#issuecomment-1919576523 Nice work, @amogh-jahagirdar! Thanks for getting this done for the 1.5 release! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [I] Core: complete FileScanTaskParser for other FileScanTask implementation classes (like StaticDataTask) [iceberg]

2024-01-31 Thread via GitHub
stevenzwu commented on issue #9597: URL: https://github.com/apache/iceberg/issues/9597#issuecomment-1919599639 Regarding `StaticDataTask`, we can make this constructor package private. ``` private StaticDataTask( InputFile metadata, Schema tableSchema, Schema projectedSchema,

Re: [PR] Convert Hugo versioned docs to mkdocs format [iceberg]

2024-01-31 Thread via GitHub
rdblue commented on code in PR #9591: URL: https://github.com/apache/iceberg/pull/9591#discussion_r1473271642 ## docs/docs/branching.md: ## @@ -33,14 +25,14 @@ menu: Iceberg table metadata maintains a snapshot log, which represents the changes applied to a table. Snapshots

Re: [PR] Partition Evolution [iceberg-python]

2024-01-31 Thread via GitHub
Fokko commented on code in PR #245: URL: https://github.com/apache/iceberg-python/pull/245#discussion_r1473278876 ## pyiceberg/table/__init__.py: ## @@ -2271,3 +2325,240 @@ def commit(self) -> Snapshot: ) return snapshot + + +class UpdateSpec: +_table

Re: [PR] Support usage of Separate OIDC Authorization Server URI [iceberg]

2024-01-31 Thread via GitHub
danielcweeks commented on code in PR #8976: URL: https://github.com/apache/iceberg/pull/8976#discussion_r1473292686 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Properties.java: ## @@ -27,6 +27,9 @@ private OAuth2Properties() {} /** A credential to exchange for a

Re: [PR] Support usage of Separate OIDC Authorization Server URI [iceberg]

2024-01-31 Thread via GitHub
danielcweeks commented on code in PR #8976: URL: https://github.com/apache/iceberg/pull/8976#discussion_r1473296167 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java: ## @@ -361,6 +392,23 @@ public static class AuthSession { private final String credential;

Re: [PR] Convert Hugo versioned docs to mkdocs format [iceberg]

2024-01-31 Thread via GitHub
bitsondatadev commented on code in PR #9591: URL: https://github.com/apache/iceberg/pull/9591#discussion_r1473299407 ## docs/docs/branching.md: ## @@ -33,14 +25,14 @@ menu: Iceberg table metadata maintains a snapshot log, which represents the changes applied to a table. Sna

Re: [PR] Convert Hugo versioned docs to mkdocs format [iceberg]

2024-01-31 Thread via GitHub
bitsondatadev commented on code in PR #9591: URL: https://github.com/apache/iceberg/pull/9591#discussion_r1473299407 ## docs/docs/branching.md: ## @@ -33,14 +25,14 @@ menu: Iceberg table metadata maintains a snapshot log, which represents the changes applied to a table. Sna

Re: [PR] Convert Hugo versioned docs to mkdocs format [iceberg]

2024-01-31 Thread via GitHub
bitsondatadev commented on code in PR #9591: URL: https://github.com/apache/iceberg/pull/9591#discussion_r1473299407 ## docs/docs/branching.md: ## @@ -33,14 +25,14 @@ menu: Iceberg table metadata maintains a snapshot log, which represents the changes applied to a table. Sna

Re: [PR] Support usage of Separate OIDC Authorization Server URI [iceberg]

2024-01-31 Thread via GitHub
danielcweeks commented on code in PR #8976: URL: https://github.com/apache/iceberg/pull/8976#discussion_r1473308176 ## core/src/main/java/org/apache/iceberg/rest/HTTPClient.java: ## @@ -186,8 +186,21 @@ private static void throwFailure( throw new RESTException("Unhandled er

Re: [PR] Support usage of Separate OIDC Authorization Server URI [iceberg]

2024-01-31 Thread via GitHub
danielcweeks commented on code in PR #8976: URL: https://github.com/apache/iceberg/pull/8976#discussion_r1473311127 ## core/src/main/java/org/apache/iceberg/rest/HTTPClient.java: ## @@ -217,22 +230,22 @@ private URI buildUri(String path, Map params) { */ private T execu

Re: [PR] Convert Hugo versioned docs to mkdocs format [iceberg]

2024-01-31 Thread via GitHub
Fokko commented on code in PR #9591: URL: https://github.com/apache/iceberg/pull/9591#discussion_r1473311804 ## docs/docs/branching.md: ## @@ -33,14 +25,14 @@ menu: Iceberg table metadata maintains a snapshot log, which represents the changes applied to a table. Snapshots a

Re: [PR] Support usage of Separate OIDC Authorization Server URI [iceberg]

2024-01-31 Thread via GitHub
danielcweeks commented on code in PR #8976: URL: https://github.com/apache/iceberg/pull/8976#discussion_r1473314309 ## core/src/main/java/org/apache/iceberg/rest/HTTPClient.java: ## @@ -314,7 +323,7 @@ private T execute( @Override public void head(String path, Map heade

Re: [PR] Support usage of Separate OIDC Authorization Server URI [iceberg]

2024-01-31 Thread via GitHub
danielcweeks commented on code in PR #8976: URL: https://github.com/apache/iceberg/pull/8976#discussion_r1473325248 ## core/src/test/java/org/apache/iceberg/rest/RESTCatalogAdapter.java: ## @@ -105,6 +105,8 @@ enum HTTPMethod { enum Route { TOKENS(HTTPMethod.POST, "v1/

Re: [PR] Spark: Support altering view properties [iceberg]

2024-01-31 Thread via GitHub
nastra commented on code in PR #9582: URL: https://github.com/apache/iceberg/pull/9582#discussion_r1473327348 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1362,6 +1362,127 @@ public void showCreateComplexView() { as

Re: [PR] Convert Hugo versioned docs to mkdocs format [iceberg]

2024-01-31 Thread via GitHub
bitsondatadev commented on code in PR #9591: URL: https://github.com/apache/iceberg/pull/9591#discussion_r1473338102 ## docs/docs/branching.md: ## @@ -33,14 +25,14 @@ menu: Iceberg table metadata maintains a snapshot log, which represents the changes applied to a table. Sna

Re: [PR] feat(catalog): add initial rest catalog impl [iceberg-go]

2024-01-31 Thread via GitHub
zeroshade commented on PR #58: URL: https://github.com/apache/iceberg-go/pull/58#issuecomment-1919765148 CC @Fokko @wolfeidau @nastra @HonahX @jackye1995 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] Spark 3.4: Rework DeleteFileIndexBenchmark [iceberg]

2024-01-31 Thread via GitHub
aokolnychyi closed pull request #9600: Spark 3.4: Rework DeleteFileIndexBenchmark URL: https://github.com/apache/iceberg/pull/9600 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] Core: Fix setting updated parquet compression property [iceberg]

2024-01-31 Thread via GitHub
rdblue commented on PR #9503: URL: https://github.com/apache/iceberg/pull/9503#issuecomment-1919889634 It looks like we don't have consensus that this is the right direction, so I'm going to at least remove it from the 1.5.0 milestone. We can discuss it more at the next community sync maybe

Re: [I] [Proposal] Iceberg Materialized View Spec [iceberg]

2024-01-31 Thread via GitHub
szehon-ho commented on issue #6420: URL: https://github.com/apache/iceberg/issues/6420#issuecomment-1919922742 Thanks its a lot clearer now. I guess we still have some open questions that came up, which can talk there. I will try to get some other folks to look at it there as well. --

Re: [PR] Support usage of Separate OIDC Authorization Server URI [iceberg]

2024-01-31 Thread via GitHub
syun64 commented on code in PR #8976: URL: https://github.com/apache/iceberg/pull/8976#discussion_r1473467310 ## core/src/test/java/org/apache/iceberg/rest/RESTCatalogAdapter.java: ## @@ -105,6 +105,8 @@ enum HTTPMethod { enum Route { TOKENS(HTTPMethod.POST, "v1/oauth/

[I] What is Table Identifier? [iceberg-python]

2024-01-31 Thread via GitHub
kevinjqliu opened a new issue, #341: URL: https://github.com/apache/iceberg-python/issues/341 ### Question In `test_base.py` the Table Identifier, `TEST_TABLE_IDENTIFIER`, is a tuple of 4 elements ``` TEST_TABLE_IDENTIFIER = ("com", "organization", "department", "my_table")

[PR] Build: Bump mypy-boto3-glue from 1.34.7 to 1.34.32 [iceberg-python]

2024-01-31 Thread via GitHub
dependabot[bot] opened a new pull request, #342: URL: https://github.com/apache/iceberg-python/pull/342 Bumps [mypy-boto3-glue](https://github.com/youtype/mypy_boto3_builder) from 1.34.7 to 1.34.32. Commits See full diff in https://github.com/youtype/mypy_boto3_builder/commits"

[PR] Build: Bump pypa/cibuildwheel from 2.16.4 to 2.16.5 [iceberg-python]

2024-01-31 Thread via GitHub
dependabot[bot] opened a new pull request, #343: URL: https://github.com/apache/iceberg-python/pull/343 Bumps [pypa/cibuildwheel](https://github.com/pypa/cibuildwheel) from 2.16.4 to 2.16.5. Release notes Sourced from https://github.com/pypa/cibuildwheel/releases";>pypa/cibuildwhee

Re: [PR] append, support identity transform partitioned write [iceberg-python]

2024-01-31 Thread via GitHub
jqin61 closed pull request #339: append, support identity transform partitioned write URL: https://github.com/apache/iceberg-python/pull/339 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [I] [Proposal] Iceberg Materialized View Spec [iceberg]

2024-01-31 Thread via GitHub
wmoustafa commented on issue #6420: URL: https://github.com/apache/iceberg/issues/6420#issuecomment-1920163073 The specification draft as in the PR or the Google doc? If there are open questions, let us continue them on the Google doc instead of creating a third place for discussion? It is

Re: [PR] Spark 3.4: Add support for reading Iceberg views [iceberg]

2024-01-31 Thread via GitHub
wmoustafa commented on code in PR #9422: URL: https://github.com/apache/iceberg/pull/9422#discussion_r1473614948 ## spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -0,0 +1,647 @@ +/* + * Licensed to the Apache Software Foundati

[I] Change avro schema names for `ManifestList` and `Manifest` to the official ones [iceberg-rust]

2024-01-31 Thread via GitHub
zeodtr opened a new issue, #179: URL: https://github.com/apache/iceberg-rust/issues/179 Currently, the avro schema names in the source code are as follows: * ManifestList: "manifest_list" * Manifest: "manifest" (Search for `schema_to_avro_schema` function calls to see them) Bu

Re: [I] [Proposal] Iceberg Materialized View Spec [iceberg]

2024-01-31 Thread via GitHub
szehon-ho commented on issue #6420: URL: https://github.com/apache/iceberg/issues/6420#issuecomment-1920421234 Yea , we mean , I think we will continue on the google doc until the open questions addressed. @JanKaul made the specification draft section, it should be clearer now. -- This

Re: [I] Change avro schema names for `ManifestList` and `Manifest` to the official ones [iceberg-rust]

2024-01-31 Thread via GitHub
JanKaul commented on issue #179: URL: https://github.com/apache/iceberg-rust/issues/179#issuecomment-1920543377 I would like to take this one. It was me who introduced the wrong names. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[I] InMemoryCatalog's FiloIO in memory map isn't persistent in RestCatalog [iceberg]

2024-01-31 Thread via GitHub
geruh opened a new issue, #9604: URL: https://github.com/apache/iceberg/issues/9604 ### Query engine None ### Question We are seeing some unexpected behavior when testing the `RESTCatalog`. The `RESTCatalog` is intended to use an `InMemoryCatalog` for testing purposes. I

[PR] Spark 3.5: Add deleted_snapshots_count to result of expire_snapshots procedure [iceberg]

2024-01-31 Thread via GitHub
manuzhang opened a new pull request, #9605: URL: https://github.com/apache/iceberg/pull/9605 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [PR] Spark 3.5: Add an option not to delete files in ExpireSnapshots [iceberg]

2024-01-31 Thread via GitHub
manuzhang commented on code in PR #9584: URL: https://github.com/apache/iceberg/pull/9584#discussion_r1473862546 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/ExpireSnapshotsSparkAction.java: ## @@ -67,6 +67,8 @@ public class ExpireSnapshotsSparkAction exten

Re: [PR] Add UnboundSortOrder [iceberg-rust]

2024-01-31 Thread via GitHub
liurenjie1024 commented on PR #115: URL: https://github.com/apache/iceberg-rust/pull/115#issuecomment-1920612436 cc @Xuanwo @ZENOTME @Fokko PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] Spark 3.3: Add RemoveDanglingDeletes action [iceberg]

2024-01-31 Thread via GitHub
zinking commented on code in PR #6581: URL: https://github.com/apache/iceberg/pull/6581#discussion_r1469455133 ## api/src/main/java/org/apache/iceberg/DeleteFiles.java: ## @@ -55,6 +55,17 @@ default DeleteFiles deleteFile(DataFile file) { return this; } + /** + * De

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-01-31 Thread via GitHub
liurenjie1024 commented on code in PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#discussion_r1473864622 ## crates/iceberg/src/error.rs: ## @@ -44,6 +44,9 @@ pub enum ErrorKind { /// /// This error is returned when given iceberg feature is not supported.

Re: [PR] Build: Bump pypa/cibuildwheel from 2.16.4 to 2.16.5 [iceberg-python]

2024-01-31 Thread via GitHub
Fokko merged PR #343: URL: https://github.com/apache/iceberg-python/pull/343 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Change avro schema names for `ManifestList` and `Manifest` to the official ones [iceberg-rust]

2024-01-31 Thread via GitHub
liurenjie1024 commented on issue #179: URL: https://github.com/apache/iceberg-rust/issues/179#issuecomment-1920668680 Thanks @zeodtr for reporting this! And thanks @JanKaul for taking it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] Build: Bump mypy-boto3-glue from 1.34.7 to 1.34.32 [iceberg-python]

2024-01-31 Thread via GitHub
Fokko merged PR #342: URL: https://github.com/apache/iceberg-python/pull/342 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

[I] Tracking issues of Iceberg Rust 0.2.0 Release [iceberg-rust]

2024-01-31 Thread via GitHub
liurenjie1024 opened a new issue, #180: URL: https://github.com/apache/iceberg-rust/issues/180 This issue is used to track tasks of the iceberg rust 0.2.0 release. ## Tasks ### Blockers No. ### Build Release GitHub Side - [ ] Bump version in proj

Re: [PR] Spark: Support altering view properties [iceberg]

2024-01-31 Thread via GitHub
nastra commented on code in PR #9582: URL: https://github.com/apache/iceberg/pull/9582#discussion_r1473919837 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkView.java: ## @@ -36,7 +36,7 @@ public class SparkView implements org.apache.spark.sql.connector.

Re: [PR] Spark: Support altering view properties [iceberg]

2024-01-31 Thread via GitHub
nastra commented on PR #9582: URL: https://github.com/apache/iceberg/pull/9582#issuecomment-1920676510 Thanks for the review @rdblue, I've updated the error msgs and also renamed the internal property to `spark.query-column-names` -- This is an automated message from the Apache Git Servic

[PR] feat: Bump version 0.2.0 to prepare for release. [iceberg-rust]

2024-01-31 Thread via GitHub
liurenjie1024 opened a new pull request, #181: URL: https://github.com/apache/iceberg-rust/pull/181 We have reached consensus to run the first release: `0.2.0`, see [the discussion](https://lists.apache.org/thread/8sj66kf02qd0qtyyrj4gox3yozjk3vtd) in mail list. This is the first ste

Re: [PR] feat: Bump version 0.2.0 to prepare for release. [iceberg-rust]

2024-01-31 Thread via GitHub
liurenjie1024 commented on PR #181: URL: https://github.com/apache/iceberg-rust/pull/181#issuecomment-1920689566 cc @ZENOTME @Xuanwo @Fokko PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] Spark: Throw exception on `ALTER VIEW AS ` [iceberg]

2024-01-31 Thread via GitHub
nastra commented on code in PR #9510: URL: https://github.com/apache/iceberg/pull/9510#discussion_r1473932334 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -60,6 +62,9 @@ case class RewriteViewCommands(spark

Re: [PR] Open-API: Add table updates for statistics [iceberg]

2024-01-31 Thread via GitHub
nastra merged PR #9564: URL: https://github.com/apache/iceberg/pull/9564 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] API, Spark: Fix aggregation pushdown on struct fields [iceberg]

2024-01-31 Thread via GitHub
nastra commented on code in PR #9176: URL: https://github.com/apache/iceberg/pull/9176#discussion_r1473941848 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java: ## @@ -478,6 +480,126 @@ public void testAggregateWithComplexType() {

  1   2   >