[I] Spark read failed when migrate hive orc table with `timestamp` column [iceberg]

2024-02-23 Thread via GitHub
ming95 opened a new issue, #9784: URL: https://github.com/apache/iceberg/issues/9784 ### Apache Iceberg version None ### Query engine None ### Please describe the bug 🐞 ``` spark.sql("CREATE EXTERNAL TABLE mytable (foo timestamp) STORED AS orc LO

Re: [PR] Add Pagination To List Apis [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1500365522 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -224,6 +229,12 @@ public void initialize(String name, Map unresolved) { clie

Re: [PR] Add Pagination To List Apis [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1500368758 ## core/src/main/java/org/apache/iceberg/rest/responses/ListTablesResponse.java: ## @@ -30,13 +30,15 @@ public class ListTablesResponse implements RESTResponse { Revie

Re: [PR] Add Pagination To List Apis [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1500369207 ## core/src/main/java/org/apache/iceberg/rest/responses/ListTablesResponse.java: ## @@ -60,6 +69,7 @@ public static Builder builder() { public static class Builder

Re: [PR] Add Pagination To List Apis [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1500369784 ## core/src/main/java/org/apache/iceberg/rest/responses/ListNamespacesResponse.java: ## @@ -29,13 +29,15 @@ public class ListNamespacesResponse implements RESTResponse

Re: [PR] Add Pagination To List Apis [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1500371045 ## core/src/main/java/org/apache/iceberg/rest/responses/ListNamespacesResponse.java: ## @@ -59,6 +68,7 @@ public static Builder builder() { public static class Buil

Re: [PR] Add Pagination To List Apis [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1500375047 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -490,12 +522,29 @@ public void createNamespace( @Override public List listNamespaces

Re: [PR] Add Pagination To List Apis [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1500378492 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -1045,6 +1094,27 @@ public void commitTransaction(SessionContext context, List commits)

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500420191 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +471,163 @@ public void testRewriteLargeManifestsPartiti

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500420191 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +471,163 @@ public void testRewriteLargeManifestsPartiti

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500421966 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +471,163 @@ public void testRewriteLargeManifestsPartiti

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r150049 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +471,163 @@ public void testRewriteLargeManifestsPartiti

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500422547 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +471,163 @@ public void testRewriteLargeManifestsPartiti

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500422996 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +471,163 @@ public void testRewriteLargeManifestsPartiti

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500423875 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +471,163 @@ public void testRewriteLargeManifestsPartiti

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500424697 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +471,163 @@ public void testRewriteLargeManifestsPartiti

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500425302 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +471,163 @@ public void testRewriteLargeManifestsPartiti

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500426585 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +471,163 @@ public void testRewriteLargeManifestsPartiti

Re: [PR] doc: Add download page [iceberg-rust]

2024-02-23 Thread via GitHub
Fokko merged PR #219: URL: https://github.com/apache/iceberg-rust/pull/219 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [I] Spark: Iceberg: java.io.InvalidClassException: org.apache.iceberg.Schema; local class incompatible: stream classdesc serialVersionUID = 3320367012418887609, local class serialVersionUID = -885

2024-02-23 Thread via GitHub
FWLamb commented on issue #5970: URL: https://github.com/apache/iceberg/issues/5970#issuecomment-1961093423 @jornfranke Hello, Did you finally confirm that there was an older version of the classpath that caused the issue? -- This is an automated message from the Apache Git Service. To re

Re: [PR] chore(deps): Update derive_builder requirement from 0.13.0 to 0.20.0 [iceberg-rust]

2024-02-23 Thread via GitHub
Fokko merged PR #203: URL: https://github.com/apache/iceberg-rust/pull/203 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

[I] S3FileIO does not support Iceberg Cross-Region API Calls to Amazon S3 buckets [iceberg]

2024-02-23 Thread via GitHub
elmehdibelgasmi opened a new issue, #9785: URL: https://github.com/apache/iceberg/issues/9785 ### Apache Iceberg version None ### Query engine None ### Please describe the bug 🐞 ## Issue Description With **S3FileIO** implementation, Cross-Region API c

Re: [I] ValidationException: Missing required files to delete [iceberg]

2024-02-23 Thread via GitHub
bk-mz closed issue #9741: ValidationException: Missing required files to delete URL: https://github.com/apache/iceberg/issues/9741 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [I] flink1.14.4+iceberg0.13.1+hive-metastore3.1.2+minio(S3) error! [iceberg]

2024-02-23 Thread via GitHub
adamyasharma2797 commented on issue #4743: URL: https://github.com/apache/iceberg/issues/4743#issuecomment-1961235915 Approach provided by @lvyanquan worked for me! Thanks!! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] S3FileIO does not support Iceberg Cross-Region API Calls to Amazon S3 buckets [iceberg]

2024-02-23 Thread via GitHub
nastra commented on issue #9785: URL: https://github.com/apache/iceberg/issues/9785#issuecomment-1961277571 @elmehdibelgasmi would you be interested in opening up a PR and fixing this for S3FileIO? -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [I] S3FileIO does not support Iceberg Cross-Region API Calls to Amazon S3 buckets [iceberg]

2024-02-23 Thread via GitHub
elmehdibelgasmi commented on issue #9785: URL: https://github.com/apache/iceberg/issues/9785#issuecomment-1961286970 > @elmehdibelgasmi would you be interested in opening up a PR and fixing this for S3FileIO? @nastra sure, will do. -- This is an automated message from the Apache Gi

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
tomtongue commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500651939 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMigrateTableProcedure.java: ## @@ -18,51 +18,45 @@ */ package org.apache.icebe

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
tomtongue commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500651939 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMigrateTableProcedure.java: ## @@ -18,51 +18,45 @@ */ package org.apache.icebe

Re: [PR] Docs: Sync specs to site via symlinks [iceberg]

2024-02-23 Thread via GitHub
nastra commented on PR #9779: URL: https://github.com/apache/iceberg/pull/9779#issuecomment-1961319538 I like that idea, but this doesn't render the Spec site correctly when running the site locally. This is because `{{% icebergVersion %}}` needs to be replaced with `{{ icebergVersion }}` s

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500677979 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAddFilesProcedure.java: ## @@ -95,8 +97,8 @@ public static Object[][] parameters() {

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500691064 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveOrphanFilesProcedure.java: ## @@ -18,8 +18,10 @@ */ package org.apache.iceb

Re: [PR] Add filter pushdown API for contains and endsWith [iceberg]

2024-02-23 Thread via GitHub
yabola commented on PR #9710: URL: https://github.com/apache/iceberg/pull/9710#issuecomment-1961363628 @amogh-jahagirdar Hi, if you have time, please take a look~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
tomtongue commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500699369 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAddFilesProcedure.java: ## @@ -95,8 +97,8 @@ public static Object[][] parameters(

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
tomtongue commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500706263 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveOrphanFilesProcedure.java: ## @@ -18,8 +18,10 @@ */ package org.apache.i

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
tomtongue commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500706263 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveOrphanFilesProcedure.java: ## @@ -18,8 +18,10 @@ */ package org.apache.i

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
tomtongue commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500706263 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveOrphanFilesProcedure.java: ## @@ -18,8 +18,10 @@ */ package org.apache.i

Re: [I] Support rewrite_manifests [iceberg-python]

2024-02-23 Thread via GitHub
Fokko commented on issue #465: URL: https://github.com/apache/iceberg-python/issues/465#issuecomment-1961385834 @nicor88 It is recorded here https://github.com/apache/iceberg-python/issues/270 It is high on the list since with the fast-appends we're creating quite a bit of metadata. -- T

Re: [I] Support rewrite_manifests [iceberg-python]

2024-02-23 Thread via GitHub
nicor88 commented on issue #465: URL: https://github.com/apache/iceberg-python/issues/465#issuecomment-1961391888 Ah thanks, I did a search on rewrite_manifests, somehow https://github.com/apache/iceberg-python/issues/270 didn't pop up. I believe that this can be a duplicated then WDY

Re: [I] Support rewrite_manifests [iceberg-python]

2024-02-23 Thread via GitHub
Fokko closed issue #465: Support rewrite_manifests URL: https://github.com/apache/iceberg-python/issues/465 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-ma

Re: [I] Support rewrite_manifests [iceberg-python]

2024-02-23 Thread via GitHub
Fokko commented on issue #465: URL: https://github.com/apache/iceberg-python/issues/465#issuecomment-1961396293 I just updated the title to clarify it :) I think this is a duplicate indeed! Lets track it in the other issue -- This is an automated message from the Apache Git Service. To re

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
tomtongue commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500720854 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveOrphanFilesProcedure.java: ## @@ -18,8 +18,10 @@ */ package org.apache.i

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500777860 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveOrphanFilesProcedure.java: ## @@ -18,8 +18,10 @@ */ package org.apache.iceb

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
tomtongue commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500780330 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveOrphanFilesProcedure.java: ## @@ -18,8 +18,10 @@ */ package org.apache.i

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
tomtongue commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500786315 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveOrphanFilesProcedure.java: ## @@ -18,8 +18,10 @@ */ package org.apache.i

Re: [I] Support different JDBC backend in the `JdbcCatalog` [iceberg]

2024-02-23 Thread via GitHub
jbonofre commented on issue #9733: URL: https://github.com/apache/iceberg/issues/9733#issuecomment-1961536143 in Apache Karaf and Pax JDBC, I experimented to use jdbi (https://jdbi.org/) to abstract the different JDBC backend. I'm testing this approach for the Iceberg `JdbcCatalog`. --

Re: [PR] Docs: Sync specs to site via symlinks [iceberg]

2024-02-23 Thread via GitHub
manuzhang commented on PR #9779: URL: https://github.com/apache/iceberg/pull/9779#issuecomment-1961536940 @nastra Updated! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
tomtongue commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500780330 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveOrphanFilesProcedure.java: ## @@ -18,8 +18,10 @@ */ package org.apache.i

Re: [PR] Docs: Sync specs to site via symlinks [iceberg]

2024-02-23 Thread via GitHub
nastra merged PR #9779: URL: https://github.com/apache/iceberg/pull/9779 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Add filter pushdown API for contains and endsWith [iceberg]

2024-02-23 Thread via GitHub
amogh-jahagirdar commented on PR #9710: URL: https://github.com/apache/iceberg/pull/9710#issuecomment-1961629871 Sorry for the delay @yabola , got busy with some other work, I haven't forgotten about this PR! When my time frees up I'll review this. -- This is an automated message from the

Re: [PR] Spark: Use current namespace for SHOW VIEWS cmd [iceberg]

2024-02-23 Thread via GitHub
rdblue commented on PR #9787: URL: https://github.com/apache/iceberg/pull/9787#issuecomment-1961643716 @nastra, could you add tests for this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500913132 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -90,13 +94,13 @@ public class RewriteManifestsSparkAction

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500913132 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -90,13 +94,13 @@ public class RewriteManifestsSparkAction

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500915765 ## api/src/main/java/org/apache/iceberg/actions/RewriteManifests.java: ## @@ -44,6 +45,20 @@ public interface RewriteManifests */ RewriteManifests rewriteIf(Pre

Re: [PR] Deprecate DynamoDB Catalog to Reduce Catalog Scope [iceberg]

2024-02-23 Thread via GitHub
jackye1995 commented on PR #9783: URL: https://github.com/apache/iceberg/pull/9783#issuecomment-1961650652 Thanks for the help, merging! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Deprecate DynamoDB Catalog to Reduce Catalog Scope [iceberg]

2024-02-23 Thread via GitHub
jackye1995 merged PR #9783: URL: https://github.com/apache/iceberg/pull/9783 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500922017 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -250,12 +282,40 @@ private List writeUnpartitionedManifests(

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-23 Thread via GitHub
jackye1995 commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1500924999 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1607,17 @@ components: type: string example: [ "accounting", "tax" ] +PageToken: +

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500927289 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -250,12 +282,40 @@ private List writeUnpartitionedManifests(

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500927289 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -250,12 +282,40 @@ private List writeUnpartitionedManifests(

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500932975 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +471,163 @@ public void testRewriteLargeManifestsParti

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500936484 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +471,163 @@ public void testRewriteLargeManifestsParti

Re: [PR] Deprecate DynamoDB Catalog to Reduce Catalog Scope [iceberg]

2024-02-23 Thread via GitHub
nastra commented on PR #9783: URL: https://github.com/apache/iceberg/pull/9783#issuecomment-1961679888 @jackye1995 as I mentioned we should be holding off the merge until 1.5.0 is out. If we do another RC, then this diff will be included and I'm not sure this is what we wanted -- This is

Re: [PR] Deprecate DynamoDB Catalog to Reduce Catalog Scope [iceberg]

2024-02-23 Thread via GitHub
jackye1995 commented on PR #9783: URL: https://github.com/apache/iceberg/pull/9783#issuecomment-1961703475 Oh sorry I totally missed that comment. I think we can put another PR to change the comment to say deprecated since 1.5.0, what do you think? -- This is an automated message from the

Re: [PR] Deprecate DynamoDB Catalog to Reduce Catalog Scope [iceberg]

2024-02-23 Thread via GitHub
nastra commented on PR #9783: URL: https://github.com/apache/iceberg/pull/9783#issuecomment-1961712021 That might work too. Also do we want to update `aws.md` since it has some examples with that catalog? -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [I] Support Table Migration features from Spark [iceberg-python]

2024-02-23 Thread via GitHub
kevinjqliu commented on issue #354: URL: https://github.com/apache/iceberg-python/issues/354#issuecomment-1961759374 hey @syun64, yes definitely go ahead! I think that command will be very helpful. Please let me know if I can help in any way. -- This is an automated message from the

Re: [PR] Deprecate DynamoDB Catalog to Reduce Catalog Scope [iceberg]

2024-02-23 Thread via GitHub
jackye1995 commented on PR #9783: URL: https://github.com/apache/iceberg/pull/9783#issuecomment-1961768557 I see, let's do that quickly @geruh -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Writer Design [iceberg-rust]

2024-02-23 Thread via GitHub
lukekim commented on issue #34: URL: https://github.com/apache/iceberg-rust/issues/34#issuecomment-1961809931 @ZENOTME got it, thanks. We may be able to help here if you create some discrete sub-issues for the conversion. -- This is an automated message from the Apache Git Service. To res

[PR] Adjust Deprecation Version for DynamoDB Catalog to 1.5.0 [iceberg]

2024-02-23 Thread via GitHub
geruh opened a new pull request, #9788: URL: https://github.com/apache/iceberg/pull/9788 Fast follow update to correct the planned deprecation version for the `DynamoDbCatalog`. The previous PR set the deprecation at version 1.6.0. This change brings it forward to 1.5.0 to account for poten

Re: [PR] Adjust Deprecation Version for DynamoDB Catalog to 1.5.0 [iceberg]

2024-02-23 Thread via GitHub
jackye1995 commented on PR #9788: URL: https://github.com/apache/iceberg/pull/9788#issuecomment-1961856685 @nastra could you take a look if this looks good? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
zachdisc commented on PR #9731: URL: https://github.com/apache/iceberg/pull/9731#issuecomment-1961864865 ## Rev 3 * Addressed last round of comments * Added a new API option to sort with a custom supplied `Function`. Illustration (in unit test) - say you have a table that is pa

Re: [PR] Adjust Deprecation Version for DynamoDB Catalog to 1.5.0 [iceberg]

2024-02-23 Thread via GitHub
jackye1995 commented on PR #9788: URL: https://github.com/apache/iceberg/pull/9788#issuecomment-1961875767 Thanks for the quick fix and review! Merging -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Adjust Deprecation Version for DynamoDB Catalog to 1.5.0 [iceberg]

2024-02-23 Thread via GitHub
jackye1995 merged PR #9788: URL: https://github.com/apache/iceberg/pull/9788 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Spark: Iceberg: java.io.InvalidClassException: org.apache.iceberg.Schema; local class incompatible: stream classdesc serialVersionUID = 3320367012418887609, local class serialVersionUID = -885

2024-02-23 Thread via GitHub
jornfranke commented on issue #5970: URL: https://github.com/apache/iceberg/issues/5970#issuecomment-1961905762 Not 100% sure (as now also the environment evolved), but we had also no other explanation. -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] Core: Fix list tables/views behavior with empty namespace / Use current namespace for SHOW VIEWS cmd [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1501124041 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -1179,4 +1179,12 @@ public void testDatabaseLocationWithSlashInWarehouseDir() {

Re: [PR] [WIP] Bin Pack Writes [iceberg-python]

2024-02-23 Thread via GitHub
kevinjqliu commented on PR #444: URL: https://github.com/apache/iceberg-python/pull/444#issuecomment-1961952840 @Fokko PRs ready for review. Please give it a try. I've linked an example notebook in the PR description. I've also noticed that writing one RecordBatch at a time seems to

Re: [PR] Core: Fix list tables/views behavior with empty namespace / Use current namespace for SHOW VIEWS cmd [iceberg]

2024-02-23 Thread via GitHub
nastra commented on PR #9787: URL: https://github.com/apache/iceberg/pull/9787#issuecomment-1961965465 @rdblue I've split this into two commits. In the first commit you can see that once the behavior in `InMemoryCatalog` is fixed, running `SHOW VIEWS` or `SHOW VIEWS in ` fails as expected a

Re: [PR] Core: Fix list tables/views behavior with empty namespace / Use current namespace for SHOW VIEWS cmd [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1501124041 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -1179,4 +1179,12 @@ public void testDatabaseLocationWithSlashInWarehouseDir() {

Re: [PR] Core: Fix list tables/views behavior with empty namespace / Use current namespace for SHOW VIEWS cmd [iceberg]

2024-02-23 Thread via GitHub
rdblue commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1501169076 ## core/src/main/java/org/apache/iceberg/inmemory/InMemoryCatalog.java: ## @@ -138,13 +138,13 @@ public boolean dropTable(TableIdentifier tableIdentifier, boolean purge

Re: [PR] Core: Fix list tables/views behavior with empty namespace / Use current namespace for SHOW VIEWS cmd [iceberg]

2024-02-23 Thread via GitHub
rdblue commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1501174113 ## core/src/main/java/org/apache/iceberg/inmemory/InMemoryCatalog.java: ## @@ -138,13 +138,13 @@ public boolean dropTable(TableIdentifier tableIdentifier, boolean purge

Re: [PR] Core: Fix list tables/views behavior with empty namespace / Use current namespace for SHOW VIEWS cmd [iceberg]

2024-02-23 Thread via GitHub
rdblue commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1501174544 ## core/src/main/java/org/apache/iceberg/inmemory/InMemoryCatalog.java: ## @@ -315,13 +315,13 @@ public void close() throws IOException { @Override public List l

Re: [PR] Core: Fix list tables/views behavior with empty namespace / Use current namespace for SHOW VIEWS cmd [iceberg]

2024-02-23 Thread via GitHub
rdblue commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1501176399 ## core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java: ## @@ -987,6 +987,13 @@ public void testListTables() { assertEmpty("Should not contain ns_2.tabl

Re: [PR] Core: Fix list tables/views behavior with empty namespace / Use current namespace for SHOW VIEWS cmd [iceberg]

2024-02-23 Thread via GitHub
rdblue commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1501179380 ## spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1325,22 +1324,35 @@ public void showViews() throws NoSuchTableExc

Re: [PR] Flink 1.18: Fix continuous enumerator lost enumeration history state when restore from checkpoint. [iceberg]

2024-02-23 Thread via GitHub
mas-chen commented on PR #9762: URL: https://github.com/apache/iceberg/pull/9762#issuecomment-1962046987 @Reo-LEI we are also hitting this issue. could you also backport it to 1.17? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] Spark: Detect temp functions in views [iceberg]

2024-02-23 Thread via GitHub
rdblue commented on code in PR #9675: URL: https://github.com/apache/iceberg/pull/9675#discussion_r1501265008 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -102,20 +113,23 @@ case class RewriteViewCommands(s

Re: [PR] Spark: Detect temp functions in views [iceberg]

2024-02-23 Thread via GitHub
rdblue commented on code in PR #9675: URL: https://github.com/apache/iceberg/pull/9675#discussion_r1501265639 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -149,4 +163,20 @@ case class RewriteViewCommands(sp

Re: [PR] Spark: Detect temp functions in views [iceberg]

2024-02-23 Thread via GitHub
rdblue commented on code in PR #9675: URL: https://github.com/apache/iceberg/pull/9675#discussion_r1501265932 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -102,20 +113,23 @@ case class RewriteViewCommands(s

Re: [PR] Spark: Detect temp functions in views [iceberg]

2024-02-23 Thread via GitHub
rdblue commented on code in PR #9675: URL: https://github.com/apache/iceberg/pull/9675#discussion_r1501266819 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -471,6 +471,39 @@ public void readFromViewReferencingGlobalTempVie

Re: [PR] Spark: Detect temp functions in views [iceberg]

2024-02-23 Thread via GitHub
rdblue commented on code in PR #9675: URL: https://github.com/apache/iceberg/pull/9675#discussion_r1501265008 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -102,20 +113,23 @@ case class RewriteViewCommands(s

Re: [I] Parallel Table.append [iceberg-python]

2024-02-23 Thread via GitHub
kevinjqliu commented on issue #428: URL: https://github.com/apache/iceberg-python/issues/428#issuecomment-1962142370 @bigluck #444 should allow parallelized write. Do you mind giving it a try? -- This is an automated message from the Apache Git Service. To respond to the message, please l

[PR] Make issued_token_type optional to support OAuth2 Client Credential Flow [iceberg-python]

2024-02-23 Thread via GitHub
flyrain opened a new pull request, #466: URL: https://github.com/apache/iceberg-python/pull/466 To fix issue(https://github.com/apache/iceberg-python/issues/463). In short, Client Credential Flow's http response doesn't have the field `issued_token_type`. Make it optional to avoid validatio

Re: [I] Implement the split logic in DataTask [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] closed issue #976: Implement the split logic in DataTask URL: https://github.com/apache/iceberg/issues/976 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [I] Implement the split logic in DataTask [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] commented on issue #976: URL: https://github.com/apache/iceberg/issues/976#issuecomment-1962161030 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git

Re: [I] New warnings when building Iceberg [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] commented on issue #988: URL: https://github.com/apache/iceberg/issues/988#issuecomment-1962161051 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git

Re: [I] official site none offer get start and in action document [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] closed issue #1009: official site none offer get start and in action document URL: https://github.com/apache/iceberg/issues/1009 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [I] [Update notification follow up] We should publish update events only when a transaction succeeds [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] closed issue #998: [Update notification follow up] We should publish update events only when a transaction succeeds URL: https://github.com/apache/iceberg/issues/998 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [I] [Update notification follow up] Add remaining events for SnapshotManager [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] commented on issue #997: URL: https://github.com/apache/iceberg/issues/997#issuecomment-1962161063 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git

Re: [I] [Update notification follow up] Add remaining events for SnapshotManager [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] closed issue #997: [Update notification follow up] Add remaining events for SnapshotManager URL: https://github.com/apache/iceberg/issues/997 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [I] [Update notification follow up] We should publish update events only when a transaction succeeds [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] commented on issue #998: URL: https://github.com/apache/iceberg/issues/998#issuecomment-1962161084 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git

Re: [I] New warnings when building Iceberg [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] closed issue #988: New warnings when building Iceberg URL: https://github.com/apache/iceberg/issues/988 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [I] Add a merge-based row filter for equality deletes [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] closed issue #1024: Add a merge-based row filter for equality deletes URL: https://github.com/apache/iceberg/issues/1024 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

  1   2   >