Re: [PR] Flink: Move ParquetReader to LogicalTypeAnnotationVisitor [iceberg]

2024-02-23 Thread via GitHub
pvary commented on PR #9719: URL: https://github.com/apache/iceberg/pull/9719#issuecomment-1962275482 Could we add a test too? Minimally for the new timestamp? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] Flink: Move ParquetReader to LogicalTypeAnnotationVisitor [iceberg]

2024-02-23 Thread via GitHub
pvary commented on code in PR #9719: URL: https://github.com/apache/iceberg/pull/9719#discussion_r1501356650 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/data/FlinkParquetReaders.java: ## @@ -193,6 +195,122 @@ public ParquetValueReader map( ParquetValue

Re: [PR] Flink: Move ParquetReader to LogicalTypeAnnotationVisitor [iceberg]

2024-02-23 Thread via GitHub
pvary commented on code in PR #9719: URL: https://github.com/apache/iceberg/pull/9719#discussion_r1501356608 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/data/FlinkParquetReaders.java: ## @@ -193,6 +195,122 @@ public ParquetValueReader map( ParquetValue

Re: [PR] Flink: Made IcebergFilesCommitter work with single phase commit [iceberg]

2024-02-23 Thread via GitHub
pvary commented on PR #9694: URL: https://github.com/apache/iceberg/pull/9694#issuecomment-1962273864 How would the single phase commit solution support the exact one semantics for the Iceberg Sink? -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [I] Spark read failed when migrate hive orc table with `timestamp` column [iceberg]

2024-02-23 Thread via GitHub
ming95 commented on issue #9784: URL: https://github.com/apache/iceberg/issues/9784#issuecomment-1962271191 @RussellSpitzer can you take a look on this issue? thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Spark: Fix SparkTable to use name and effective snapshotID for comparing [iceberg]

2024-02-23 Thread via GitHub
wooyeong commented on PR #9455: URL: https://github.com/apache/iceberg/pull/9455#issuecomment-1962260320 @aokolnychyi Sorry for the late, I updated it. Plus I added a test to check related to main branch. Thank you for the comment. -- This is an automated message from the Apache Git Servi

Re: [PR] Make issued_token_type optional to support OAuth2 Client Credential Flow [iceberg-python]

2024-02-23 Thread via GitHub
syun64 commented on code in PR #466: URL: https://github.com/apache/iceberg-python/pull/466#discussion_r1501297005 ## pyiceberg/catalog/rest.py: ## @@ -157,7 +157,7 @@ class TokenResponse(IcebergBaseModel): access_token: str = Field() token_type: str = Field() exp

Re: [I] Create table from plain Parquet files [iceberg-python]

2024-02-23 Thread via GitHub
syun64 commented on issue #445: URL: https://github.com/apache/iceberg-python/issues/445#issuecomment-1962179349 @Fokko would this process be similar to running the following steps? 1. create_table from PyArrow schema 2. invoking [add_files](https://iceberg.apache.org/docs/latest/s

Re: [I] Support iceberg hadoop catalog in python library [iceberg-python]

2024-02-23 Thread via GitHub
corleyma commented on issue #17: URL: https://github.com/apache/iceberg-python/issues/17#issuecomment-1962179002 This would really help us out, where we use Hadoop catalog for unit testing PySpark code, and are increasingly encountering cases where we want to test code that uses both pyiceb

Re: [I] official site none offer get start and in action document [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] commented on issue #1009: URL: https://github.com/apache/iceberg/issues/1009#issuecomment-1962161108 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Add an action to rewrite equality deletes as position deletes [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] commented on issue #1026: URL: https://github.com/apache/iceberg/issues/1026#issuecomment-1962161136 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] [SPARK-3] Extend iceberg Catalog interface with namespace operations [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] commented on issue #1039: URL: https://github.com/apache/iceberg/issues/1039#issuecomment-1962161147 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Add an action to rewrite equality deletes as position deletes [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] closed issue #1026: Add an action to rewrite equality deletes as position deletes URL: https://github.com/apache/iceberg/issues/1026 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] [SPARK-3] Extend iceberg Catalog interface with namespace operations [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] closed issue #1039: [SPARK-3] Extend iceberg Catalog interface with namespace operations URL: https://github.com/apache/iceberg/issues/1039 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] Add a merge-based row filter for equality deletes [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] closed issue #1024: Add a merge-based row filter for equality deletes URL: https://github.com/apache/iceberg/issues/1024 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [I] New warnings when building Iceberg [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] closed issue #988: New warnings when building Iceberg URL: https://github.com/apache/iceberg/issues/988 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [I] [Update notification follow up] We should publish update events only when a transaction succeeds [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] commented on issue #998: URL: https://github.com/apache/iceberg/issues/998#issuecomment-1962161084 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git

Re: [I] [Update notification follow up] Add remaining events for SnapshotManager [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] commented on issue #997: URL: https://github.com/apache/iceberg/issues/997#issuecomment-1962161063 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git

Re: [I] [Update notification follow up] Add remaining events for SnapshotManager [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] closed issue #997: [Update notification follow up] Add remaining events for SnapshotManager URL: https://github.com/apache/iceberg/issues/997 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [I] official site none offer get start and in action document [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] closed issue #1009: official site none offer get start and in action document URL: https://github.com/apache/iceberg/issues/1009 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [I] [Update notification follow up] We should publish update events only when a transaction succeeds [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] closed issue #998: [Update notification follow up] We should publish update events only when a transaction succeeds URL: https://github.com/apache/iceberg/issues/998 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [I] New warnings when building Iceberg [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] commented on issue #988: URL: https://github.com/apache/iceberg/issues/988#issuecomment-1962161051 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git

Re: [I] Implement the split logic in DataTask [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] commented on issue #976: URL: https://github.com/apache/iceberg/issues/976#issuecomment-1962161030 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git

Re: [I] Implement the split logic in DataTask [iceberg]

2024-02-23 Thread via GitHub
github-actions[bot] closed issue #976: Implement the split logic in DataTask URL: https://github.com/apache/iceberg/issues/976 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

[PR] Make issued_token_type optional to support OAuth2 Client Credential Flow [iceberg-python]

2024-02-23 Thread via GitHub
flyrain opened a new pull request, #466: URL: https://github.com/apache/iceberg-python/pull/466 To fix issue(https://github.com/apache/iceberg-python/issues/463). In short, Client Credential Flow's http response doesn't have the field `issued_token_type`. Make it optional to avoid validatio

Re: [I] Parallel Table.append [iceberg-python]

2024-02-23 Thread via GitHub
kevinjqliu commented on issue #428: URL: https://github.com/apache/iceberg-python/issues/428#issuecomment-1962142370 @bigluck #444 should allow parallelized write. Do you mind giving it a try? -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [PR] Spark: Detect temp functions in views [iceberg]

2024-02-23 Thread via GitHub
rdblue commented on code in PR #9675: URL: https://github.com/apache/iceberg/pull/9675#discussion_r1501265008 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -102,20 +113,23 @@ case class RewriteViewCommands(s

Re: [PR] Spark: Detect temp functions in views [iceberg]

2024-02-23 Thread via GitHub
rdblue commented on code in PR #9675: URL: https://github.com/apache/iceberg/pull/9675#discussion_r1501266819 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -471,6 +471,39 @@ public void readFromViewReferencingGlobalTempVie

Re: [PR] Spark: Detect temp functions in views [iceberg]

2024-02-23 Thread via GitHub
rdblue commented on code in PR #9675: URL: https://github.com/apache/iceberg/pull/9675#discussion_r1501265932 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -102,20 +113,23 @@ case class RewriteViewCommands(s

Re: [PR] Spark: Detect temp functions in views [iceberg]

2024-02-23 Thread via GitHub
rdblue commented on code in PR #9675: URL: https://github.com/apache/iceberg/pull/9675#discussion_r1501265639 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -149,4 +163,20 @@ case class RewriteViewCommands(sp

Re: [PR] Spark: Detect temp functions in views [iceberg]

2024-02-23 Thread via GitHub
rdblue commented on code in PR #9675: URL: https://github.com/apache/iceberg/pull/9675#discussion_r1501265008 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteViewCommands.scala: ## @@ -102,20 +113,23 @@ case class RewriteViewCommands(s

Re: [PR] Flink 1.18: Fix continuous enumerator lost enumeration history state when restore from checkpoint. [iceberg]

2024-02-23 Thread via GitHub
mas-chen commented on PR #9762: URL: https://github.com/apache/iceberg/pull/9762#issuecomment-1962046987 @Reo-LEI we are also hitting this issue. could you also backport it to 1.17? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] Core: Fix list tables/views behavior with empty namespace / Use current namespace for SHOW VIEWS cmd [iceberg]

2024-02-23 Thread via GitHub
rdblue commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1501179380 ## spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1325,22 +1324,35 @@ public void showViews() throws NoSuchTableExc

Re: [PR] Core: Fix list tables/views behavior with empty namespace / Use current namespace for SHOW VIEWS cmd [iceberg]

2024-02-23 Thread via GitHub
rdblue commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1501176399 ## core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java: ## @@ -987,6 +987,13 @@ public void testListTables() { assertEmpty("Should not contain ns_2.tabl

Re: [PR] Core: Fix list tables/views behavior with empty namespace / Use current namespace for SHOW VIEWS cmd [iceberg]

2024-02-23 Thread via GitHub
rdblue commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1501174113 ## core/src/main/java/org/apache/iceberg/inmemory/InMemoryCatalog.java: ## @@ -138,13 +138,13 @@ public boolean dropTable(TableIdentifier tableIdentifier, boolean purge

Re: [PR] Core: Fix list tables/views behavior with empty namespace / Use current namespace for SHOW VIEWS cmd [iceberg]

2024-02-23 Thread via GitHub
rdblue commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1501174544 ## core/src/main/java/org/apache/iceberg/inmemory/InMemoryCatalog.java: ## @@ -315,13 +315,13 @@ public void close() throws IOException { @Override public List l

Re: [PR] Core: Fix list tables/views behavior with empty namespace / Use current namespace for SHOW VIEWS cmd [iceberg]

2024-02-23 Thread via GitHub
rdblue commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1501169076 ## core/src/main/java/org/apache/iceberg/inmemory/InMemoryCatalog.java: ## @@ -138,13 +138,13 @@ public boolean dropTable(TableIdentifier tableIdentifier, boolean purge

Re: [PR] Core: Fix list tables/views behavior with empty namespace / Use current namespace for SHOW VIEWS cmd [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1501124041 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -1179,4 +1179,12 @@ public void testDatabaseLocationWithSlashInWarehouseDir() {

Re: [PR] Core: Fix list tables/views behavior with empty namespace / Use current namespace for SHOW VIEWS cmd [iceberg]

2024-02-23 Thread via GitHub
nastra commented on PR #9787: URL: https://github.com/apache/iceberg/pull/9787#issuecomment-1961965465 @rdblue I've split this into two commits. In the first commit you can see that once the behavior in `InMemoryCatalog` is fixed, running `SHOW VIEWS` or `SHOW VIEWS in ` fails as expected a

Re: [PR] [WIP] Bin Pack Writes [iceberg-python]

2024-02-23 Thread via GitHub
kevinjqliu commented on PR #444: URL: https://github.com/apache/iceberg-python/pull/444#issuecomment-1961952840 @Fokko PRs ready for review. Please give it a try. I've linked an example notebook in the PR description. I've also noticed that writing one RecordBatch at a time seems to

Re: [PR] Core: Fix list tables/views behavior with empty namespace / Use current namespace for SHOW VIEWS cmd [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9787: URL: https://github.com/apache/iceberg/pull/9787#discussion_r1501124041 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -1179,4 +1179,12 @@ public void testDatabaseLocationWithSlashInWarehouseDir() {

Re: [I] Spark: Iceberg: java.io.InvalidClassException: org.apache.iceberg.Schema; local class incompatible: stream classdesc serialVersionUID = 3320367012418887609, local class serialVersionUID = -885

2024-02-23 Thread via GitHub
jornfranke commented on issue #5970: URL: https://github.com/apache/iceberg/issues/5970#issuecomment-1961905762 Not 100% sure (as now also the environment evolved), but we had also no other explanation. -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] Adjust Deprecation Version for DynamoDB Catalog to 1.5.0 [iceberg]

2024-02-23 Thread via GitHub
jackye1995 merged PR #9788: URL: https://github.com/apache/iceberg/pull/9788 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Adjust Deprecation Version for DynamoDB Catalog to 1.5.0 [iceberg]

2024-02-23 Thread via GitHub
jackye1995 commented on PR #9788: URL: https://github.com/apache/iceberg/pull/9788#issuecomment-1961875767 Thanks for the quick fix and review! Merging -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
zachdisc commented on PR #9731: URL: https://github.com/apache/iceberg/pull/9731#issuecomment-1961864865 ## Rev 3 * Addressed last round of comments * Added a new API option to sort with a custom supplied `Function`. Illustration (in unit test) - say you have a table that is pa

Re: [PR] Adjust Deprecation Version for DynamoDB Catalog to 1.5.0 [iceberg]

2024-02-23 Thread via GitHub
jackye1995 commented on PR #9788: URL: https://github.com/apache/iceberg/pull/9788#issuecomment-1961856685 @nastra could you take a look if this looks good? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[PR] Adjust Deprecation Version for DynamoDB Catalog to 1.5.0 [iceberg]

2024-02-23 Thread via GitHub
geruh opened a new pull request, #9788: URL: https://github.com/apache/iceberg/pull/9788 Fast follow update to correct the planned deprecation version for the `DynamoDbCatalog`. The previous PR set the deprecation at version 1.6.0. This change brings it forward to 1.5.0 to account for poten

Re: [I] Writer Design [iceberg-rust]

2024-02-23 Thread via GitHub
lukekim commented on issue #34: URL: https://github.com/apache/iceberg-rust/issues/34#issuecomment-1961809931 @ZENOTME got it, thanks. We may be able to help here if you create some discrete sub-issues for the conversion. -- This is an automated message from the Apache Git Service. To res

Re: [PR] Deprecate DynamoDB Catalog to Reduce Catalog Scope [iceberg]

2024-02-23 Thread via GitHub
jackye1995 commented on PR #9783: URL: https://github.com/apache/iceberg/pull/9783#issuecomment-1961768557 I see, let's do that quickly @geruh -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Support Table Migration features from Spark [iceberg-python]

2024-02-23 Thread via GitHub
kevinjqliu commented on issue #354: URL: https://github.com/apache/iceberg-python/issues/354#issuecomment-1961759374 hey @syun64, yes definitely go ahead! I think that command will be very helpful. Please let me know if I can help in any way. -- This is an automated message from the

Re: [PR] Deprecate DynamoDB Catalog to Reduce Catalog Scope [iceberg]

2024-02-23 Thread via GitHub
nastra commented on PR #9783: URL: https://github.com/apache/iceberg/pull/9783#issuecomment-1961712021 That might work too. Also do we want to update `aws.md` since it has some examples with that catalog? -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [PR] Deprecate DynamoDB Catalog to Reduce Catalog Scope [iceberg]

2024-02-23 Thread via GitHub
jackye1995 commented on PR #9783: URL: https://github.com/apache/iceberg/pull/9783#issuecomment-1961703475 Oh sorry I totally missed that comment. I think we can put another PR to change the comment to say deprecated since 1.5.0, what do you think? -- This is an automated message from the

Re: [PR] Deprecate DynamoDB Catalog to Reduce Catalog Scope [iceberg]

2024-02-23 Thread via GitHub
nastra commented on PR #9783: URL: https://github.com/apache/iceberg/pull/9783#issuecomment-1961679888 @jackye1995 as I mentioned we should be holding off the merge until 1.5.0 is out. If we do another RC, then this diff will be included and I'm not sure this is what we wanted -- This is

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500936484 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +471,163 @@ public void testRewriteLargeManifestsParti

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500932975 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +471,163 @@ public void testRewriteLargeManifestsParti

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500927289 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -250,12 +282,40 @@ private List writeUnpartitionedManifests(

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500927289 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -250,12 +282,40 @@ private List writeUnpartitionedManifests(

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-23 Thread via GitHub
jackye1995 commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1500924999 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1607,17 @@ components: type: string example: [ "accounting", "tax" ] +PageToken: +

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500922017 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -250,12 +282,40 @@ private List writeUnpartitionedManifests(

Re: [PR] Deprecate DynamoDB Catalog to Reduce Catalog Scope [iceberg]

2024-02-23 Thread via GitHub
jackye1995 merged PR #9783: URL: https://github.com/apache/iceberg/pull/9783 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Deprecate DynamoDB Catalog to Reduce Catalog Scope [iceberg]

2024-02-23 Thread via GitHub
jackye1995 commented on PR #9783: URL: https://github.com/apache/iceberg/pull/9783#issuecomment-1961650652 Thanks for the help, merging! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500915765 ## api/src/main/java/org/apache/iceberg/actions/RewriteManifests.java: ## @@ -44,6 +45,20 @@ public interface RewriteManifests */ RewriteManifests rewriteIf(Pre

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500913132 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -90,13 +94,13 @@ public class RewriteManifestsSparkAction

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500913132 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -90,13 +94,13 @@ public class RewriteManifestsSparkAction

Re: [PR] Spark: Use current namespace for SHOW VIEWS cmd [iceberg]

2024-02-23 Thread via GitHub
rdblue commented on PR #9787: URL: https://github.com/apache/iceberg/pull/9787#issuecomment-1961643716 @nastra, could you add tests for this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Add filter pushdown API for contains and endsWith [iceberg]

2024-02-23 Thread via GitHub
amogh-jahagirdar commented on PR #9710: URL: https://github.com/apache/iceberg/pull/9710#issuecomment-1961629871 Sorry for the delay @yabola , got busy with some other work, I haven't forgotten about this PR! When my time frees up I'll review this. -- This is an automated message from the

Re: [PR] Docs: Sync specs to site via symlinks [iceberg]

2024-02-23 Thread via GitHub
nastra merged PR #9779: URL: https://github.com/apache/iceberg/pull/9779 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
tomtongue commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500780330 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveOrphanFilesProcedure.java: ## @@ -18,8 +18,10 @@ */ package org.apache.i

Re: [PR] Docs: Sync specs to site via symlinks [iceberg]

2024-02-23 Thread via GitHub
manuzhang commented on PR #9779: URL: https://github.com/apache/iceberg/pull/9779#issuecomment-1961536940 @nastra Updated! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [I] Support different JDBC backend in the `JdbcCatalog` [iceberg]

2024-02-23 Thread via GitHub
jbonofre commented on issue #9733: URL: https://github.com/apache/iceberg/issues/9733#issuecomment-1961536143 in Apache Karaf and Pax JDBC, I experimented to use jdbi (https://jdbi.org/) to abstract the different JDBC backend. I'm testing this approach for the Iceberg `JdbcCatalog`. --

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
tomtongue commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500786315 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveOrphanFilesProcedure.java: ## @@ -18,8 +18,10 @@ */ package org.apache.i

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
tomtongue commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500780330 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveOrphanFilesProcedure.java: ## @@ -18,8 +18,10 @@ */ package org.apache.i

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500777860 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveOrphanFilesProcedure.java: ## @@ -18,8 +18,10 @@ */ package org.apache.iceb

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
tomtongue commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500720854 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveOrphanFilesProcedure.java: ## @@ -18,8 +18,10 @@ */ package org.apache.i

Re: [I] Support rewrite_manifests [iceberg-python]

2024-02-23 Thread via GitHub
Fokko commented on issue #465: URL: https://github.com/apache/iceberg-python/issues/465#issuecomment-1961396293 I just updated the title to clarify it :) I think this is a duplicate indeed! Lets track it in the other issue -- This is an automated message from the Apache Git Service. To re

Re: [I] Support rewrite_manifests [iceberg-python]

2024-02-23 Thread via GitHub
Fokko closed issue #465: Support rewrite_manifests URL: https://github.com/apache/iceberg-python/issues/465 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-ma

Re: [I] Support rewrite_manifests [iceberg-python]

2024-02-23 Thread via GitHub
nicor88 commented on issue #465: URL: https://github.com/apache/iceberg-python/issues/465#issuecomment-1961391888 Ah thanks, I did a search on rewrite_manifests, somehow https://github.com/apache/iceberg-python/issues/270 didn't pop up. I believe that this can be a duplicated then WDY

Re: [I] Support rewrite_manifests [iceberg-python]

2024-02-23 Thread via GitHub
Fokko commented on issue #465: URL: https://github.com/apache/iceberg-python/issues/465#issuecomment-1961385834 @nicor88 It is recorded here https://github.com/apache/iceberg-python/issues/270 It is high on the list since with the fast-appends we're creating quite a bit of metadata. -- T

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
tomtongue commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500706263 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveOrphanFilesProcedure.java: ## @@ -18,8 +18,10 @@ */ package org.apache.i

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
tomtongue commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500706263 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveOrphanFilesProcedure.java: ## @@ -18,8 +18,10 @@ */ package org.apache.i

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
tomtongue commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500706263 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveOrphanFilesProcedure.java: ## @@ -18,8 +18,10 @@ */ package org.apache.i

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
tomtongue commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500699369 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAddFilesProcedure.java: ## @@ -95,8 +97,8 @@ public static Object[][] parameters(

Re: [PR] Add filter pushdown API for contains and endsWith [iceberg]

2024-02-23 Thread via GitHub
yabola commented on PR #9710: URL: https://github.com/apache/iceberg/pull/9710#issuecomment-1961363628 @amogh-jahagirdar Hi, if you have time, please take a look~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500691064 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveOrphanFilesProcedure.java: ## @@ -18,8 +18,10 @@ */ package org.apache.iceb

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500677979 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAddFilesProcedure.java: ## @@ -95,8 +97,8 @@ public static Object[][] parameters() {

Re: [PR] Docs: Sync specs to site via symlinks [iceberg]

2024-02-23 Thread via GitHub
nastra commented on PR #9779: URL: https://github.com/apache/iceberg/pull/9779#issuecomment-1961319538 I like that idea, but this doesn't render the Spec site correctly when running the site locally. This is because `{{% icebergVersion %}}` needs to be replaced with `{{ icebergVersion }}` s

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
tomtongue commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500651939 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMigrateTableProcedure.java: ## @@ -18,51 +18,45 @@ */ package org.apache.icebe

Re: [PR] Migrate Procedure sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-23 Thread via GitHub
tomtongue commented on code in PR #9760: URL: https://github.com/apache/iceberg/pull/9760#discussion_r1500651939 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMigrateTableProcedure.java: ## @@ -18,51 +18,45 @@ */ package org.apache.icebe

Re: [I] S3FileIO does not support Iceberg Cross-Region API Calls to Amazon S3 buckets [iceberg]

2024-02-23 Thread via GitHub
elmehdibelgasmi commented on issue #9785: URL: https://github.com/apache/iceberg/issues/9785#issuecomment-1961286970 > @elmehdibelgasmi would you be interested in opening up a PR and fixing this for S3FileIO? @nastra sure, will do. -- This is an automated message from the Apache Gi

Re: [I] S3FileIO does not support Iceberg Cross-Region API Calls to Amazon S3 buckets [iceberg]

2024-02-23 Thread via GitHub
nastra commented on issue #9785: URL: https://github.com/apache/iceberg/issues/9785#issuecomment-1961277571 @elmehdibelgasmi would you be interested in opening up a PR and fixing this for S3FileIO? -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [I] flink1.14.4+iceberg0.13.1+hive-metastore3.1.2+minio(S3) error! [iceberg]

2024-02-23 Thread via GitHub
adamyasharma2797 commented on issue #4743: URL: https://github.com/apache/iceberg/issues/4743#issuecomment-1961235915 Approach provided by @lvyanquan worked for me! Thanks!! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] ValidationException: Missing required files to delete [iceberg]

2024-02-23 Thread via GitHub
bk-mz closed issue #9741: ValidationException: Missing required files to delete URL: https://github.com/apache/iceberg/issues/9741 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[I] S3FileIO does not support Iceberg Cross-Region API Calls to Amazon S3 buckets [iceberg]

2024-02-23 Thread via GitHub
elmehdibelgasmi opened a new issue, #9785: URL: https://github.com/apache/iceberg/issues/9785 ### Apache Iceberg version None ### Query engine None ### Please describe the bug 🐞 ## Issue Description With **S3FileIO** implementation, Cross-Region API c

Re: [PR] chore(deps): Update derive_builder requirement from 0.13.0 to 0.20.0 [iceberg-rust]

2024-02-23 Thread via GitHub
Fokko merged PR #203: URL: https://github.com/apache/iceberg-rust/pull/203 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [I] Spark: Iceberg: java.io.InvalidClassException: org.apache.iceberg.Schema; local class incompatible: stream classdesc serialVersionUID = 3320367012418887609, local class serialVersionUID = -885

2024-02-23 Thread via GitHub
FWLamb commented on issue #5970: URL: https://github.com/apache/iceberg/issues/5970#issuecomment-1961093423 @jornfranke Hello, Did you finally confirm that there was an older version of the classpath that caused the issue? -- This is an automated message from the Apache Git Service. To re

Re: [PR] doc: Add download page [iceberg-rust]

2024-02-23 Thread via GitHub
Fokko merged PR #219: URL: https://github.com/apache/iceberg-rust/pull/219 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500426585 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +471,163 @@ public void testRewriteLargeManifestsPartiti

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500425302 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +471,163 @@ public void testRewriteLargeManifestsPartiti

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500424697 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +471,163 @@ public void testRewriteLargeManifestsPartiti

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-02-23 Thread via GitHub
nastra commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1500423875 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -466,6 +471,163 @@ public void testRewriteLargeManifestsPartiti

  1   2   >