[GitHub] [iceberg] zhangbutao commented on pull request #6653: API: Fix Transform backward compatibility in PartitionSpec

2023-01-31 Thread via GitHub
zhangbutao commented on PR #6653: URL: https://github.com/apache/iceberg/pull/6653#issuecomment-1409922846 > I am ok with either solution, however, noticed a potentially no-go comment in your PR and decided to take the same approach as @Fokko in #6220. Note this is a blocker for Hive Iceber

[GitHub] [iceberg] Fokko merged pull request #6705: Python: Update pyproject.toml to include dev folder.

2023-01-31 Thread via GitHub
Fokko merged PR #6705: URL: https://github.com/apache/iceberg/pull/6705 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[GitHub] [iceberg] Fokko commented on pull request #6705: Python: Update pyproject.toml to include dev folder.

2023-01-31 Thread via GitHub
Fokko commented on PR #6705: URL: https://github.com/apache/iceberg/pull/6705#issuecomment-1409962986 Thanks @amogh-jahagirdar for fixing this, and @jackye1995 for the review 👏🏻 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [iceberg] Fokko commented on pull request #6704: Python: Update pyproject.toml to include dev folder.

2023-01-31 Thread via GitHub
Fokko commented on PR #6704: URL: https://github.com/apache/iceberg/pull/6704#issuecomment-1409963309 Thanks @amogh-jahagirdar for fixing this, and @jackye1995 and @singhpk234 for the review 👏🏻 -- This is an automated message from the Apache Git Service. To respond to the message, pleas

[GitHub] [iceberg] Fokko merged pull request #6704: Python: Update pyproject.toml to include dev folder.

2023-01-31 Thread via GitHub
Fokko merged PR #6704: URL: https://github.com/apache/iceberg/pull/6704 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[GitHub] [iceberg] simhadri-g opened a new issue, #6709: BasicStats TOTAL_RECORDS_PROP does not update after records are deleted this results in unoptimised query plans.

2023-01-31 Thread via GitHub
simhadri-g opened a new issue, #6709: URL: https://github.com/apache/iceberg/issues/6709 ### Apache Iceberg version 1.1.0 (latest release) ### Query engine Hive ### Please describe the bug 🐞 **Issue:** - Hive planner uses basic stats for query plannin

[GitHub] [iceberg] youngxinler commented on pull request #6554: Parquet: Improve Test Coverage of RowGroupFilter Code with Nans #6518

2023-01-31 Thread via GitHub
youngxinler commented on PR #6554: URL: https://github.com/apache/iceberg/pull/6554#issuecomment-1410278806 please take a review if you have time. @RussellSpitzer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

[GitHub] [iceberg] nastra commented on issue #5993: Quick start docker-compose demo doesn't work

2023-01-31 Thread via GitHub
nastra commented on issue #5993: URL: https://github.com/apache/iceberg/issues/5993#issuecomment-1410476388 @zhangjiuyang1993 are you using `tabulario/iceberg-rest:0.1.0`? If so, you need to use the latest one with Iceberg 1.1.0 support, which is `tabulario/iceberg-rest:0.2.0` -- This is

[GitHub] [iceberg] nastra commented on issue #6708: Quick start docker-compose demo doesn't work

2023-01-31 Thread via GitHub
nastra commented on issue #6708: URL: https://github.com/apache/iceberg/issues/6708#issuecomment-1410481495 You need to upgrade `tabulario/iceberg-rest:0.1.0` (uses Iceberg 1.0.0) to `tabulario/iceberg-rest:0.2.0` (uses Iceberg 1.1.0). That should fix it. -- This is an automated message f

[GitHub] [iceberg] nastra commented on a diff in pull request #6701: Add missing `last-column-id` to spec

2023-01-31 Thread via GitHub
nastra commented on code in PR #6701: URL: https://github.com/apache/iceberg/pull/6701#discussion_r1092044493 ## open-api/rest-catalog-open-api.yaml: ## @@ -1541,6 +1541,8 @@ components: properties: snapshot: $ref: '#/components/schemas/Sna

[GitHub] [iceberg] danielcweeks merged pull request #6702: Python: Let pytest know that it shouldn't collect TestType

2023-01-31 Thread via GitHub
danielcweeks merged PR #6702: URL: https://github.com/apache/iceberg/pull/6702 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

[GitHub] [iceberg] snazy opened a new issue, #6710: REST-Catalog: missing conflict-checks for `dropTable` and `updateTable`

2023-01-31 Thread via GitHub
snazy opened a new issue, #6710: URL: https://github.com/apache/iceberg/issues/6710 ### Apache Iceberg version main (development) ### Query engine None ### Please describe the bug 🐞 Concurrent operations against the same table (like updating the same table

[GitHub] [iceberg] snazy opened a new issue, #6711: REST-Catalog: `Table.newTransaction()....commit()` does nothing

2023-01-31 Thread via GitHub
snazy opened a new issue, #6711: URL: https://github.com/apache/iceberg/issues/6711 ### Apache Iceberg version main (development) ### Query engine None ### Please describe the bug 🐞 This one works: `table.updateProperties().set("meep", "woof").commit();`

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-01-31 Thread via GitHub
aokolnychyi commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1092249993 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-01-31 Thread via GitHub
aokolnychyi commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1092250698 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-01-31 Thread via GitHub
aokolnychyi commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1092254793 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

[GitHub] [iceberg] ajantha-bhat opened a new pull request, #6712: [WIP] Nessie: Support ApiV2 for Nessie client

2023-01-31 Thread via GitHub
ajantha-bhat opened a new pull request, #6712: URL: https://github.com/apache/iceberg/pull/6712 - The default is still kept as v1 as the v2 API cannot list implicit namespaces and need to provide a tool to convert implicit namespaces to explicit namespaces for an existing table if we make v

[GitHub] [iceberg] ajantha-bhat commented on a diff in pull request #6712: [WIP] Nessie: Support ApiV2 for Nessie client

2023-01-31 Thread via GitHub
ajantha-bhat commented on code in PR #6712: URL: https://github.com/apache/iceberg/pull/6712#discussion_r1092270108 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieUtil.java: ## @@ -37,6 +37,8 @@ public final class NessieUtil { public static final String NESSIE_CONFIG

[GitHub] [iceberg] jedrek-VL opened a new issue, #6713: PyIceberg fails when querying REST catalog

2023-01-31 Thread via GitHub
jedrek-VL opened a new issue, #6713: URL: https://github.com/apache/iceberg/issues/6713 ### Apache Iceberg version 1.1.0 (latest release) ### Query engine Other ### Please describe the bug 🐞 I start the spark/iceberg docker containers (as explained [here](h

[GitHub] [iceberg] nastra commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-01-31 Thread via GitHub
nastra commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1092274484 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-01-31 Thread via GitHub
aokolnychyi commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1092288701 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-01-31 Thread via GitHub
aokolnychyi commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1092289003 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-01-31 Thread via GitHub
aokolnychyi commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1092291477 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

[GitHub] [iceberg] yyanyy commented on a diff in pull request #6638: Spark: REPLACE BRANCH SQL implementation

2023-01-31 Thread via GitHub
yyanyy commented on code in PR #6638: URL: https://github.com/apache/iceberg/pull/6638#discussion_r1092293687 ## spark/v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateOrReplaceBranchExec.scala: ## @@ -0,0 +1,82 @@ +/* + * Licensed to the

[GitHub] [iceberg] yyanyy commented on a diff in pull request #6638: Spark: REPLACE BRANCH SQL implementation

2023-01-31 Thread via GitHub
yyanyy commented on code in PR #6638: URL: https://github.com/apache/iceberg/pull/6638#discussion_r1092294333 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestReplaceBranch.java: ## @@ -0,0 +1,222 @@ +/* + * Licensed to the Apache Software Fou

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-01-31 Thread via GitHub
aokolnychyi commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1092289003 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

[GitHub] [iceberg] stevenzwu commented on issue #6711: REST-Catalog: `Table.newTransaction()....commit()` does nothing

2023-01-31 Thread via GitHub
stevenzwu commented on issue #6711: URL: https://github.com/apache/iceberg/issues/6711#issuecomment-1410828722 @snazy I thought you need to call `.commitTransaction()` in the end -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [iceberg] nastra commented on issue #6711: REST-Catalog: `Table.newTransaction()....commit()` does nothing

2023-01-31 Thread via GitHub
nastra commented on issue #6711: URL: https://github.com/apache/iceberg/issues/6711#issuecomment-1410842372 Indeed this is missing the `commitTransaction() call. The below test passes for me: ``` @Test public void testUpdatePropertiesViaTransaction() { C catalog = catalog()

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-01-31 Thread via GitHub
aokolnychyi commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1092313811 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-01-31 Thread via GitHub
aokolnychyi commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1092314449 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

[GitHub] [iceberg] snazy commented on issue #6711: REST-Catalog: `Table.newTransaction()....commit()` does nothing

2023-01-31 Thread via GitHub
snazy commented on issue #6711: URL: https://github.com/apache/iceberg/issues/6711#issuecomment-1410860361 Oh, okay - NVM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [iceberg] snazy closed issue #6711: REST-Catalog: `Table.newTransaction()....commit()` does nothing

2023-01-31 Thread via GitHub
snazy closed issue #6711: REST-Catalog: `Table.newTransaction()commit()` does nothing URL: https://github.com/apache/iceberg/issues/6711 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6700: Snapshot ref type public

2023-01-31 Thread via GitHub
aokolnychyi commented on code in PR #6700: URL: https://github.com/apache/iceberg/pull/6700#discussion_r1092317116 ## api/src/main/java/org/apache/iceberg/SnapshotRefType.java: ## @@ -21,7 +21,7 @@ import java.util.Locale; import org.apache.iceberg.relocated.com.google.common.

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-01-31 Thread via GitHub
aokolnychyi commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1092320846 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

[GitHub] [iceberg] aokolnychyi commented on pull request #6695: Spark-3.3: Handle no-op for rewrite manifests procedure/action

2023-01-31 Thread via GitHub
aokolnychyi commented on PR #6695: URL: https://github.com/apache/iceberg/pull/6695#issuecomment-1410869692 I should be able to take a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6695: Spark-3.3: Handle no-op for rewrite manifests procedure/action

2023-01-31 Thread via GitHub
aokolnychyi commented on code in PR #6695: URL: https://github.com/apache/iceberg/pull/6695#discussion_r1092327457 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -172,6 +172,10 @@ private RewriteManifests.Result doExecu

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6695: Spark-3.3: Handle no-op for rewrite manifests procedure/action

2023-01-31 Thread via GitHub
aokolnychyi commented on code in PR #6695: URL: https://github.com/apache/iceberg/pull/6695#discussion_r1092327895 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -172,6 +172,10 @@ private RewriteManifests.Result doExecu

[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #6695: Spark-3.3: Handle no-op for rewrite manifests procedure/action

2023-01-31 Thread via GitHub
aokolnychyi commented on code in PR #6695: URL: https://github.com/apache/iceberg/pull/6695#discussion_r1092328163 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -172,6 +172,10 @@ private RewriteManifests.Result doExecu

[GitHub] [iceberg] aokolnychyi commented on pull request #6695: Spark-3.3: Handle no-op for rewrite manifests procedure/action

2023-01-31 Thread via GitHub
aokolnychyi commented on PR #6695: URL: https://github.com/apache/iceberg/pull/6695#issuecomment-1410876532 The change makes sense but we must be able to rewrite large manifests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-01-31 Thread via GitHub
danielcweeks commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1092334697 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-01-31 Thread via GitHub
danielcweeks commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1092336781 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-01-31 Thread via GitHub
danielcweeks commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1092338783 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-01-31 Thread via GitHub
danielcweeks commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1092339544 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[GitHub] [iceberg] dimas-b commented on a diff in pull request #6712: [WIP] Nessie: Support ApiV2 for Nessie client

2023-01-31 Thread via GitHub
dimas-b commented on code in PR #6712: URL: https://github.com/apache/iceberg/pull/6712#discussion_r1092348487 ## nessie/src/test/java/org/apache/iceberg/nessie/BaseTestIceberg.java: ## @@ -144,6 +144,9 @@ NessieCatalog initCatalog(String ref, String hash) { .put(Ca

[GitHub] [iceberg] dimas-b commented on a diff in pull request #6712: [WIP] Nessie: Support ApiV2 for Nessie client

2023-01-31 Thread via GitHub
dimas-b commented on code in PR #6712: URL: https://github.com/apache/iceberg/pull/6712#discussion_r1092348487 ## nessie/src/test/java/org/apache/iceberg/nessie/BaseTestIceberg.java: ## @@ -144,6 +144,9 @@ NessieCatalog initCatalog(String ref, String hash) { .put(Ca

[GitHub] [iceberg] dimas-b commented on a diff in pull request #6712: [WIP] Nessie: Support ApiV2 for Nessie client

2023-01-31 Thread via GitHub
dimas-b commented on code in PR #6712: URL: https://github.com/apache/iceberg/pull/6712#discussion_r1092350427 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNamespace.java: ## @@ -63,14 +65,56 @@ public void testListNamespaces() { tables = catalog.listTables(null);

[GitHub] [iceberg] rdblue commented on a diff in pull request #6598: Core: View representation core implementation

2023-01-31 Thread via GitHub
rdblue commented on code in PR #6598: URL: https://github.com/apache/iceberg/pull/6598#discussion_r1092354135 ## api/src/main/java/org/apache/iceberg/view/ViewRepresentation.java: ## @@ -18,21 +18,16 @@ */ package org.apache.iceberg.view; -import java.util.Locale; +import o

[GitHub] [iceberg] rdblue commented on a diff in pull request #6090: Core: Handle statistics file clean up from expireSnapshots

2023-01-31 Thread via GitHub
rdblue commented on code in PR #6090: URL: https://github.com/apache/iceberg/pull/6090#discussion_r1092361210 ## core/src/test/java/org/apache/iceberg/TestRemoveSnapshots.java: ## @@ -1234,6 +1243,95 @@ public void testMultipleRefsAndCleanExpiredFilesFailsForIncrementalCleanup(

[GitHub] [iceberg] rdblue commented on a diff in pull request #6090: Core: Handle statistics file clean up from expireSnapshots

2023-01-31 Thread via GitHub
rdblue commented on code in PR #6090: URL: https://github.com/apache/iceberg/pull/6090#discussion_r1092363213 ## core/src/test/java/org/apache/iceberg/TestRemoveSnapshots.java: ## @@ -1515,4 +1613,51 @@ private RemoveSnapshots removeSnapshots(Table table) { RemoveSnapshots

[GitHub] [iceberg] rdblue commented on a diff in pull request #6090: Core: Handle statistics file clean up from expireSnapshots

2023-01-31 Thread via GitHub
rdblue commented on code in PR #6090: URL: https://github.com/apache/iceberg/pull/6090#discussion_r1092364464 ## core/src/test/java/org/apache/iceberg/TestRemoveSnapshots.java: ## @@ -1234,6 +1243,95 @@ public void testMultipleRefsAndCleanExpiredFilesFailsForIncrementalCleanup(

[GitHub] [iceberg] rdblue commented on a diff in pull request #6090: Core: Handle statistics file clean up from expireSnapshots

2023-01-31 Thread via GitHub
rdblue commented on code in PR #6090: URL: https://github.com/apache/iceberg/pull/6090#discussion_r1092365363 ## core/src/test/java/org/apache/iceberg/TestRemoveSnapshots.java: ## @@ -1234,6 +1243,95 @@ public void testMultipleRefsAndCleanExpiredFilesFailsForIncrementalCleanup(

[GitHub] [iceberg] rdblue commented on a diff in pull request #6090: Core: Handle statistics file clean up from expireSnapshots

2023-01-31 Thread via GitHub
rdblue commented on code in PR #6090: URL: https://github.com/apache/iceberg/pull/6090#discussion_r1092367108 ## core/src/test/java/org/apache/iceberg/TestRemoveSnapshots.java: ## @@ -1234,6 +1243,95 @@ public void testMultipleRefsAndCleanExpiredFilesFailsForIncrementalCleanup(

[GitHub] [iceberg] rdblue commented on pull request #6090: Core: Handle statistics file clean up from expireSnapshots

2023-01-31 Thread via GitHub
rdblue commented on PR #6090: URL: https://github.com/apache/iceberg/pull/6090#issuecomment-1410923876 Thanks, @ajantha-bhat! I made some comments in tests to fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

[GitHub] [iceberg] rdblue commented on a diff in pull request #6651: Spark 3.3 write to branch snapshot

2023-01-31 Thread via GitHub
rdblue commented on code in PR #6651: URL: https://github.com/apache/iceberg/pull/6651#discussion_r1092375045 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java: ## @@ -247,9 +247,6 @@ public ScanBuilder newScanBuilder(CaseInsensitiveStringMap opt

[GitHub] [iceberg] rdblue commented on a diff in pull request #6582: Add a Spark procedure to collect NDV

2023-01-31 Thread via GitHub
rdblue commented on code in PR #6582: URL: https://github.com/apache/iceberg/pull/6582#discussion_r1092379542 ## core/src/main/java/org/apache/iceberg/puffin/StandardBlobTypes.java: ## @@ -26,4 +26,6 @@ private StandardBlobTypes() {} * href="https://datasketches.apache.org/"

[GitHub] [iceberg] aokolnychyi commented on issue #6679: Change Default Write Distribution Mode

2023-01-31 Thread via GitHub
aokolnychyi commented on issue #6679: URL: https://github.com/apache/iceberg/issues/6679#issuecomment-1410945521 I would be careful with `range` as it may cause performance regressions. Especially, for MERGE. The range distribution requires sampling that leads to double scanning and re-eval

[GitHub] [iceberg] rdblue commented on pull request #6432: Consider moving to ParallelIterable in Deletes::toPositionIndex

2023-01-31 Thread via GitHub
rdblue commented on PR #6432: URL: https://github.com/apache/iceberg/pull/6432#issuecomment-1410946484 The changes look good, but Spark OOMed. I kicked off the tests to run again. This could be something to fix since the parallelism will require copying delete rows. -- This is an automat

[GitHub] [iceberg] aokolnychyi commented on pull request #6683: Doc: Add update and merge distribution mode table props

2023-01-31 Thread via GitHub
aokolnychyi commented on PR #6683: URL: https://github.com/apache/iceberg/pull/6683#issuecomment-1410949529 I think we need a section showing examples we have in `TestSparkDistributionAndOrderingUtil` once we agree on #6679. -- This is an automated message from the Apache Git Service. To

[GitHub] [iceberg] aokolnychyi commented on pull request #6683: Doc: Add update and merge distribution mode table props

2023-01-31 Thread via GitHub
aokolnychyi commented on PR #6683: URL: https://github.com/apache/iceberg/pull/6683#issuecomment-1410951836 I am going to merge this one for now. We can add a section about tradeoffs later. -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [iceberg] aokolnychyi merged pull request #6683: Doc: Add update and merge distribution mode table props

2023-01-31 Thread via GitHub
aokolnychyi merged PR #6683: URL: https://github.com/apache/iceberg/pull/6683 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

[GitHub] [iceberg] aokolnychyi commented on pull request #6683: Doc: Add update and merge distribution mode table props

2023-01-31 Thread via GitHub
aokolnychyi commented on PR #6683: URL: https://github.com/apache/iceberg/pull/6683#issuecomment-1410952802 Thanks, @singhpk234! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

[GitHub] [iceberg] rdblue commented on a diff in pull request #6673: Python: Optimize PyArrow reads

2023-01-31 Thread via GitHub
rdblue commented on code in PR #6673: URL: https://github.com/apache/iceberg/pull/6673#discussion_r1092389857 ## python/pyiceberg/expressions/visitors.py: ## @@ -940,7 +963,9 @@ def visit_or( raise ValueError(f"Not allowed: {left_result} || {right_result}") -def exp

[GitHub] [iceberg] rdblue merged pull request #6673: Python: Optimize PyArrow reads

2023-01-31 Thread via GitHub
rdblue merged PR #6673: URL: https://github.com/apache/iceberg/pull/6673 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[GitHub] [iceberg] rdblue commented on pull request #6673: Python: Optimize PyArrow reads

2023-01-31 Thread via GitHub
rdblue commented on PR #6673: URL: https://github.com/apache/iceberg/pull/6673#issuecomment-1410953886 Thanks, @Fokko! Nice work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

[GitHub] [iceberg] rdblue commented on a diff in pull request #6707: Mark updated auth fields volatile for refresh

2023-01-31 Thread via GitHub
rdblue commented on code in PR #6707: URL: https://github.com/apache/iceberg/pull/6707#discussion_r1092391292 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java: ## @@ -354,10 +354,10 @@ public static class AuthSession { private static int tokenRefreshNumRet

[GitHub] [iceberg] rdblue merged pull request #6707: Mark updated auth fields volatile for refresh

2023-01-31 Thread via GitHub
rdblue merged PR #6707: URL: https://github.com/apache/iceberg/pull/6707 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[GitHub] [iceberg] rdblue commented on a diff in pull request #6701: Add missing `last-column-id` to spec

2023-01-31 Thread via GitHub
rdblue commented on code in PR #6701: URL: https://github.com/apache/iceberg/pull/6701#discussion_r1092392295 ## open-api/rest-catalog-open-api.yaml: ## @@ -1541,6 +1541,8 @@ components: properties: snapshot: $ref: '#/components/schemas/Sna

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-01-31 Thread via GitHub
danielcweeks commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1092392305 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-01-31 Thread via GitHub
danielcweeks commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1092394083 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[GitHub] [iceberg] rdblue commented on pull request #6701: Add missing `last-column-id` to spec

2023-01-31 Thread via GitHub
rdblue commented on PR #6701: URL: https://github.com/apache/iceberg/pull/6701#issuecomment-1410958514 The reason why this wasn't in the original spec is because we can technically update it by getting the highest assigned ID from the new schema. Because it didn't strictly need to be part o

[GitHub] [iceberg] rdblue commented on a diff in pull request #6700: Snapshot ref type public

2023-01-31 Thread via GitHub
rdblue commented on code in PR #6700: URL: https://github.com/apache/iceberg/pull/6700#discussion_r1092395055 ## api/src/main/java/org/apache/iceberg/SnapshotRefType.java: ## @@ -21,7 +21,7 @@ import java.util.Locale; import org.apache.iceberg.relocated.com.google.common.base.

[GitHub] [iceberg] aokolnychyi commented on pull request #5280: Docs: add missing table properties for update and merge write distrib…

2023-01-31 Thread via GitHub
aokolnychyi commented on PR #5280: URL: https://github.com/apache/iceberg/pull/5280#issuecomment-1410959675 Oops, I missed this PR and merged #6683 instead as that one popped up earlier in my feed. Sorry, @dramaticlly! I am going to close this one as that PR covers this. We will ne

[GitHub] [iceberg] aokolnychyi closed pull request #5280: Docs: add missing table properties for update and merge write distrib…

2023-01-31 Thread via GitHub
aokolnychyi closed pull request #5280: Docs: add missing table properties for update and merge write distrib… URL: https://github.com/apache/iceberg/pull/5280 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [iceberg] aokolnychyi commented on issue #6679: Change Default Write Distribution Mode

2023-01-31 Thread via GitHub
aokolnychyi commented on issue #6679: URL: https://github.com/apache/iceberg/issues/6679#issuecomment-1410960780 We have examples in `TestSparkDistributionAndOrderingUtil` that should become a section in the docs. -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [iceberg] rdblue commented on a diff in pull request #6698: Core, Hive: Support pluggable ClientPool

2023-01-31 Thread via GitHub
rdblue commented on code in PR #6698: URL: https://github.com/apache/iceberg/pull/6698#discussion_r1092396812 ## core/src/main/java/org/apache/iceberg/CatalogProperties.java: ## @@ -119,6 +119,8 @@ private CatalogProperties() {} "client.pool.cache.eviction-interval-ms";

[GitHub] [iceberg] rdblue commented on a diff in pull request #6698: Core, Hive: Support pluggable ClientPool

2023-01-31 Thread via GitHub
rdblue commented on code in PR #6698: URL: https://github.com/apache/iceberg/pull/6698#discussion_r1092397423 ## core/src/main/java/org/apache/iceberg/CatalogUtil.java: ## @@ -439,4 +439,44 @@ public static MetricsReporter loadMetricsReporter(String impl) { return report

[GitHub] [iceberg] rdblue commented on pull request #6696: Build: Bump Arrow from 10.0.1 to 11.0.0

2023-01-31 Thread via GitHub
rdblue commented on PR #6696: URL: https://github.com/apache/iceberg/pull/6696#issuecomment-1410962799 @ajantha-bhat, can you please run benchmarks to ensure there are no regressions? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [iceberg] rdblue commented on a diff in pull request #6657: Python: Allow to pass in a string as filter

2023-01-31 Thread via GitHub
rdblue commented on code in PR #6657: URL: https://github.com/apache/iceberg/pull/6657#discussion_r1092399725 ## python/pyiceberg/expressions/parser.py: ## @@ -232,6 +233,9 @@ def handle_or(result: ParseResults) -> Or: ).set_name("expr") -def parse(expr: str) -> BooleanExpr

[GitHub] [iceberg] rdblue commented on a diff in pull request #6657: Python: Allow to pass in a string as filter

2023-01-31 Thread via GitHub
rdblue commented on code in PR #6657: URL: https://github.com/apache/iceberg/pull/6657#discussion_r1092400176 ## python/pyiceberg/table/__init__.py: ## @@ -183,14 +185,17 @@ class TableScan(Generic[S], ABC): def __init__( self, table: Table, -row_f

[GitHub] [iceberg] rdblue commented on a diff in pull request #6703: Python: Fix warnings from pytest

2023-01-31 Thread via GitHub
rdblue commented on code in PR #6703: URL: https://github.com/apache/iceberg/pull/6703#discussion_r1092402076 ## python/tests/test_transforms.py: ## @@ -422,6 +422,7 @@ def test_void_transform() -> None: class TestType(IcebergBaseModel): __root__: Transform[Any, Any] +

[GitHub] [iceberg] rdblue commented on a diff in pull request #6703: Python: Fix warnings from pytest

2023-01-31 Thread via GitHub
rdblue commented on code in PR #6703: URL: https://github.com/apache/iceberg/pull/6703#discussion_r1092402367 ## python/tests/catalog/test_glue.py: ## @@ -67,7 +67,18 @@ def get_random_databases(n: int) -> Set[str]: @pytest.fixture(name="_bucket_initialize") def fixture_s3_b

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-01-31 Thread via GitHub
danielcweeks commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1092411299 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[GitHub] [iceberg] aokolnychyi commented on pull request #6682: Bulk delete

2023-01-31 Thread via GitHub
aokolnychyi commented on PR #6682: URL: https://github.com/apache/iceberg/pull/6682#issuecomment-1410979830 I am getting to this today, hopefully. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [iceberg] deniskuzZ commented on pull request #6653: API: Fix Transform backward compatibility in PartitionSpec

2023-01-31 Thread via GitHub
deniskuzZ commented on PR #6653: URL: https://github.com/apache/iceberg/pull/6653#issuecomment-1411019041 @Fokko could you please take a look -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6682: Bulk delete

2023-01-31 Thread via GitHub
amogh-jahagirdar commented on code in PR #6682: URL: https://github.com/apache/iceberg/pull/6682#discussion_r1092454022 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/actions/BaseSparkAction.java: ## @@ -85,6 +88,7 @@ private static final Logger LOG = LoggerFacto

[GitHub] [iceberg] GabeChurch closed issue #6667: Spark Hive Iceberg Table Locks -- Settings Unclear in Docs + Overrides Not Working

2023-01-31 Thread via GitHub
GabeChurch closed issue #6667: Spark Hive Iceberg Table Locks -- Settings Unclear in Docs + Overrides Not Working URL: https://github.com/apache/iceberg/issues/6667 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [iceberg] GabeChurch commented on issue #6667: Spark Hive Iceberg Table Locks -- Settings Unclear in Docs + Overrides Not Working

2023-01-31 Thread via GitHub
GabeChurch commented on issue #6667: URL: https://github.com/apache/iceberg/issues/6667#issuecomment-1411097955 Closing as I found a workaround mentioned above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6582: Add a Spark procedure to collect NDV

2023-01-31 Thread via GitHub
RussellSpitzer commented on code in PR #6582: URL: https://github.com/apache/iceberg/pull/6582#discussion_r1092522395 ## core/src/main/java/org/apache/iceberg/puffin/StandardBlobTypes.java: ## @@ -26,4 +26,6 @@ private StandardBlobTypes() {} * href="https://datasketches.apac

[GitHub] [iceberg] github-actions[bot] commented on issue #5356: Actions:API Binary Compatibility Checks #29 BUILD FAILED

2023-01-31 Thread github-actions
github-actions[bot] commented on issue #5356: URL: https://github.com/apache/iceberg/issues/5356#issuecomment-1411251107 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

[GitHub] [iceberg] github-actions[bot] closed issue #5243: iceberg + hive4.0

2023-01-31 Thread github-actions
github-actions[bot] closed issue #5243: iceberg + hive4.0 URL: https://github.com/apache/iceberg/issues/5243 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

[GitHub] [iceberg] github-actions[bot] commented on issue #5243: iceberg + hive4.0

2023-01-31 Thread github-actions
github-actions[bot] commented on issue #5243: URL: https://github.com/apache/iceberg/issues/5243#issuecomment-1411251195 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-01-31 Thread via GitHub
amogh-jahagirdar commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1092606929 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -821,8 +778,9 @@ public static class Builder { private List sortOrders; private fi

[GitHub] [iceberg] Fokko commented on a diff in pull request #6703: Python: Fix warnings from pytest

2023-01-31 Thread via GitHub
Fokko commented on code in PR #6703: URL: https://github.com/apache/iceberg/pull/6703#discussion_r1092655258 ## python/tests/catalog/test_glue.py: ## @@ -67,7 +67,18 @@ def get_random_databases(n: int) -> Set[str]: @pytest.fixture(name="_bucket_initialize") def fixture_s3_bu

[GitHub] [iceberg] Fokko commented on a diff in pull request #6703: Python: Fix warnings from pytest

2023-01-31 Thread via GitHub
Fokko commented on code in PR #6703: URL: https://github.com/apache/iceberg/pull/6703#discussion_r1092655410 ## python/tests/test_transforms.py: ## @@ -422,6 +422,7 @@ def test_void_transform() -> None: class TestType(IcebergBaseModel): __root__: Transform[Any, Any] +

[GitHub] [iceberg] Fokko commented on pull request #6703: Python: Fix warnings from pytest

2023-01-31 Thread via GitHub
Fokko commented on PR #6703: URL: https://github.com/apache/iceberg/pull/6703#issuecomment-1411314813 This PR needs a bit of TLC; it looks like the adlfs test also throws some warnings that we need to fix. -- This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [iceberg] danielcweeks commented on a diff in pull request #6706: Refactor table metadata snapshot management

2023-01-31 Thread via GitHub
danielcweeks commented on code in PR #6706: URL: https://github.com/apache/iceberg/pull/6706#discussion_r1092655694 ## core/src/main/java/org/apache/iceberg/SnapshotOperations.java: ## @@ -0,0 +1,202 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[GitHub] [iceberg] stevenzwu opened a new issue, #6715: AWS: WebIdentityTokenFileCredentialsProvider httpclient issue with EKS service account

2023-01-31 Thread via GitHub
stevenzwu opened a new issue, #6715: URL: https://github.com/apache/iceberg/issues/6715 ### Feature Request / Improvement `WebIdentityTokenFileCredentialsProvider` from `DefaultCredentialsProvider`chain uses DefaultSdkHttpClientBuilder for STS client to retrieve IAM credentials

[GitHub] [iceberg] zhangjiuyang1993 commented on issue #6708: Quick start docker-compose demo doesn't work

2023-01-31 Thread via GitHub
zhangjiuyang1993 commented on issue #6708: URL: https://github.com/apache/iceberg/issues/6708#issuecomment-1411351099 It works. Thanks a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

[GitHub] [iceberg] ajantha-bhat commented on a diff in pull request #6712: [WIP] Nessie: Support ApiV2 for Nessie client

2023-01-31 Thread via GitHub
ajantha-bhat commented on code in PR #6712: URL: https://github.com/apache/iceberg/pull/6712#discussion_r1092693475 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNamespace.java: ## @@ -63,14 +65,56 @@ public void testListNamespaces() { tables = catalog.listTables(nu

[GitHub] [iceberg] ajantha-bhat commented on a diff in pull request #6712: [WIP] Nessie: Support ApiV2 for Nessie client

2023-01-31 Thread via GitHub
ajantha-bhat commented on code in PR #6712: URL: https://github.com/apache/iceberg/pull/6712#discussion_r1092693947 ## nessie/src/test/java/org/apache/iceberg/nessie/BaseTestIceberg.java: ## @@ -144,6 +144,9 @@ NessieCatalog initCatalog(String ref, String hash) { .p

  1   2   >