Re: [PR] API, Core: Add uuid() to View [iceberg]

2023-10-30 Thread via GitHub
nastra merged PR #8851: URL: https://github.com/apache/iceberg/pull/8851 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [I] Long overflow when Iceberg reading INT96 timestamp column from Spark parquet table [iceberg]

2023-10-30 Thread via GitHub
yabola commented on issue #8949: URL: https://github.com/apache/iceberg/issues/8949#issuecomment-1786543799 @manuzhang Hi, Please correct me if I understand wrong, your error case is vectorized reading, and timestamp96 is not using dictionary. But I have dealt with this case actually. Pleas

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-10-30 Thread via GitHub
ajantha-bhat commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1377100203 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -264,6 +251,146 @@ public void renameTable(TableIdentifier from, TableIdentifier o

Re: [PR] JDBC catalog fix namespaceExists check [iceberg]

2023-10-30 Thread via GitHub
dramaticlly commented on code in PR #8340: URL: https://github.com/apache/iceberg/pull/8340#discussion_r1377104943 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java: ## @@ -601,15 +601,81 @@ public void testDropNamespace() { public void testCreateNamespace()

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-10-30 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1377095566 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -400,8 +400,15 @@ public void replaceTableViaTransactionThatAlreadyExistsAsView() {

Re: [I] Long overflow when Iceberg reading INT96 timestamp column from Spark parquet table [iceberg]

2023-10-30 Thread via GitHub
manuzhang commented on issue #8949: URL: https://github.com/apache/iceberg/issues/8949#issuecomment-1786508385 As per [parquet format encoding](https://github.com/apache/parquet-format/blob/master/Encodings.md#dictionary-encoding-plain_dictionary--2-and-rle_dictionary--8), > If the dictio

Re: [PR] Spark: Avoid extra copies of manifests while optimizing V2 tables [iceberg]

2023-10-30 Thread via GitHub
aokolnychyi commented on code in PR #8928: URL: https://github.com/apache/iceberg/pull/8928#discussion_r1377066516 ## core/src/main/java/org/apache/iceberg/BaseRewriteManifests.java: ## @@ -82,10 +83,12 @@ public class BaseRewriteManifests extends SnapshotProducer this.man

Re: [PR] Spark: Avoid extra copies of manifests while optimizing V2 tables [iceberg]

2023-10-30 Thread via GitHub
aokolnychyi commented on code in PR #8928: URL: https://github.com/apache/iceberg/pull/8928#discussion_r1373976469 ## core/src/main/java/org/apache/iceberg/BaseRewriteManifests.java: ## @@ -160,11 +163,10 @@ public RewriteManifests addManifest(ManifestFile manifest) { }

Re: [PR] Spark 3.5: Remove AssertHelpers usage [iceberg]

2023-10-30 Thread via GitHub
coded9 commented on code in PR #8948: URL: https://github.com/apache/iceberg/pull/8948#discussion_r1377020868 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRequiredDistributionAndOrdering.java: ## @@ -186,21 +186,22 @@ public void testDisab

Re: [PR] Spark 3.5: Remove AssertHelpers usage [iceberg]

2023-10-30 Thread via GitHub
coded9 commented on code in PR #8948: URL: https://github.com/apache/iceberg/pull/8948#discussion_r1377025566 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestTagDDL.java: ## @@ -137,11 +136,11 @@ public void testCreateTagUseDefaultConfig() th

Re: [PR] Spark 3.5: Remove AssertHelpers usage [iceberg]

2023-10-30 Thread via GitHub
coded9 commented on code in PR #8948: URL: https://github.com/apache/iceberg/pull/8948#discussion_r1377025066 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java: ## @@ -1019,21 +1015,21 @@ public void testMergeWithMultipleUpdatesForT

Re: [PR] Spark 3.5: Remove AssertHelpers usage [iceberg]

2023-10-30 Thread via GitHub
coded9 commented on code in PR #8948: URL: https://github.com/apache/iceberg/pull/8948#discussion_r1377022140 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestUpdate.java: ## @@ -1383,36 +1380,35 @@ public void testUpdateWithInvalidAssignments

Re: [PR] Spark 3.5: Remove AssertHelpers usage [iceberg]

2023-10-30 Thread via GitHub
coded9 commented on code in PR #8948: URL: https://github.com/apache/iceberg/pull/8948#discussion_r1377020868 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRequiredDistributionAndOrdering.java: ## @@ -186,21 +186,22 @@ public void testDisab

Re: [PR] Test: Add a test utility method to programmatically create expected partition specs [iceberg]

2023-10-30 Thread via GitHub
jerqi commented on PR #8467: URL: https://github.com/apache/iceberg/pull/8467#issuecomment-1786382503 @RussellSpitzer Comments are addressed. Could you take another when you have time? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] feat: Implement load table api. [iceberg-rust]

2023-10-30 Thread via GitHub
liurenjie1024 opened a new pull request, #89: URL: https://github.com/apache/iceberg-rust/pull/89 This pr implements load table in rest catalog, also add accesses to table metadata. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [I] Discussion: Replace usage of `Builder` with `TypedBuilder` [iceberg-rust]

2023-10-30 Thread via GitHub
liurenjie1024 commented on issue #88: URL: https://github.com/apache/iceberg-rust/issues/88#issuecomment-1786333662 cc @Fokko @ZENOTME @JanKaul @Xuanwo -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Test: Add a test utility method to programmatically create expected partition specs [iceberg]

2023-10-30 Thread via GitHub
jerqi commented on code in PR #8467: URL: https://github.com/apache/iceberg/pull/8467#discussion_r1376967622 ## api/src/test/java/org/apache/iceberg/TestHelpers.java: ## @@ -667,4 +671,40 @@ public List splitOffsets() { return null; } } + + public static class Pa

Re: [I] Iceberg Materialized View Spec [iceberg]

2023-10-30 Thread via GitHub
wmoustafa commented on issue #6420: URL: https://github.com/apache/iceberg/issues/6420#issuecomment-1786292375 Could you clarify how it does not work with the catalog API? At the end of the day, the catalog API takes a table identifier and returns an object. That can be table object, virtua

Re: [PR] Spark: Use SerializableTableWithSize when optimizing metadata [iceberg]

2023-10-30 Thread via GitHub
aokolnychyi commented on code in PR #8957: URL: https://github.com/apache/iceberg/pull/8957#discussion_r1376946080 ## spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -222,7 +222,9 @@ private Dataset buildManifestEntryDF(Li

[PR] Spark: Use SerializableTableWithSize when optimizing metadata [iceberg]

2023-10-30 Thread via GitHub
aokolnychyi opened a new pull request, #8957: URL: https://github.com/apache/iceberg/pull/8957 This PR migrates our rewrite manifest actions to use `SerializableTableWithSize`, a version of `SerializableTable` adapted for Spark. In particular, it implements `KnownSizeEstimation` for perform

Re: [PR] Spark 3.4: Don't cache or reuse manifest entries while rewriting metadata by default [iceberg]

2023-10-30 Thread via GitHub
aokolnychyi merged PR #8954: URL: https://github.com/apache/iceberg/pull/8954 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Spark 3.4: Don't cache or reuse manifest entries while rewriting metadata by default [iceberg]

2023-10-30 Thread via GitHub
aokolnychyi commented on PR #8954: URL: https://github.com/apache/iceberg/pull/8954#issuecomment-1786255408 There was a sporadic failure in Spark 3.5 which is not affected by this change. I'll merge this PR as is. -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Spark 3.3: Don't cache or reuse manifest entries while rewriting metadata by default [iceberg]

2023-10-30 Thread via GitHub
aokolnychyi merged PR #8955: URL: https://github.com/apache/iceberg/pull/8955 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Spark 3.2: Don't cache or reuse manifest entries while rewriting metadata by default [iceberg]

2023-10-30 Thread via GitHub
aokolnychyi merged PR #8956: URL: https://github.com/apache/iceberg/pull/8956 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [I] operations fail after upgrading to spark 3.4 [iceberg]

2023-10-30 Thread via GitHub
aokolnychyi commented on issue #8904: URL: https://github.com/apache/iceberg/issues/8904#issuecomment-1786253960 I replied on the Spark issue. If my understanding is correct, we can switch it off. -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [I] Support UNSET of sortOrder from the SQL [iceberg]

2023-10-30 Thread via GitHub
puchengy commented on issue #5071: URL: https://github.com/apache/iceberg/issues/5071#issuecomment-1786239047 Can we reopen this given this feature is useful? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [I] Spark analyzer exception during `MERGE INTO`: `Max iterations reached for batch resolution` [iceberg]

2023-10-30 Thread via GitHub
github-actions[bot] commented on issue #7502: URL: https://github.com/apache/iceberg/issues/7502#issuecomment-1786237342 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Provide a protected TableMetadata() constructor to allow inheritance [iceberg]

2023-10-30 Thread via GitHub
github-actions[bot] commented on issue #7509: URL: https://github.com/apache/iceberg/issues/7509#issuecomment-1786237312 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Provide a protected TableMetadata.MetadataLogEntry(long, String) constructor for inheritance [iceberg]

2023-10-30 Thread via GitHub
github-actions[bot] commented on issue #7510: URL: https://github.com/apache/iceberg/issues/7510#issuecomment-1786237286 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Expose ManifestLists as a public class [iceberg]

2023-10-30 Thread via GitHub
github-actions[bot] commented on issue #7511: URL: https://github.com/apache/iceberg/issues/7511#issuecomment-1786237260 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Refactor AwsProperties into separated properties classes [iceberg]

2023-10-30 Thread via GitHub
github-actions[bot] commented on issue #7515: URL: https://github.com/apache/iceberg/issues/7515#issuecomment-1786237203 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] About ICEBERG's experience in using large data tables [iceberg]

2023-10-30 Thread via GitHub
github-actions[bot] commented on issue #7514: URL: https://github.com/apache/iceberg/issues/7514#issuecomment-1786237231 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Avoid loading all AWS client dependencies for AwsClientFactory [iceberg]

2023-10-30 Thread via GitHub
github-actions[bot] commented on issue #7516: URL: https://github.com/apache/iceberg/issues/7516#issuecomment-1786237165 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Spark: Avoid extra copies of manifests while optimizing V2 tables [iceberg]

2023-10-30 Thread via GitHub
flyrain commented on code in PR #8928: URL: https://github.com/apache/iceberg/pull/8928#discussion_r1376706853 ## spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -337,7 +337,7 @@ private void replaceManifests( addedM

[PR] fix: partition evaluator thread safety [iceberg-python]

2023-10-30 Thread via GitHub
skellys opened a new pull request, #115: URL: https://github.com/apache/iceberg-python/pull/115 I found a thread safety bug where some unwanted partitions would make it past the partition filter. I found and tested this fix on a simple Iceberg table with one schema spec, 5 manifest files (s

Re: [I] operations fail after upgrading to spark 3.4 [iceberg]

2023-10-30 Thread via GitHub
aokolnychyi commented on issue #8904: URL: https://github.com/apache/iceberg/issues/8904#issuecomment-1785773348 Let me catch up later today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Spark 3.5: Don't cache or reuse manifest entries while rewriting metadata by default [iceberg]

2023-10-30 Thread via GitHub
aokolnychyi commented on PR #8935: URL: https://github.com/apache/iceberg/pull/8935#issuecomment-1785758870 Thanks, @RussellSpitzer! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Spark 3.5: Don't cache or reuse manifest entries while rewriting metadata by default [iceberg]

2023-10-30 Thread via GitHub
aokolnychyi merged PR #8935: URL: https://github.com/apache/iceberg/pull/8935 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Spark 3.5: Don't cache or reuse manifest entries while rewriting metadata by default [iceberg]

2023-10-30 Thread via GitHub
aokolnychyi commented on code in PR #8935: URL: https://github.com/apache/iceberg/pull/8935#discussion_r1376602061 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -277,16 +275,9 @@ private List writeManifestsForPartitio

Re: [PR] Spark 3.5: Don't cache or reuse manifest entries while rewriting metadata by default [iceberg]

2023-10-30 Thread via GitHub
RussellSpitzer commented on code in PR #8935: URL: https://github.com/apache/iceberg/pull/8935#discussion_r1376537307 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -277,16 +275,9 @@ private List writeManifestsForParti

Re: [PR] Spark 3.5: Don't cache or reuse manifest entries while rewriting metadata by default [iceberg]

2023-10-30 Thread via GitHub
RussellSpitzer commented on code in PR #8935: URL: https://github.com/apache/iceberg/pull/8935#discussion_r1376534827 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -277,16 +275,9 @@ private List writeManifestsForParti

[I] Duplicate file name in Iceberg's metadata [iceberg]

2023-10-30 Thread via GitHub
github-raphael-douyere opened a new issue, #8953: URL: https://github.com/apache/iceberg/issues/8953 ### Apache Iceberg version 1.3.1 ### Query engine Spark ### Please describe the bug 🐞 While writing data to an Iceberg table using Spark Streaming 3.4.1 / Ic

Re: [PR] Open-API: Refactor TableRequirements [iceberg]

2023-10-30 Thread via GitHub
Fokko merged PR #7710: URL: https://github.com/apache/iceberg/pull/7710 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Open-API: Refactor TableRequirements [iceberg]

2023-10-30 Thread via GitHub
Fokko commented on PR #7710: URL: https://github.com/apache/iceberg/pull/7710#issuecomment-1785528935 Thanks for the review @rdblue and @nastra for the review. I'll might revisitor other parts as well 👍 -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [PR] Open-API: Refactor TableRequirements [iceberg]

2023-10-30 Thread via GitHub
Fokko commented on code in PR #7710: URL: https://github.com/apache/iceberg/pull/7710#discussion_r1376457242 ## open-api/rest-catalog-open-api.yaml: ## @@ -1824,58 +1824,146 @@ components: - $ref: '#/components/schemas/SetPropertiesUpdate' - $ref: '#/components

Re: [PR] Core: Use avro compression properties from table properties when writing manifests and manifest lists [iceberg]

2023-10-30 Thread via GitHub
wypoon commented on PR #6799: URL: https://github.com/apache/iceberg/pull/6799#issuecomment-1785524764 @nastra @aokolnychyi please let me know what you think. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] Docs: Fix some typos in docs [iceberg]

2023-10-30 Thread via GitHub
Fokko commented on PR #8892: URL: https://github.com/apache/iceberg/pull/8892#issuecomment-1785518691 Thanks @hussein-awala for taking the time to create this PR, and @jbonofre, @ajantha-bhat for the review 👍 -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Docs: Fix some typos in docs [iceberg]

2023-10-30 Thread via GitHub
Fokko merged PR #8892: URL: https://github.com/apache/iceberg/pull/8892 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [I] Partitions metadata table returns inconsistent results size between 1.3.1 and 1.4.1 [iceberg]

2023-10-30 Thread via GitHub
zhangbutao commented on issue #8951: URL: https://github.com/apache/iceberg/issues/8951#issuecomment-1785506472 > This is why I asked if you checked the "files" metadata table. If the Files metadata table has no tuples of {data: a} then there is no partition of data = a. After trunca

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-10-30 Thread via GitHub
ajantha-bhat commented on PR #8909: URL: https://github.com/apache/iceberg/pull/8909#issuecomment-1785481731 @nastra and @adutra: Can you please take another look? I have addressed the comments. -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [I] Spark write abort result in table miss metadata location file [iceberg]

2023-10-30 Thread via GitHub
dyno commented on issue #8927: URL: https://github.com/apache/iceberg/issues/8927#issuecomment-1785479202 thanks @ConeyLiu ! will try 1.3.1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Build: Bump moto from 4.2.6 to 4.2.7 [iceberg-python]

2023-10-30 Thread via GitHub
Fokko merged PR #110: URL: https://github.com/apache/iceberg-python/pull/110 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Fix typo in `_primitive_to_phyisical` [iceberg-python]

2023-10-30 Thread via GitHub
Fokko commented on issue #107: URL: https://github.com/apache/iceberg-python/issues/107#issuecomment-1785446711 Closing this, since the PR has been merged. Thanks @whisk for working on this -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [I] Fix typo in `_primitive_to_phyisical` [iceberg-python]

2023-10-30 Thread via GitHub
Fokko closed issue #107: Fix typo in `_primitive_to_phyisical` URL: https://github.com/apache/iceberg-python/issues/107 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]

2023-10-30 Thread via GitHub
nastra commented on code in PR #8919: URL: https://github.com/apache/iceberg/pull/8919#discussion_r1371606620 ## site/README.md: ## @@ -65,7 +65,7 @@ The non-versioned site pages are all the `/site/docs/.*md` files and the docs ar > This build process is currently missing olde

Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]

2023-10-30 Thread via GitHub
Fokko commented on code in PR #8919: URL: https://github.com/apache/iceberg/pull/8919#discussion_r1376385112 ## site/README.md: ## @@ -65,7 +65,7 @@ The non-versioned site pages are all the `/site/docs/.*md` files and the docs ar > This build process is currently missing older

Re: [PR] WIP: Shift site build to use monorepo and asf infra [iceberg]

2023-10-30 Thread via GitHub
Fokko commented on code in PR #8919: URL: https://github.com/apache/iceberg/pull/8919#discussion_r1376383741 ## site/README.md: ## @@ -125,17 +124,16 @@ mkdocs serve ## Release process Deploying a version of the docs is a two step process: - 1. ~~Cut a new release from the c

Re: [I] Partitions metadata table returns inconsistent results size between 1.3.1 and 1.4.1 [iceberg]

2023-10-30 Thread via GitHub
ajantha-bhat commented on issue #8951: URL: https://github.com/apache/iceberg/issues/8951#issuecomment-1785379174 You are right about behaviour change. I think because of new stats addition like delete stats, we don't produce the effective stats for merge-on-read now. You can also ch

Re: [I] Partitions metadata table returns inconsistent results size between 1.3.1 and 1.4.1 [iceberg]

2023-10-30 Thread via GitHub
RussellSpitzer commented on issue #8951: URL: https://github.com/apache/iceberg/issues/8951#issuecomment-1785346959 @zhangbutao Iceberg cannot have partitions without data files. Either there is a file which has a tuple for a partition or not. If there is no file entry then the partitio

Re: [PR] Test: Add a test utility method to programmatically create expected partition specs [iceberg]

2023-10-30 Thread via GitHub
RussellSpitzer commented on code in PR #8467: URL: https://github.com/apache/iceberg/pull/8467#discussion_r1376310696 ## api/src/test/java/org/apache/iceberg/TestHelpers.java: ## @@ -667,4 +671,40 @@ public List splitOffsets() { return null; } } + + public static

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-10-30 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1376297916 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieTableOperations.java: ## @@ -135,71 +135,26 @@ protected void doCommit(TableMetadata base, TableMetadata m

Re: [PR] API, Core: Add uuid() to View [iceberg]

2023-10-30 Thread via GitHub
nastra commented on code in PR #8851: URL: https://github.com/apache/iceberg/pull/8851#discussion_r1376276601 ## api/src/main/java/org/apache/iceberg/view/View.java: ## @@ -111,4 +112,13 @@ default ReplaceViewVersion replaceVersion() { default UpdateLocation updateLocation()

Re: [PR] API, Core: Add uuid() to View [iceberg]

2023-10-30 Thread via GitHub
nastra commented on code in PR #8851: URL: https://github.com/apache/iceberg/pull/8851#discussion_r1376275776 ## api/src/main/java/org/apache/iceberg/view/View.java: ## @@ -111,4 +112,13 @@ default ReplaceViewVersion replaceVersion() { default UpdateLocation updateLocation()

Re: [PR] API, Core: Add UUID API to Table [iceberg]

2023-10-30 Thread via GitHub
nastra merged PR #8800: URL: https://github.com/apache/iceberg/pull/8800 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [I] Does Spark 3.3 support Tagging in AWS? [iceberg]

2023-10-30 Thread via GitHub
DavidCGarcia89 closed issue #8952: Does Spark 3.3 support Tagging in AWS? URL: https://github.com/apache/iceberg/issues/8952 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Update CachingCatalog to use expireAfterWrite instead of expireAfterAccess [iceberg]

2023-10-30 Thread via GitHub
nastra closed pull request #8844: Update CachingCatalog to use expireAfterWrite instead of expireAfterAccess URL: https://github.com/apache/iceberg/pull/8844 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [I] Does Spark 3.3 support Tagging in AWS? [iceberg]

2023-10-30 Thread via GitHub
DavidCGarcia89 commented on issue #8952: URL: https://github.com/apache/iceberg/issues/8952#issuecomment-1785270140 @nastra Hello Nastra, I think that's the problem. It seems that AWS [only have compatibility with 1.0.0](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-fo

Re: [PR] Core: Add table session cache [iceberg]

2023-10-30 Thread via GitHub
nastra closed pull request #8920: Core: Add table session cache URL: https://github.com/apache/iceberg/pull/8920 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-10-30 Thread via GitHub
adutra commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1376237462 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieTableOperations.java: ## @@ -135,71 +135,26 @@ protected void doCommit(TableMetadata base, TableMetadata metadat

Re: [I] the snapshot file is lost when write iceberg using flink Failed to open input stream for file File does not exist [iceberg]

2023-10-30 Thread via GitHub
nastra closed issue #4550: the snapshot file is lost when write iceberg using flink Failed to open input stream for fileFile does not exist URL: https://github.com/apache/iceberg/issues/4550 -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [PR] Core: fix `NullPointerException` when calling `InMemoryLockManager#release` using Hadoop catalog [iceberg]

2023-10-30 Thread via GitHub
nastra merged PR #8494: URL: https://github.com/apache/iceberg/pull/8494 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Core: fix `NullPointerException` when calling `InMemoryLockManager#release` using Hadoop catalog [iceberg]

2023-10-30 Thread via GitHub
nastra commented on PR #8494: URL: https://github.com/apache/iceberg/pull/8494#issuecomment-1785209915 @wangtaohz sorry for the delay, I was waiting for input of others. Given that this has been open for quite a while, we can go ahead and merge this. -- This is an automated message from t

Re: [I] Does Spark 3.3 support Tagging in AWS? [iceberg]

2023-10-30 Thread via GitHub
nastra commented on issue #8952: URL: https://github.com/apache/iceberg/issues/8952#issuecomment-1785206734 @DavidCGarcia89 which Iceberg version are you using? This should work with Iceberg 1.2.0 -- This is an automated message from the Apache Git Service. To respond to the message, plea

[I] Does Spark 3.3 support Tagging in AWS? [iceberg]

2023-10-30 Thread via GitHub
DavidCGarcia89 opened a new issue, #8952: URL: https://github.com/apache/iceberg/issues/8952 ### Query engine Spark 3.3 AWS Glue Catalog Scala ### Question I've been working with iceberg table in the glue table for a while and so far so good. But now i'm try

Re: [PR] Flink v1.15 : Remove usage of AssertHelpers [iceberg]

2023-10-30 Thread via GitHub
nastra commented on code in PR #8945: URL: https://github.com/apache/iceberg/pull/8945#discussion_r1376219994 ## flink/v1.15/flink/src/test/java/org/apache/iceberg/flink/source/enumerator/TestContinuousSplitPlannerImpl.java: ## @@ -337,12 +337,10 @@ public void testIncrementalFr

Re: [PR] Flink v1.15 : Remove usage of AssertHelpers [iceberg]

2023-10-30 Thread via GitHub
nastra commented on code in PR #8945: URL: https://github.com/apache/iceberg/pull/8945#discussion_r1376218462 ## flink/v1.15/flink/src/test/java/org/apache/iceberg/flink/source/TestStreamScanSql.java: ## @@ -219,15 +219,13 @@ public void testConsumeFilesWithBranch() throws Excep

Re: [I] Unable to use GlueCatalog in flink environments without hadoop [iceberg]

2023-10-30 Thread via GitHub
mgmarino commented on issue #3044: URL: https://github.com/apache/iceberg/issues/3044#issuecomment-1785192369 @rajcoolguy I've updated my comment 👆 to include a link to a gist with the relevant code. HTH. -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [PR] Flink v1.15 : Remove usage of AssertHelpers [iceberg]

2023-10-30 Thread via GitHub
nastra commented on code in PR #8945: URL: https://github.com/apache/iceberg/pull/8945#discussion_r1376219315 ## flink/v1.15/flink/src/test/java/org/apache/iceberg/flink/source/TestStreamingMonitorFunction.java: ## @@ -254,30 +254,24 @@ public void testInvalidMaxPlanningSnapshot

Re: [PR] Flink v1.15 : Remove usage of AssertHelpers [iceberg]

2023-10-30 Thread via GitHub
nastra commented on code in PR #8945: URL: https://github.com/apache/iceberg/pull/8945#discussion_r1376216574 ## flink/v1.15/flink/src/test/java/org/apache/iceberg/flink/source/TestFlinkSourceConfig.java: ## @@ -31,14 +31,12 @@ public class TestFlinkSourceConfig extends TestFli

Re: [I] Does spark-iceberg support create or drop tag/branch? [iceberg]

2023-10-30 Thread via GitHub
DavidCGarcia89 commented on issue #6902: URL: https://github.com/apache/iceberg/issues/6902#issuecomment-1785188729 I have also the same problem in AWS. I'm trying to execute a job with Scala trying to create a TAG but get this error: ![image](https://github.com/apache/iceberg/assets

Re: [PR] Flink v1.15 : Remove usage of AssertHelpers [iceberg]

2023-10-30 Thread via GitHub
nastra commented on code in PR #8945: URL: https://github.com/apache/iceberg/pull/8945#discussion_r1376216030 ## flink/v1.15/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSinkV2Base.java: ## @@ -278,20 +278,18 @@ protected void testUpsertOnIdKey(String branch

Re: [PR] Flink v1.15 : Remove usage of AssertHelpers [iceberg]

2023-10-30 Thread via GitHub
nastra commented on code in PR #8945: URL: https://github.com/apache/iceberg/pull/8945#discussion_r1376215669 ## flink/v1.15/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSinkV2Base.java: ## @@ -231,20 +231,20 @@ protected void testChangeLogOnIdKey(String bra

Re: [PR] Spark 3.5: Remove AssertHelpers usage [iceberg]

2023-10-30 Thread via GitHub
nastra commented on code in PR #8948: URL: https://github.com/apache/iceberg/pull/8948#discussion_r1376210094 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java: ## @@ -1019,21 +1015,21 @@ public void testMergeWithMultipleUpdatesForT

Re: [PR] Spark 3.5: Remove AssertHelpers usage [iceberg]

2023-10-30 Thread via GitHub
nastra commented on code in PR #8948: URL: https://github.com/apache/iceberg/pull/8948#discussion_r1376207854 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java: ## @@ -865,23 +864,23 @@ public void testMergeWithMultipleUpdatesForTar

Re: [PR] Spark 3.5: Remove AssertHelpers usage [iceberg]

2023-10-30 Thread via GitHub
nastra commented on code in PR #8948: URL: https://github.com/apache/iceberg/pull/8948#discussion_r1376208503 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java: ## @@ -906,24 +905,23 @@ public void testMergeWithMultipleUpdatesForTar

Re: [PR] Spark 3.5: Remove AssertHelpers usage [iceberg]

2023-10-30 Thread via GitHub
nastra commented on code in PR #8948: URL: https://github.com/apache/iceberg/pull/8948#discussion_r1376205939 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestUpdate.java: ## @@ -1383,36 +1380,35 @@ public void testUpdateWithInvalidAssignments

Re: [PR] Spark 3.5: Remove AssertHelpers usage [iceberg]

2023-10-30 Thread via GitHub
nastra commented on code in PR #8948: URL: https://github.com/apache/iceberg/pull/8948#discussion_r1376203445 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestTagDDL.java: ## @@ -207,11 +200,11 @@ public void testReplaceTagFailsForBranch() thr

Re: [PR] Spark 3.5: Remove AssertHelpers usage [iceberg]

2023-10-30 Thread via GitHub
nastra commented on code in PR #8948: URL: https://github.com/apache/iceberg/pull/8948#discussion_r1376204137 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestTagDDL.java: ## @@ -338,11 +328,10 @@ public void testDropTagFailesForBranch() throw

Re: [PR] Spark 3.5: Remove AssertHelpers usage [iceberg]

2023-10-30 Thread via GitHub
nastra commented on code in PR #8948: URL: https://github.com/apache/iceberg/pull/8948#discussion_r1376202288 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestTagDDL.java: ## @@ -137,11 +136,11 @@ public void testCreateTagUseDefaultConfig() th

Re: [PR] Spark 3.5: Remove AssertHelpers usage [iceberg]

2023-10-30 Thread via GitHub
nastra commented on code in PR #8948: URL: https://github.com/apache/iceberg/pull/8948#discussion_r1376202828 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestTagDDL.java: ## @@ -151,17 +147,14 @@ public void testCreateTagUseDefaultConfig() th

Re: [PR] Spark 3.5: Remove AssertHelpers usage [iceberg]

2023-10-30 Thread via GitHub
nastra commented on code in PR #8948: URL: https://github.com/apache/iceberg/pull/8948#discussion_r1376201141 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRequiredDistributionAndOrdering.java: ## @@ -186,21 +186,22 @@ public void testDisab

Re: [PR] AWS: Remove AssertHelpers usage [iceberg]

2023-10-30 Thread via GitHub
nastra merged PR #8937: URL: https://github.com/apache/iceberg/pull/8937 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-10-30 Thread via GitHub
adutra commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1376192197 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieTableOperations.java: ## @@ -135,71 +135,26 @@ protected void doCommit(TableMetadata base, TableMetadata metadat

Re: [PR] Core: fix `NullPointerException` when calling `InMemoryLockManager#release` using Hadoop catalog [iceberg]

2023-10-30 Thread via GitHub
wangtaohz commented on PR #8494: URL: https://github.com/apache/iceberg/pull/8494#issuecomment-1785130961 @nastra Can this PR be merged? In fact, we have been using the [fixed code](https://github.com/NetEase/amoro/blob/master/core/src/main/java/com/netease/arctic/utils/IcebergInMemo

Re: [PR] Iceberg delete files are read multiple times during query processing causing delays [iceberg]

2023-10-30 Thread via GitHub
Fokko commented on PR #6527: URL: https://github.com/apache/iceberg/pull/6527#issuecomment-1785114341 I believe @bryanck also ran into this, he might be interested in reviewing this as well -- This is an automated message from the Apache Git Service. To respond to the message, please log

[PR] Iceberg delete files are read multiple times during query processing causing delays [iceberg]

2023-10-30 Thread via GitHub
deniskuzZ opened a new pull request, #6527: URL: https://github.com/apache/iceberg/pull/6527 ref https://issues.apache.org/jira/browse/HIVE-26714 Current optimization covers only positional deletes by creating a PositionDeleteIndex bitmap for every task in a combined TableScan, avoidi

Re: [PR] Iceberg delete files are read multiple times during query processing causing delays [iceberg]

2023-10-30 Thread via GitHub
deniskuzZ closed pull request #6527: Iceberg delete files are read multiple times during query processing causing delays URL: https://github.com/apache/iceberg/pull/6527 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-10-30 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1376116783 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieTableOperations.java: ## @@ -135,71 +135,26 @@ protected void doCommit(TableMetadata base, TableMetadata m

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-10-30 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1376100816 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieViewOperations.java: ## @@ -0,0 +1,153 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-10-30 Thread via GitHub
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1376101149 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieTableOperations.java: ## @@ -135,71 +135,26 @@ protected void doCommit(TableMetadata base, TableMetadata m

  1   2   >