Re: [PR] [WIP] Add `PartitionEvaluator` to allow filtering of files in a table scan (Issue #152) [iceberg-rust]

2024-03-11 Thread via GitHub
sdd commented on PR #241: URL: https://github.com/apache/iceberg-rust/pull/241#issuecomment-1987748406 Sounds good @liurenjie1024 - will do. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

[PR] Docs: Enhance Spark pages [iceberg]

2024-03-11 Thread via GitHub
manuzhang opened a new pull request, #9920: URL: https://github.com/apache/iceberg/pull/9920 1. Fix internal links 2. Move `Type Compatibility` section from spark-writes.md to spark-getting-started.md -- This is an automated message from the Apache Git Service. To respond to the messag

[PR] Docs: update document release steps [iceberg]

2024-03-11 Thread via GitHub
ajantha-bhat opened a new pull request, #9921: URL: https://github.com/apache/iceberg/pull/9921 Remove the references about archived `iceberg-docs` repo and point to README. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Docs: update document release steps [iceberg]

2024-03-11 Thread via GitHub
ajantha-bhat commented on code in PR #9921: URL: https://github.com/apache/iceberg/pull/9921#discussion_r1519242204 ## site/docs/how-to-release.md: ## @@ -311,74 +311,7 @@ Create a PR in the `iceberg` repo to make revapi run on the new release. For an ### Documentation Release

Re: [PR] Docs: update document release steps [iceberg]

2024-03-11 Thread via GitHub
ajantha-bhat commented on code in PR #9921: URL: https://github.com/apache/iceberg/pull/9921#discussion_r1519242204 ## site/docs/how-to-release.md: ## @@ -311,74 +311,7 @@ Create a PR in the `iceberg` repo to make revapi run on the new release. For an ### Documentation Release

[PR] Update ASF DOAP rdf file [iceberg]

2024-03-11 Thread via GitHub
ajantha-bhat opened a new pull request, #9922: URL: https://github.com/apache/iceberg/pull/9922 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Docs: Move Iceberg Spark type compatibility to getting-started page [iceberg]

2024-03-11 Thread via GitHub
manuzhang closed pull request #9918: Docs: Move Iceberg Spark type compatibility to getting-started page URL: https://github.com/apache/iceberg/pull/9918 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Docs: Move Iceberg Spark type compatibility to getting-started page [iceberg]

2024-03-11 Thread via GitHub
manuzhang commented on PR #9918: URL: https://github.com/apache/iceberg/pull/9918#issuecomment-1987775202 superseded by #9920 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] feat: Implement binding expression [iceberg-rust]

2024-03-11 Thread via GitHub
sdd commented on PR #231: URL: https://github.com/apache/iceberg-rust/pull/231#issuecomment-1987797610 > I took a look at python's code again and realised that we should not couple not rewrite with binding, since not rewrite is not as widely used as binding, and we still need a rewriter. cc

Re: [PR] chore: Enable projects. [iceberg-rust]

2024-03-11 Thread via GitHub
nastra merged PR #247: URL: https://github.com/apache/iceberg-rust/pull/247 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Docs: Enhance Flink pages [iceberg]

2024-03-11 Thread via GitHub
nastra commented on code in PR #9919: URL: https://github.com/apache/iceberg/pull/9919#discussion_r1519268222 ## docs/docs/flink-writes.md: ## @@ -59,20 +59,20 @@ Iceberg supports `UPSERT` based on the primary key when writing data into v2 tab 1. Enable the `UPSERT` mode as

Re: [PR] Docs: Enhance Spark pages [iceberg]

2024-03-11 Thread via GitHub
nastra commented on PR #9920: URL: https://github.com/apache/iceberg/pull/9920#issuecomment-1987806754 @manuzhang have you verified that all links works by locally running the site as described [here](https://github.com/apache/iceberg/blob/main/site/README.md)? -- This is an automated me

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-11 Thread via GitHub
nastra commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1519276012 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveOperationsBase.java: ## @@ -62,6 +81,57 @@ interface HiveOperationsBase { String table(); + String ca

Re: [PR] Build: Bump orc from 1.9.2 to 2.0.0 [iceberg]

2024-03-11 Thread via GitHub
nastra commented on PR #9913: URL: https://github.com/apache/iceberg/pull/9913#issuecomment-1987823537 > It looks ignoring major version update in dependabot is not working. @nastra can you help check dependabot logs? This was never configured to be ignored -- This is an automated

Re: [PR] [Bug Fix] Allow Partition data to be nullable in ManifestEntry [iceberg-python]

2024-03-11 Thread via GitHub
Fokko merged PR #509: URL: https://github.com/apache/iceberg-python/pull/509 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Allow fsspec up to 2025.1 [iceberg-python]

2024-03-11 Thread via GitHub
Fokko merged PR #510: URL: https://github.com/apache/iceberg-python/pull/510 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Site: Update release notes for 1.5.0 [iceberg]

2024-03-11 Thread via GitHub
nastra commented on code in PR #9835: URL: https://github.com/apache/iceberg/pull/9835#discussion_r1519286131 ## site/docs/releases.md: ## @@ -67,6 +66,74 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`: ``` +### 1.5.0 release + +Apache Iceb

Re: [PR] Site: Update release notes for 1.5.0 [iceberg]

2024-03-11 Thread via GitHub
nastra commented on code in PR #9835: URL: https://github.com/apache/iceberg/pull/9835#discussion_r1519289312 ## site/docs/releases.md: ## @@ -67,6 +66,74 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`: ``` +### 1.5.0 release + +Apache Iceb

Re: [PR] Site: Update release notes for 1.5.0 [iceberg]

2024-03-11 Thread via GitHub
nastra commented on code in PR #9835: URL: https://github.com/apache/iceberg/pull/9835#discussion_r1519291068 ## site/docs/releases.md: ## @@ -67,6 +66,74 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`: ``` +### 1.5.0 release + +Apache Iceb

Re: [I] Add `table_exists` method to the Catalog [iceberg-python]

2024-03-11 Thread via GitHub
Fokko commented on issue #507: URL: https://github.com/apache/iceberg-python/issues/507#issuecomment-1987850682 @jayceslesar I would keep it simple by just returning a bool. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Migrate Snapshot files in Core to JUnit5 [iceberg]

2024-03-11 Thread via GitHub
nastra commented on code in PR #9892: URL: https://github.com/apache/iceberg/pull/9892#discussion_r1519303478 ## core/src/test/java/org/apache/iceberg/TestSnapshotJson.java: ## @@ -149,29 +142,26 @@ public void testJsonConversionWithV1Manifests() { timestampMillis);

[I] Add metadata tables [iceberg-python]

2024-03-11 Thread via GitHub
Fokko opened a new issue, #511: URL: https://github.com/apache/iceberg-python/issues/511 ### Feature Request / Improvement In Iceberg Spark there are metadata tables that provide information around the table: https://iceberg.apache.org/docs/latest/spark-queries/ The most import

Re: [PR] Migrate Snapshot files in Core to JUnit5 [iceberg]

2024-03-11 Thread via GitHub
nastra commented on code in PR #9892: URL: https://github.com/apache/iceberg/pull/9892#discussion_r1519305652 ## core/src/test/java/org/apache/iceberg/TestSnapshotManager.java: ## @@ -700,81 +686,82 @@ public void testAttemptToRollbackToCurrentSnapshot() { table.manageSnaps

Re: [PR] Migrate Snapshot files in Core to JUnit5 [iceberg]

2024-03-11 Thread via GitHub
nastra commented on code in PR #9892: URL: https://github.com/apache/iceberg/pull/9892#discussion_r1519306478 ## core/src/test/java/org/apache/iceberg/TestSnapshotManager.java: ## @@ -700,81 +686,82 @@ public void testAttemptToRollbackToCurrentSnapshot() { table.manageSnaps

Re: [PR] Migrate Snapshot files in Core to JUnit5 [iceberg]

2024-03-11 Thread via GitHub
nastra commented on code in PR #9892: URL: https://github.com/apache/iceberg/pull/9892#discussion_r1519306801 ## core/src/test/java/org/apache/iceberg/TestSnapshotManager.java: ## @@ -700,81 +686,82 @@ public void testAttemptToRollbackToCurrentSnapshot() { table.manageSnaps

[I] Calling `rewrite_position_delete_files` fails on tables with more than 1k columns [iceberg]

2024-03-11 Thread via GitHub
bk-mz opened a new issue, #9923: URL: https://github.com/apache/iceberg/issues/9923 ### Apache Iceberg version 1.4.3 (latest release) ### Query engine Spark ### Please describe the bug 🐞 When calling maintenance procedure `rewrite_position_delete_files`:

Re: [I] Calling `rewrite_position_delete_files` fails on tables with more than 1k columns [iceberg]

2024-03-11 Thread via GitHub
bk-mz commented on issue #9923: URL: https://github.com/apache/iceberg/issues/9923#issuecomment-1987912598 Other links on similar: https://github.com/apache/iceberg/issues/6368 https://github.com/apache/iceberg/pull/6369 https://github.com/apache/iceberg/pull/9904/files -- Th

Re: [I] Delete/Update fails for tables with more than 1000 columns [iceberg]

2024-03-11 Thread via GitHub
ssandona commented on issue #6368: URL: https://github.com/apache/iceberg/issues/6368#issuecomment-1987923779 Here a quick code to reproduce the error: ``` from pyspark.sql.functions import col ICEBERG_DB_NAME="mydb" ICEBERG_TABLE_NAME_MOR="my_mor_table" # Define th

Re: [I] Calling `rewrite_position_delete_files` fails on tables with more than 1k columns [iceberg]

2024-03-11 Thread via GitHub
ssandona commented on issue #9923: URL: https://github.com/apache/iceberg/issues/9923#issuecomment-1987928914 Here a quick code to reproduce the error (pyspark): ``` from pyspark.sql.functions import col ICEBERG_DB_NAME="mydb" ICEBERG_TABLE_NAME_MOR="my_mor_table" #

Re: [PR] Docs: Enhance Flink pages [iceberg]

2024-03-11 Thread via GitHub
manuzhang commented on PR #9919: URL: https://github.com/apache/iceberg/pull/9919#issuecomment-1987972482 @nastra This is how the numbered list with code blocks look like without indentation https://github.com/apache/iceberg/assets/1191767/7193fd01-49e1-441e-a889-de932c48e820";>

Re: [PR] Site: Update release notes for 1.5.0 [iceberg]

2024-03-11 Thread via GitHub
Fokko commented on code in PR #9835: URL: https://github.com/apache/iceberg/pull/9835#discussion_r1519393811 ## site/docs/releases.md: ## @@ -67,6 +66,74 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`: ``` +### 1.5.0 release + +Apache Icebe

Re: [PR] Build: Bump orc from 1.9.2 to 2.0.0 [iceberg]

2024-03-11 Thread via GitHub
manuzhang commented on PR #9913: URL: https://github.com/apache/iceberg/pull/9913#issuecomment-1987974805 Sorry to mention my attempt to ignore major version here in https://github.com/apache/iceberg/pull/9806 -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] refactor: Make plan_files as asynchronous stream [iceberg-rust]

2024-03-11 Thread via GitHub
liurenjie1024 merged PR #243: URL: https://github.com/apache/iceberg-rust/pull/243 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Docs: Enhance Flink pages [iceberg]

2024-03-11 Thread via GitHub
nastra commented on PR #9919: URL: https://github.com/apache/iceberg/pull/9919#issuecomment-1988022320 @manuzhang is it then correctly intended with your changes? Can you post a screenshot please? -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] Site: Update release notes for 1.5.0 [iceberg]

2024-03-11 Thread via GitHub
ajantha-bhat commented on code in PR #9835: URL: https://github.com/apache/iceberg/pull/9835#discussion_r1519454165 ## site/docs/releases.md: ## @@ -67,6 +66,74 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`: ``` +### 1.5.0 release + +Apach

Re: [PR] Data, Spark, Flink: Migrate TestAppenderFactory and subclasses to JUnit5 [iceberg]

2024-03-11 Thread via GitHub
nastra commented on code in PR #9862: URL: https://github.com/apache/iceberg/pull/9862#discussion_r1519457567 ## data/src/test/java/org/apache/iceberg/io/TestTaskEqualityDeltaWriter.java: ## @@ -290,38 +294,38 @@ public void testUpsertData() throws IOException { // Commit

Re: [I] Implement `TableMetadataBuilder`. [iceberg-rust]

2024-03-11 Thread via GitHub
liurenjie1024 commented on issue #250: URL: https://github.com/apache/iceberg-rust/issues/250#issuecomment-1988042951 See the discussion in https://github.com/apache/iceberg-rust/issues/250 -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] Site: Update release notes for 1.5.0 [iceberg]

2024-03-11 Thread via GitHub
nastra commented on code in PR #9835: URL: https://github.com/apache/iceberg/pull/9835#discussion_r1519462341 ## site/docs/releases.md: ## @@ -67,6 +66,74 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`: ``` +### 1.5.0 release + +Apache Iceb

Re: [PR] Sql catalog [iceberg-rust]

2024-03-11 Thread via GitHub
liurenjie1024 commented on code in PR #229: URL: https://github.com/apache/iceberg-rust/pull/229#discussion_r1513742759 ## crates/catalog/sql/src/lib.rs: ## @@ -0,0 +1,24 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements

Re: [I] Implement Sql Catalog. [iceberg-rust]

2024-03-11 Thread via GitHub
liurenjie1024 commented on issue #248: URL: https://github.com/apache/iceberg-rust/issues/248#issuecomment-1988047733 Blocked by #250 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Implement Glue Catalog [iceberg-rust]

2024-03-11 Thread via GitHub
liurenjie1024 commented on issue #249: URL: https://github.com/apache/iceberg-rust/issues/249#issuecomment-1988048213 Blocked by #250 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Add hive metastore catalog support [iceberg-rust]

2024-03-11 Thread via GitHub
liurenjie1024 commented on issue #113: URL: https://github.com/apache/iceberg-rust/issues/113#issuecomment-1988048778 The left parts are blocked by #250 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Sql catalog [iceberg-rust]

2024-03-11 Thread via GitHub
Xuanwo commented on code in PR #229: URL: https://github.com/apache/iceberg-rust/pull/229#discussion_r1519484516 ## crates/catalog/sql/Cargo.toml: ## @@ -0,0 +1,52 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See th

Re: [PR] Sql catalog [iceberg-rust]

2024-03-11 Thread via GitHub
Xuanwo commented on code in PR #229: URL: https://github.com/apache/iceberg-rust/pull/229#discussion_r1519485352 ## crates/catalog/sql/src/catalog.rs: ## @@ -0,0 +1,397 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] Sql catalog [iceberg-rust]

2024-03-11 Thread via GitHub
Xuanwo commented on code in PR #229: URL: https://github.com/apache/iceberg-rust/pull/229#discussion_r1519485352 ## crates/catalog/sql/src/catalog.rs: ## @@ -0,0 +1,397 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] Docs: Enhance Flink pages [iceberg]

2024-03-11 Thread via GitHub
manuzhang commented on PR #9919: URL: https://github.com/apache/iceberg/pull/9919#issuecomment-1988090799 Here it is. https://github.com/apache/iceberg/assets/1191767/974005ab-aba9-4426-9404-f98bf81fa458";> -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Docs: Enhance Flink pages [iceberg]

2024-03-11 Thread via GitHub
nastra merged PR #9919: URL: https://github.com/apache/iceberg/pull/9919 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Hive: Common namespace validation for HiveCatalog [iceberg]

2024-03-11 Thread via GitHub
nastra commented on code in PR #9924: URL: https://github.com/apache/iceberg/pull/9924#discussion_r1519506172 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -554,10 +556,6 @@ private Map convertToMetadata(Database database) { } Database

Re: [PR] Site: Update release notes for 1.5.0 [iceberg]

2024-03-11 Thread via GitHub
ajantha-bhat commented on code in PR #9835: URL: https://github.com/apache/iceberg/pull/9835#discussion_r1519520018 ## site/docs/releases.md: ## @@ -67,6 +66,74 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`: ``` +### 1.5.0 release + +Apach

Re: [PR] Build: Bump com.fasterxml.jackson.dataformat:jackson-dataformat-xml from 2.16.1 to 2.16.2 [iceberg]

2024-03-11 Thread via GitHub
dependabot[bot] commented on PR #9914: URL: https://github.com/apache/iceberg/pull/9914#issuecomment-1988154763 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let

Re: [PR] Build: Bump com.fasterxml.jackson.dataformat:jackson-dataformat-xml from 2.16.1 to 2.16.2 [iceberg]

2024-03-11 Thread via GitHub
nastra closed pull request #9914: Build: Bump com.fasterxml.jackson.dataformat:jackson-dataformat-xml from 2.16.1 to 2.16.2 URL: https://github.com/apache/iceberg/pull/9914 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Build: Bump com.fasterxml.jackson.core:jackson-annotations from 2.16.0 to 2.16.2 [iceberg]

2024-03-11 Thread via GitHub
nastra closed pull request #9915: Build: Bump com.fasterxml.jackson.core:jackson-annotations from 2.16.0 to 2.16.2 URL: https://github.com/apache/iceberg/pull/9915 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] Build: Bump com.fasterxml.jackson.core:jackson-annotations from 2.16.0 to 2.16.2 [iceberg]

2024-03-11 Thread via GitHub
dependabot[bot] commented on PR #9915: URL: https://github.com/apache/iceberg/pull/9915#issuecomment-1988160165 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let

Re: [PR] Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.34.0 [iceberg]

2024-03-11 Thread via GitHub
nastra closed pull request #9916: Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.34.0 URL: https://github.com/apache/iceberg/pull/9916 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.34.0 [iceberg]

2024-03-11 Thread via GitHub
dependabot[bot] commented on PR #9916: URL: https://github.com/apache/iceberg/pull/9916#issuecomment-1988160444 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let

Re: [PR] Build: Bump org.roaringbitmap:RoaringBitmap from 1.0.1 to 1.0.5 [iceberg]

2024-03-11 Thread via GitHub
nastra merged PR #9911: URL: https://github.com/apache/iceberg/pull/9911 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Site: Update release notes for 1.5.0 [iceberg]

2024-03-11 Thread via GitHub
Fokko merged PR #9835: URL: https://github.com/apache/iceberg/pull/9835 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Hive: Common namespace validation for HiveCatalog [iceberg]

2024-03-11 Thread via GitHub
nk1506 commented on code in PR #9924: URL: https://github.com/apache/iceberg/pull/9924#discussion_r1519550142 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -554,10 +556,6 @@ private Map convertToMetadata(Database database) { } Database

Re: [PR] Hive: Common namespace validation for HiveCatalog [iceberg]

2024-03-11 Thread via GitHub
nk1506 commented on code in PR #9924: URL: https://github.com/apache/iceberg/pull/9924#discussion_r1519550142 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -554,10 +556,6 @@ private Map convertToMetadata(Database database) { } Database

Re: [I] Add hive metastore catalog support [iceberg-rust]

2024-03-11 Thread via GitHub
marvinlanhenke commented on issue #113: URL: https://github.com/apache/iceberg-rust/issues/113#issuecomment-1988201903 > The left parts are blocked by #250 thank you so much for setting up the projects and preparing release 0.3.0 - I like the structure. For this issue - I'll tr

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-11 Thread via GitHub
nk1506 commented on PR #9852: URL: https://github.com/apache/iceberg/pull/9852#issuecomment-1988285382 @nastra build is failing because of following reason: `Error: The log was not found. It may have been deleted based on retention settings.` Ref: https://github.com/apache/iceberg

Re: [PR] Migrate Snapshot files in Core to JUnit5 [iceberg]

2024-03-11 Thread via GitHub
tomtongue commented on PR #9892: URL: https://github.com/apache/iceberg/pull/9892#issuecomment-1988354068 Thanks for the review @nastra -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] Add `table_exists` method to Catalog [iceberg-python]

2024-03-11 Thread via GitHub
syun64 commented on code in PR #512: URL: https://github.com/apache/iceberg-python/pull/512#discussion_r1519661896 ## pyiceberg/catalog/__init__.py: ## @@ -646,6 +646,13 @@ def purge_table(self, identifier: Union[str, Identifier]) -> None: delete_files(io, prev_metadat

Re: [PR] Core: Support appending files with different specs [iceberg]

2024-03-11 Thread via GitHub
fqaiser94 commented on code in PR #9860: URL: https://github.com/apache/iceberg/pull/9860#discussion_r1519685957 ## core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java: ## @@ -878,7 +886,7 @@ public Object updateEvent() { @SuppressWarnings("checkstyle:Cycloma

Re: [PR] Core: Support appending files with different specs [iceberg]

2024-03-11 Thread via GitHub
fqaiser94 commented on code in PR #9860: URL: https://github.com/apache/iceberg/pull/9860#discussion_r1519687485 ## core/src/test/java/org/apache/iceberg/TestMergeAppend.java: ## @@ -92,6 +94,74 @@ public void testEmptyTableAppend() { statuses(Status.ADDED, Status.ADDED

Re: [PR] Core: Support appending files with different specs [iceberg]

2024-03-11 Thread via GitHub
fqaiser94 commented on code in PR #9860: URL: https://github.com/apache/iceberg/pull/9860#discussion_r1519703373 ## core/src/main/java/org/apache/iceberg/BaseReplacePartitions.java: ## @@ -87,31 +87,40 @@ public BaseReplacePartitions toBranch(String branch) { @Override p

Re: [PR] Build: Let revapi compare against 1.5.0 [iceberg]

2024-03-11 Thread via GitHub
ajantha-bhat closed pull request #9777: Build: Let revapi compare against 1.5.0 URL: https://github.com/apache/iceberg/pull/9777 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Migrate Snapshot files in Core to JUnit5 [iceberg]

2024-03-11 Thread via GitHub
nastra merged PR #9892: URL: https://github.com/apache/iceberg/pull/9892 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Data, Spark, Flink: Migrate TestAppenderFactory and subclasses to JUnit5 [iceberg]

2024-03-11 Thread via GitHub
nastra merged PR #9862: URL: https://github.com/apache/iceberg/pull/9862 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [I] Add metadata tables [iceberg-python]

2024-03-11 Thread via GitHub
Gowthami03B commented on issue #511: URL: https://github.com/apache/iceberg-python/issues/511#issuecomment-1988520270 @Fokko I can take a stab at this! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [I] Calling `rewrite_position_delete_files` rewrites into same amount of files [iceberg]

2024-03-11 Thread via GitHub
bk-mz commented on issue #9833: URL: https://github.com/apache/iceberg/issues/9833#issuecomment-1988535507 Just to give more overview, after something, iceberg has created a mirror of existing partitions in which it has put TONS of small files. Example: we have "phantom" partition `47

Re: [PR] Core: Support appending files with different specs [iceberg]

2024-03-11 Thread via GitHub
nastra commented on code in PR #9860: URL: https://github.com/apache/iceberg/pull/9860#discussion_r1519787373 ## core/src/test/java/org/apache/iceberg/TestMergeAppend.java: ## @@ -92,6 +96,92 @@ public void testEmptyTableAppend() { statuses(Status.ADDED, Status.ADDED));

Re: [I] Calling `rewrite_position_delete_files` rewrites into same amount of files [iceberg]

2024-03-11 Thread via GitHub
bk-mz commented on issue #9833: URL: https://github.com/apache/iceberg/issues/9833#issuecomment-1988536957 So the big question would what could break in iceberg so it would create those "phantom" and odd partitions? -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Core: Support appending files with different specs [iceberg]

2024-03-11 Thread via GitHub
nastra commented on code in PR #9860: URL: https://github.com/apache/iceberg/pull/9860#discussion_r1519787882 ## core/src/test/java/org/apache/iceberg/TestMergeAppend.java: ## @@ -92,6 +96,92 @@ public void testEmptyTableAppend() { statuses(Status.ADDED, Status.ADDED));

Re: [PR] Core: Support appending files with different specs [iceberg]

2024-03-11 Thread via GitHub
nastra commented on code in PR #9860: URL: https://github.com/apache/iceberg/pull/9860#discussion_r1519789244 ## core/src/test/java/org/apache/iceberg/TestOverwrite.java: ## @@ -271,6 +280,36 @@ public void testValidatedOverwriteWithAppendOutsideOfDelete() { "Should no

Re: [PR] Core: Support appending files with different specs [iceberg]

2024-03-11 Thread via GitHub
nastra commented on code in PR #9860: URL: https://github.com/apache/iceberg/pull/9860#discussion_r1519791560 ## core/src/test/java/org/apache/iceberg/TestOverwrite.java: ## @@ -293,23 +332,178 @@ public void testValidatedOverwriteWithAppendOutsideOfDeleteMetrics() { }

Re: [PR] Core: Support appending files with different specs [iceberg]

2024-03-11 Thread via GitHub
nastra commented on code in PR #9860: URL: https://github.com/apache/iceberg/pull/9860#discussion_r1519797012 ## core/src/test/java/org/apache/iceberg/TestOverwrite.java: ## @@ -293,23 +332,178 @@ public void testValidatedOverwriteWithAppendOutsideOfDeleteMetrics() { }

[PR] feat: implement OAuth for catalog rest client [iceberg-rust]

2024-03-11 Thread via GitHub
TennyZhuang opened a new pull request, #254: URL: https://github.com/apache/iceberg-rust/pull/254 Resolve #238 The docker image of tabular doesn't support credential settings, so it's impossible to write a test. I just test it manually using their hosted iceberg service. --

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-11 Thread via GitHub
nastra commented on PR #9852: URL: https://github.com/apache/iceberg/pull/9852#issuecomment-1988556545 > @nastra build is failing because of following reason: `Error: The log was not found. It may have been deleted based on retention settings.` > > Ref: https://github.com/apache/iceb

Re: [PR] refactor: Make plan_files as asynchronous stream [iceberg-rust]

2024-03-11 Thread via GitHub
viirya commented on PR #243: URL: https://github.com/apache/iceberg-rust/pull/243#issuecomment-1988595818 Thank you @liurenjie1024 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] Docs: Add Daft into Iceberg documentation [iceberg]

2024-03-11 Thread via GitHub
bitsondatadev commented on code in PR #9836: URL: https://github.com/apache/iceberg/pull/9836#discussion_r1519832573 ## docs/docs/daft.md: ## @@ -0,0 +1,146 @@ +--- +title: "Daft" +--- + + +# Daft Review Comment: Yeah, I used to have a "nightly" build in there, but we took i

Re: [PR] Infra: Add 1.5.0 to issue template [iceberg]

2024-03-11 Thread via GitHub
Fokko merged PR #9778: URL: https://github.com/apache/iceberg/pull/9778 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Update ASF DOAP rdf file for 1.5.0 [iceberg]

2024-03-11 Thread via GitHub
nastra merged PR #9922: URL: https://github.com/apache/iceberg/pull/9922 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Build: Let revapi compare against 1.5.0 [iceberg]

2024-03-11 Thread via GitHub
nastra merged PR #9777: URL: https://github.com/apache/iceberg/pull/9777 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [I] Read Parquet data file with projection [iceberg-rust]

2024-03-11 Thread via GitHub
viirya commented on issue #244: URL: https://github.com/apache/iceberg-rust/issues/244#issuecomment-1988619183 Thank you @liurenjie1024 . I have looked the doc @sdd mentioned that describes Iceberg column projection. Looks like the projection is specified by field id as @sdd said due

[PR] [WIP] Migrate Metadata files in Core to JUnit5 [iceberg]

2024-03-11 Thread via GitHub
tomtongue opened a new pull request, #9927: URL: https://github.com/apache/iceberg/pull/9927 Migrate the following "Read" in iceberg-core to JUnit 5 for https://github.com/apache/iceberg/issues/9085. ## Current Progress - [x] TableMetadataParserCodecTest - [x] TableMetadataParse

Re: [I] Support "prefix" prop in iceberg rest client [iceberg-rust]

2024-03-11 Thread via GitHub
TennyZhuang commented on issue #255: URL: https://github.com/apache/iceberg-rust/issues/255#issuecomment-1988647663 @liurenjie1024 Please assign the issue to me, thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [I] iceberg-core: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

2024-03-11 Thread via GitHub
tomtongue commented on issue #9085: URL: https://github.com/apache/iceberg/issues/9085#issuecomment-1988648239 Created a PR for the migration of the files related to "metadata". Here's the progress: - [x] TableMetadataParserCodecTest - [x] TableMetadataParserTest - [ ] TestMetad

[I] Add ability to pickle a `Table` [iceberg-python]

2024-03-11 Thread via GitHub
Fokko opened a new issue, #513: URL: https://github.com/apache/iceberg-python/issues/513 ### Feature Request / Improvement This allows distribution of the Table object within Ray. -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [I] iceberg-core: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

2024-03-11 Thread via GitHub
seyeh commented on issue #9085: URL: https://github.com/apache/iceberg/issues/9085#issuecomment-1988656881 @tomtongue After you make changes, do you run `./gradlew build`? If so, how long does this process take on your computer. Also since you've been working this ticket, do you want

Re: [I] Calling `rewrite_position_delete_files` rewrites into same amount of files [iceberg]

2024-03-11 Thread via GitHub
RussellSpitzer commented on issue #9833: URL: https://github.com/apache/iceberg/issues/9833#issuecomment-1988660869 The path's aren't as important as the metadata, do the files have incorrect partition tuples (the values actually used for filtering)? The position delete writer here is most

Re: [I] When deleting data from Iceberg tables in Spark, the current approach is to delete all data and then rewrite the new data, which is very wasteful in terms of storage space and computation. How

2024-03-11 Thread via GitHub
RussellSpitzer closed issue #9891: When deleting data from Iceberg tables in Spark, the current approach is to delete all data and then rewrite the new data, which is very wasteful in terms of storage space and computation. However, when using the Trino engine to delete data from Iceberg table

Re: [I] When deleting data from Iceberg tables in Spark, the current approach is to delete all data and then rewrite the new data, which is very wasteful in terms of storage space and computation. How

2024-03-11 Thread via GitHub
RussellSpitzer commented on issue #9891: URL: https://github.com/apache/iceberg/issues/9891#issuecomment-1988670380 Closed as this is already supported as mentioned. https://github.com/apache/iceberg/issues/9891#issuecomment-1984132293 -- This is an automated message from the Apach

Re: [I] Add ability to pickle a `Table` [iceberg-python]

2024-03-11 Thread via GitHub
amogh-jahagirdar commented on issue #513: URL: https://github.com/apache/iceberg-python/issues/513#issuecomment-1988733788 @Fokko if nobody is already looking at this, would love to try and take a stab at this? -- This is an automated message from the Apache Git Service. To respond to th

Re: [I] About /v1/config REST API endpoint [iceberg]

2024-03-11 Thread via GitHub
danielcweeks commented on issue #9880: URL: https://github.com/apache/iceberg/issues/9880#issuecomment-1988755504 > Thanks. But may be we should use a separate PATCH/PUT instead of sending in a GET request as we want to update the resource here. The `warehouse` config is really not so

Re: [PR] [WIP] Add `PartitionEvaluator` to allow filtering of files in a table scan (Issue #152) [iceberg-rust]

2024-03-11 Thread via GitHub
Xuanwo commented on code in PR #241: URL: https://github.com/apache/iceberg-rust/pull/241#discussion_r1519947613 ## crates/iceberg/src/scan.rs: ## @@ -52,6 +62,18 @@ impl<'a> TableScanBuilder<'a> { self } +/// Sets the scan as being case-insensitive +pub

Re: [I] iceberg-core: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

2024-03-11 Thread via GitHub
tomtongue commented on issue #9085: URL: https://github.com/apache/iceberg/issues/9085#issuecomment-1988786215 > After you make changes, do you run ./gradlew build? If so, how long does this process take on your computer. basically run `./gradlew test` for specific classes and package

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-11 Thread via GitHub
danielcweeks commented on PR #9884: URL: https://github.com/apache/iceberg/pull/9884#issuecomment-1988786681 > @danielcweeks And how can we tackle if an user wants to override an existing Iceberg `ResolverFileIO` ? Imagine if the user wants to change the authentication for `s3` or add `my-s

[I] How tracker authors of iceberg snapshots? [iceberg]

2024-03-11 Thread via GitHub
nikita-sheremet-flocktory opened a new issue, #9928: URL: https://github.com/apache/iceberg/issues/9928 ### Query engine trino ### Question Every change to iceberg table leads to new snapshot and metadata file. Is there a way to map user lauched a query with snapshot in

Re: [PR] Spark: Fail on recursive cycle in view [iceberg]

2024-03-11 Thread via GitHub
amogh-jahagirdar commented on code in PR #9834: URL: https://github.com/apache/iceberg/pull/9834#discussion_r151993 ## spark/v3.5/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckViews.scala: ## @@ -63,4 +72,29 @@ object CheckViews extends (Logical

  1   2   3   >