Re: [PR] feat: First version of rest catalog. [iceberg-rust]

2023-10-15 Thread via GitHub
liurenjie1024 commented on code in PR #78: URL: https://github.com/apache/iceberg-rust/pull/78#discussion_r1360184684 ## crates/iceberg/Cargo.toml: ## @@ -41,20 +41,24 @@ either = "1" futures = "0.3" itertools = "0.11" lazy_static = "1" +log = "^0.4" murmur3 = "0.5.2" once_

Re: [PR] Build: Bump com.fasterxml.jackson.core:jackson-annotations from 2.15.2 to 2.15.3 [iceberg]

2023-10-15 Thread via GitHub
nastra merged PR #8836: URL: https://github.com/apache/iceberg/pull/8836 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Build: Bump org.xerial:sqlite-jdbc from 3.43.0.0 to 3.43.2.0 [iceberg]

2023-10-15 Thread via GitHub
nastra commented on PR #8837: URL: https://github.com/apache/iceberg/pull/8837#issuecomment-1763815524 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] feat: First version of rest catalog. [iceberg-rust]

2023-10-15 Thread via GitHub
JanKaul commented on code in PR #78: URL: https://github.com/apache/iceberg-rust/pull/78#discussion_r1360172363 ## crates/iceberg/Cargo.toml: ## @@ -41,20 +41,24 @@ either = "1" futures = "0.3" itertools = "0.11" lazy_static = "1" +log = "^0.4" murmur3 = "0.5.2" once_cell =

Re: [I] [JdbcCatalog] Issue with Namespace Exists [iceberg]

2023-10-15 Thread via GitHub
nastra commented on issue #8832: URL: https://github.com/apache/iceberg/issues/8832#issuecomment-1763814466 Sounds like a similar issue to https://github.com/apache/iceberg/issues/8321. @amogh-jahagirdar should we get #8321 reviewed first and then we can address any leftovers in this PR?

Re: [PR] Build: Bump arrow from 12.0.1 to 13.0.0 [iceberg]

2023-10-15 Thread via GitHub
nastra merged PR #8785: URL: https://github.com/apache/iceberg/pull/8785 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Core: Deprecate AssertTableUUID / introduce AssertUUID [iceberg]

2023-10-15 Thread via GitHub
nastra commented on code in PR #8831: URL: https://github.com/apache/iceberg/pull/8831#discussion_r1360143450 ## core/src/main/java/org/apache/iceberg/UpdateRequirement.java: ## @@ -36,6 +36,8 @@ public void validate(TableMetadata base) { } } + /** @deprecated will be

Re: [PR] Build: Bump nessie from 0.71.1 to 0.72.0 [iceberg]

2023-10-15 Thread via GitHub
nastra merged PR #8835: URL: https://github.com/apache/iceberg/pull/8835 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Build: Bump com.google.cloud:libraries-bom from 26.24.0 to 26.25.0 [iceberg]

2023-10-15 Thread via GitHub
nastra merged PR #8841: URL: https://github.com/apache/iceberg/pull/8841 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.20.162 to 2.21.0 [iceberg]

2023-10-15 Thread via GitHub
nastra merged PR #8838: URL: https://github.com/apache/iceberg/pull/8838 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Docs: Remove AWS Version [iceberg]

2023-10-15 Thread via GitHub
nastra merged PR #8842: URL: https://github.com/apache/iceberg/pull/8842 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Core: Deprecate AssertTableUUID / introduce AssertUUID [iceberg]

2023-10-15 Thread via GitHub
ajantha-bhat commented on code in PR #8831: URL: https://github.com/apache/iceberg/pull/8831#discussion_r1360132056 ## core/src/main/java/org/apache/iceberg/UpdateRequirement.java: ## @@ -36,6 +36,8 @@ public void validate(TableMetadata base) { } } + /** @deprecated w

Re: [PR] Infra: Cleanup labeler.yml [iceberg]

2023-10-15 Thread via GitHub
ajantha-bhat commented on code in PR #8795: URL: https://github.com/apache/iceberg/pull/8795#discussion_r1360127644 ## .github/labeler.yml: ## @@ -55,16 +55,12 @@ HIVE: - hive3/**/* - hive-metastore/**/* - hive-runtime/**/* + - hive3-orc-bundle/**/* DATA: - data/**

Re: [PR] feat: manifest list writer [iceberg-rust]

2023-10-15 Thread via GitHub
liurenjie1024 commented on PR #76: URL: https://github.com/apache/iceberg-rust/pull/76#issuecomment-1763745002 CC @Xuanwo @JanKaul PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] Flink: Custom partitioner for bucket partitions [iceberg]

2023-10-15 Thread via GitHub
stevenzwu commented on PR #7161: URL: https://github.com/apache/iceberg/pull/7161#issuecomment-1763706777 is the partition time an event time or ingestion/processing time? or asking in a different way, how many active minutes do the Flink writer job process for every commit cycle? I

Re: [PR] Core: fix reading of split offsets in manifests [iceberg]

2023-10-15 Thread via GitHub
advancedxy commented on PR #8834: URL: https://github.com/apache/iceberg/pull/8834#issuecomment-1763676271 Other than to revert the optimize in https://github.com/apache/iceberg/pull/8336, is it better to invalidate the cached `splitOffsetList`? The proposed change is in the `org.apache.ic

Re: [PR] Flink: Custom partitioner for bucket partitions [iceberg]

2023-10-15 Thread via GitHub
chenwyi2 commented on PR #7161: URL: https://github.com/apache/iceberg/pull/7161#issuecomment-1763604926 yes, I am creating very fine grained partitions, because i want to query and comput some business metrics between minutes ss fast as possible. As for bucket number, i use a fomula QPS *

Re: [I] Support data file compaction on branch via Spark `rewrite_data_files` procedure [iceberg]

2023-10-15 Thread via GitHub
ajantha-bhat commented on issue #7272: URL: https://github.com/apache/iceberg/issues/7272#issuecomment-1763555781 Tracked by : https://github.com/apache/iceberg/issues/8762 https://github.com/apache/iceberg/pull/8797 -- This is an automated message from the Apache Git Service. To respon

Re: [I] Support data file compaction on branch via Spark `rewrite_data_files` procedure [iceberg]

2023-10-15 Thread via GitHub
github-actions[bot] closed issue #7272: Support data file compaction on branch via Spark `rewrite_data_files` procedure URL: https://github.com/apache/iceberg/issues/7272 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [I] Support data file compaction on branch via Spark `rewrite_data_files` procedure [iceberg]

2023-10-15 Thread via GitHub
github-actions[bot] commented on issue #7272: URL: https://github.com/apache/iceberg/issues/7272#issuecomment-1763548134 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [PR] Flink:backport PR to 1.15 #7360: Implement data statistics coordinator to aggregate data statistics from operator subtasks [iceberg]

2023-10-15 Thread via GitHub
stevenzwu merged PR #8749: URL: https://github.com/apache/iceberg/pull/8749 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Flink:backport PR to 1.16 #7360: Implement data statistics coordinator to aggregate data statistics from operator subtasks [iceberg]

2023-10-15 Thread via GitHub
stevenzwu merged PR #8747: URL: https://github.com/apache/iceberg/pull/8747 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

[PR] Build: Bump ray from 2.7.0 to 2.7.1 [iceberg-python]

2023-10-15 Thread via GitHub
dependabot[bot] opened a new pull request, #77: URL: https://github.com/apache/iceberg-python/pull/77 Bumps [ray](https://github.com/ray-project/ray) from 2.7.0 to 2.7.1. Release notes Sourced from https://github.com/ray-project/ray/releases";>ray's releases. Ray-2.7.1 Rel

[PR] Build: Bump griffe from 0.36.4 to 0.36.5 [iceberg-python]

2023-10-15 Thread via GitHub
dependabot[bot] opened a new pull request, #76: URL: https://github.com/apache/iceberg-python/pull/76 Bumps [griffe](https://github.com/mkdocstrings/griffe) from 0.36.4 to 0.36.5. Release notes Sourced from https://github.com/mkdocstrings/griffe/releases";>griffe's releases.

[PR] Build: Bump mypy-boto3-glue from 1.28.36 to 1.28.63 [iceberg-python]

2023-10-15 Thread via GitHub
dependabot[bot] opened a new pull request, #75: URL: https://github.com/apache/iceberg-python/pull/75 Bumps [mypy-boto3-glue](https://github.com/youtype/mypy_boto3_builder) from 1.28.36 to 1.28.63. Commits See full diff in https://github.com/youtype/mypy_boto3_builder/commits";

[PR] Build: Bump mkdocstrings-python from 1.7.2 to 1.7.3 [iceberg-python]

2023-10-15 Thread via GitHub
dependabot[bot] opened a new pull request, #74: URL: https://github.com/apache/iceberg-python/pull/74 Bumps [mkdocstrings-python](https://github.com/mkdocstrings/python) from 1.7.2 to 1.7.3. Release notes Sourced from https://github.com/mkdocstrings/python/releases";>mkdocstrings-p

[PR] Build: Bump moto from 4.2.5 to 4.2.6 [iceberg-python]

2023-10-15 Thread via GitHub
dependabot[bot] opened a new pull request, #73: URL: https://github.com/apache/iceberg-python/pull/73 Bumps [moto](https://github.com/getmoto/moto) from 4.2.5 to 4.2.6. Changelog Sourced from https://github.com/getmoto/moto/blob/master/CHANGELOG.md";>moto's changelog. 4.2.6

Re: [I] throw exception : InvalidOperationException(message:The following columns have types incompatible with the existing columns in their respective positions : idd1) when add column [iceberg]

2023-10-15 Thread via GitHub
pawankukreja01 commented on issue #3747: URL: https://github.com/apache/iceberg/issues/3747#issuecomment-1763511414 Based on the error message you provided, it seems like the column idd has an incompatible type with the existing columns in their respective positions. To add a column t

Re: [PR] Core: fix reading of split offsets in manifests [iceberg]

2023-10-15 Thread via GitHub
bryanck commented on code in PR #8834: URL: https://github.com/apache/iceberg/pull/8834#discussion_r1359928626 ## core/src/test/java/org/apache/iceberg/TestManifestReader.java: ## @@ -41,6 +44,12 @@ public static Object[] parameters() { return new Object[] {1, 2}; } +

Re: [PR] Core: fix reading of split offsets in manifests [iceberg]

2023-10-15 Thread via GitHub
bryanck commented on code in PR #8834: URL: https://github.com/apache/iceberg/pull/8834#discussion_r1359928395 ## core/src/test/java/org/apache/iceberg/TableTestBase.java: ## @@ -127,6 +129,7 @@ public class TableTestBase { .withFileSizeInBytes(10) .withPar

Re: [PR] Core: fix reading of split offsets in manifests [iceberg]

2023-10-15 Thread via GitHub
RussellSpitzer commented on code in PR #8834: URL: https://github.com/apache/iceberg/pull/8834#discussion_r1359918684 ## core/src/test/java/org/apache/iceberg/TestManifestReader.java: ## @@ -41,6 +44,12 @@ public static Object[] parameters() { return new Object[] {1, 2};

Re: [PR] push down min/max/count to iceberg [iceberg]

2023-10-15 Thread via GitHub
huaxingao commented on PR #6252: URL: https://github.com/apache/iceberg/pull/6252#issuecomment-1763425359 @atifiu That's right. If the filters are not completely evaluated on iceberg, then Spark has to evaluate the filters, the Min/Max/Count values might change after the filtering. Therefor

[PR] Docs: Remove AWS Version [iceberg]

2023-10-15 Thread via GitHub
Fokko opened a new pull request, #8842: URL: https://github.com/apache/iceberg/pull/8842 It is easy to let these go out of sync. And people should just use the bundle :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Add `spec_id` back to data file [iceberg-python]

2023-10-15 Thread via GitHub
Fokko commented on PR #63: URL: https://github.com/apache/iceberg-python/pull/63#issuecomment-1763395131 This is great, thanks @puchengy for fixing this 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Add `spec_id` back to data file [iceberg-python]

2023-10-15 Thread via GitHub
Fokko merged PR #63: URL: https://github.com/apache/iceberg-python/pull/63 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

[PR] Bump version to 0.6.0 [iceberg-python]

2023-10-15 Thread via GitHub
Fokko opened a new pull request, #72: URL: https://github.com/apache/iceberg-python/pull/72 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

Re: [PR] Update to the latest version [iceberg-python]

2023-10-15 Thread via GitHub
Fokko closed pull request #67: Update to the latest version URL: https://github.com/apache/iceberg-python/pull/67 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

[PR] Remove python working directory [iceberg-python]

2023-10-15 Thread via GitHub
Fokko opened a new pull request, #71: URL: https://github.com/apache/iceberg-python/pull/71 This is not applicable anymore now Python has its own repository -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] rewrite_position_delete_files leads to error [iceberg]

2023-10-15 Thread via GitHub
Fokko commented on issue #8045: URL: https://github.com/apache/iceberg/issues/8045#issuecomment-1763386067 @atifiu The patch has been backported to Spark 3.3 as well: https://github.com/apache/iceberg/pull/8069 so you only need to upgrade your Iceberg dependency. -- This is an automated

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.40.1 to 1.40.2 [iceberg-go]

2023-10-15 Thread via GitHub
dependabot[bot] opened a new pull request, #22: URL: https://github.com/apache/iceberg-go/pull/22 Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.40.1 to 1.40.2. Commits https://github.com/aws/aws-sdk-go-v2/commit/e3b97d2a02cd4e27c4

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.13.42 to 1.13.43 [iceberg-go]

2023-10-15 Thread via GitHub
dependabot[bot] opened a new pull request, #21: URL: https://github.com/apache/iceberg-go/pull/21 Bumps [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) from 1.13.42 to 1.13.43. Commits https://github.com/aws/aws-sdk-go-v2/commit/e3b97d2a02cd4e2

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.21.1 to 1.21.2 [iceberg-go]

2023-10-15 Thread via GitHub
dependabot[bot] opened a new pull request, #20: URL: https://github.com/apache/iceberg-go/pull/20 Bumps [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) from 1.21.1 to 1.21.2. Commits https://github.com/aws/aws-sdk-go-v2/commit/e3b97d2a02cd4e27c40224f05aa1a7

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.18.44 to 1.18.45 [iceberg-go]

2023-10-15 Thread via GitHub
dependabot[bot] opened a new pull request, #19: URL: https://github.com/apache/iceberg-go/pull/19 Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.18.44 to 1.18.45. Commits https://github.com/aws/aws-sdk-go-v2/commit/e3b97d2a02cd4e27c402