Re: [I] Remove `RuntimeIOException` usage [iceberg]

2023-12-03 Thread via GitHub
ShockleysxX commented on issue #9208: URL: https://github.com/apache/iceberg/issues/9208#issuecomment-1838011225 @Fokko, i want to have a try, plz assign to me, thanks~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] iceberg-parquet: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

2023-12-03 Thread via GitHub
nastra commented on code in PR #9161: URL: https://github.com/apache/iceberg/pull/9161#discussion_r1413484702 ## parquet/src/test/java/org/apache/iceberg/parquet/TestDictionaryRowGroupFilter.java: ## @@ -18,37 +18,13 @@ */ package org.apache.iceberg.parquet; -import static

Re: [PR] iceberg-parquet: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

2023-12-03 Thread via GitHub
nastra commented on code in PR #9161: URL: https://github.com/apache/iceberg/pull/9161#discussion_r1413482785 ## parquet/src/test/java/org/apache/iceberg/parquet/TestDictionaryRowGroupFilter.java: ## @@ -156,22 +156,22 @@ public class TestDictionaryRowGroupFilter { private Me

[I] Remove `RuntimeIOException` usage [iceberg]

2023-12-03 Thread via GitHub
Fokko opened a new issue, #9208: URL: https://github.com/apache/iceberg/issues/9208 ### Feature Request / Improvement The `RuntimeIOException` has been deprecated, but is still being used in many places. Would be good to remove this. Would be a good idea to maybe remove this pe

Re: [I] How to connect apache iceberg to minio [iceberg]

2023-12-03 Thread via GitHub
nastra commented on issue #9205: URL: https://github.com/apache/iceberg/issues/9205#issuecomment-1837984057 @ExplorData24 I'm assuming you tried to follow https://iceberg.apache.org/spark-quickstart/ which uses Iceberg + Spark + Minio? I see that you're using Iceberg 1.0.0, so you might wan

Re: [I] unable to create iceberg table in minio [iceberg]

2023-12-03 Thread via GitHub
nastra commented on issue #9206: URL: https://github.com/apache/iceberg/issues/9206#issuecomment-1837978224 Closing this as this is a duplicate of https://github.com/apache/iceberg/issues/9205 -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [I] unable to create iceberg table in minio [iceberg]

2023-12-03 Thread via GitHub
nastra closed issue #9206: unable to create iceberg table in minio URL: https://github.com/apache/iceberg/issues/9206 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

Re: [PR] feat: Add hms catalog layout [iceberg-rust]

2023-12-03 Thread via GitHub
Xuanwo commented on PR #112: URL: https://github.com/apache/iceberg-rust/pull/112#issuecomment-1837974171 cc @liurenjie1024 @Fokko @ZENOTME @JanKaul -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[PR] feat: Add hms catalog layout [iceberg-rust]

2023-12-03 Thread via GitHub
Xuanwo opened a new pull request, #112: URL: https://github.com/apache/iceberg-rust/pull/112 This PR is a basic layout for hms support. This PR solely implements `list_namespaces`, allowing us to concentrate on crucial matters such as general API style and creating room for future im

Re: [I] Flink Rewrite Files Action OOM [iceberg]

2023-12-03 Thread via GitHub
bhupixb commented on issue #9193: URL: https://github.com/apache/iceberg/issues/9193#issuecomment-1837959214 @pvary We have tried increasing the memory already, but it didn't help. We are also unable that understand where it's consuming this much memory. Also, there are a total of ~350 fi

Re: [PR] Update table metadata [iceberg-python]

2023-12-03 Thread via GitHub
HonahX commented on code in PR #139: URL: https://github.com/apache/iceberg-python/pull/139#discussion_r1413440829 ## pyiceberg/table/__init__.py: ## @@ -350,6 +357,241 @@ class RemovePropertiesUpdate(TableUpdate): removals: List[str] +class TableMetadataUpdateContext:

Re: [PR] iceberg-parquet: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

2023-12-03 Thread via GitHub
pvary commented on code in PR #9161: URL: https://github.com/apache/iceberg/pull/9161#discussion_r1413428163 ## parquet/src/test/java/org/apache/iceberg/parquet/TestDictionaryRowGroupFilter.java: ## @@ -79,16 +53,39 @@ import org.apache.parquet.hadoop.metadata.ColumnChunkMetaDa

Re: [PR] Core: Add PartitionMap [iceberg]

2023-12-03 Thread via GitHub
jerqi commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1413426617 ## core/src/main/java/org/apache/iceberg/util/PartitionMap.java: ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contr

Re: [PR] Core: Add PartitionMap [iceberg]

2023-12-03 Thread via GitHub
qqqttt123 commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1413424718 ## core/src/main/java/org/apache/iceberg/util/PartitionMap.java: ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more c

Re: [PR] Core: Add PartitionMap [iceberg]

2023-12-03 Thread via GitHub
qqqttt123 commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1413424718 ## core/src/main/java/org/apache/iceberg/util/PartitionMap.java: ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more c

Re: [I] Flink Rewrite Files Action OOM [iceberg]

2023-12-03 Thread via GitHub
pvary commented on issue #9193: URL: https://github.com/apache/iceberg/issues/9193#issuecomment-1837926494 I would move forward in 2 directions: - Check which named (Iceberg/Flink class) references those big HashMaps - Check how many files are in the tables If there are many file

Re: [PR] Update table metadata [iceberg-python]

2023-12-03 Thread via GitHub
HonahX commented on code in PR #139: URL: https://github.com/apache/iceberg-python/pull/139#discussion_r1413422916 ## pyiceberg/table/__init__.py: ## @@ -350,6 +357,241 @@ class RemovePropertiesUpdate(TableUpdate): removals: List[str] +class TableMetadataUpdateContext:

Re: [PR] iceberg-parquet: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

2023-12-03 Thread via GitHub
lisirrx commented on code in PR #9161: URL: https://github.com/apache/iceberg/pull/9161#discussion_r1413411369 ## parquet/src/test/java/org/apache/iceberg/parquet/TestDictionaryRowGroupFilter.java: ## @@ -156,22 +156,22 @@ public class TestDictionaryRowGroupFilter { private M

Re: [I] why should we use file.createOrOverwrite to create a datafile or manifest file? [iceberg]

2023-12-03 Thread via GitHub
chenwyi2 closed issue #9100: why should we use file.createOrOverwrite to create a datafile or manifest file? URL: https://github.com/apache/iceberg/issues/9100 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] why should we use file.createOrOverwrite to create a datafile or manifest file? [iceberg]

2023-12-03 Thread via GitHub
chenwyi2 commented on issue #9100: URL: https://github.com/apache/iceberg/issues/9100#issuecomment-1837902246 i think this is not iceberg problem, i will close this issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Build: Bump aiohttp from 3.8.6 to 3.9.0 [iceberg-python]

2023-12-03 Thread via GitHub
Fokko commented on PR #168: URL: https://github.com/apache/iceberg-python/pull/168#issuecomment-1837867228 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Build: Bump pydantic from 2.4.2 to 2.5.2 [iceberg-python]

2023-12-03 Thread via GitHub
Fokko merged PR #159: URL: https://github.com/apache/iceberg-python/pull/159 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Build: Bump aiohttp from 3.8.6 to 3.9.0 [iceberg-python]

2023-12-03 Thread via GitHub
Fokko commented on PR #168: URL: https://github.com/apache/iceberg-python/pull/168#issuecomment-1837800326 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Build: Bump ray from 2.7.1 to 2.8.1 [iceberg-python]

2023-12-03 Thread via GitHub
Fokko commented on PR #174: URL: https://github.com/apache/iceberg-python/pull/174#issuecomment-1837800271 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Build: Bump mypy-boto3-glue from 1.29.2 to 1.33.5 [iceberg-python]

2023-12-03 Thread via GitHub
Fokko merged PR #172: URL: https://github.com/apache/iceberg-python/pull/172 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Update table metadata [iceberg-python]

2023-12-03 Thread via GitHub
Fokko commented on code in PR #139: URL: https://github.com/apache/iceberg-python/pull/139#discussion_r1413335622 ## pyiceberg/table/__init__.py: ## @@ -350,6 +357,241 @@ class RemovePropertiesUpdate(TableUpdate): removals: List[str] +class TableMetadataUpdateContext: +

Re: [PR] Fix downgrading of schema [iceberg-python]

2023-12-03 Thread via GitHub
Fokko merged PR #176: URL: https://github.com/apache/iceberg-python/pull/176 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Build: Bump mypy-boto3-glue from 1.29.2 to 1.33.5 [iceberg-python]

2023-12-03 Thread via GitHub
Fokko commented on PR #172: URL: https://github.com/apache/iceberg-python/pull/172#issuecomment-1837794552 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Fix downgrading of schema [iceberg-python]

2023-12-03 Thread via GitHub
Fokko commented on PR #176: URL: https://github.com/apache/iceberg-python/pull/176#issuecomment-1837794358 Thanks @danielcweeks and @HonahX for the review 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] Fix downgrading of schema [iceberg-python]

2023-12-03 Thread via GitHub
Fokko commented on code in PR #176: URL: https://github.com/apache/iceberg-python/pull/176#discussion_r1413327467 ## pyiceberg/table/__init__.py: ## @@ -1412,13 +1412,22 @@ def commit(self) -> None: """Apply the pending changes and commit.""" new_schema = self.

Re: [PR] test: Add integration tests for rest catalog. [iceberg-rust]

2023-12-03 Thread via GitHub
Xuanwo commented on code in PR #109: URL: https://github.com/apache/iceberg-rust/pull/109#discussion_r1413326583 ## crates/catalog/rest/Cargo.toml: ## @@ -40,5 +41,7 @@ urlencoding = { workspace = true } uuid = { workspace = true, features = ["v4"] } [dev-dependencies] +iceb

Re: [PR] test: Add integration tests for rest catalog. [iceberg-rust]

2023-12-03 Thread via GitHub
liurenjie1024 commented on code in PR #109: URL: https://github.com/apache/iceberg-rust/pull/109#discussion_r1413325009 ## crates/catalog/rest/Cargo.toml: ## @@ -40,5 +41,7 @@ urlencoding = { workspace = true } uuid = { workspace = true, features = ["v4"] } [dev-dependencies

Re: [PR] test: Add integration tests for rest catalog. [iceberg-rust]

2023-12-03 Thread via GitHub
Xuanwo commented on code in PR #109: URL: https://github.com/apache/iceberg-rust/pull/109#discussion_r1413324431 ## crates/catalog/rest/tests/rest_catalog_test.rs: ## @@ -0,0 +1,376 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license

Re: [PR] test: Add integration tests for rest catalog. [iceberg-rust]

2023-12-03 Thread via GitHub
Xuanwo commented on code in PR #109: URL: https://github.com/apache/iceberg-rust/pull/109#discussion_r1413323814 ## crates/catalog/rest/Cargo.toml: ## @@ -40,5 +41,7 @@ urlencoding = { workspace = true } uuid = { workspace = true, features = ["v4"] } [dev-dependencies] +iceb

Re: [PR] test: Add integration tests for rest catalog. [iceberg-rust]

2023-12-03 Thread via GitHub
liurenjie1024 commented on code in PR #109: URL: https://github.com/apache/iceberg-rust/pull/109#discussion_r1413311800 ## crates/catalog/rest/Cargo.toml: ## @@ -40,5 +41,7 @@ urlencoding = { workspace = true } uuid = { workspace = true, features = ["v4"] } [dev-dependencies

Re: [PR] refactor: Removing type infromation in `Struct` literal. [iceberg-rust]

2023-12-03 Thread via GitHub
liurenjie1024 commented on code in PR #103: URL: https://github.com/apache/iceberg-rust/pull/103#discussion_r1413303245 ## crates/iceberg/src/spec/values.rs: ## @@ -554,13 +554,14 @@ impl From<&Literal> for JsonValue { PrimitiveLiteral::Decimal(_) => todo!(),

Re: [PR] Fix downgrading of schema [iceberg-python]

2023-12-03 Thread via GitHub
HonahX commented on code in PR #176: URL: https://github.com/apache/iceberg-python/pull/176#discussion_r1413005467 ## pyiceberg/table/__init__.py: ## @@ -1412,13 +1412,22 @@ def commit(self) -> None: """Apply the pending changes and commit.""" new_schema = self

Re: [PR] [Draft] rewrite v2 tables by skipping deletes planning and join deletes data tables [iceberg]

2023-12-03 Thread via GitHub
zinking commented on PR #8807: URL: https://github.com/apache/iceberg/pull/8807#issuecomment-1837727725 ![big_joins](https://github.com/apache/iceberg/assets/118241/25f874bc-ef43-455b-a80e-42da01e8a21b) -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [I] Docs: Improve catalog documentation [iceberg]

2023-12-03 Thread via GitHub
bitsondatadev commented on issue #7785: URL: https://github.com/apache/iceberg/issues/7785#issuecomment-1837714666 Git outta here stale bot, go on git!! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] Explore: option of creating single delete file to reduce number of delete files in a partition in MOR [iceberg]

2023-12-03 Thread via GitHub
github-actions[bot] closed issue #7690: Explore: option of creating single delete file to reduce number of delete files in a partition in MOR URL: https://github.com/apache/iceberg/issues/7690 -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [I] Explore: option of creating single delete file to reduce number of delete files in a partition in MOR [iceberg]

2023-12-03 Thread via GitHub
github-actions[bot] commented on issue #7690: URL: https://github.com/apache/iceberg/issues/7690#issuecomment-1837652829 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] reset catalog can't reslove s3 domain name [iceberg]

2023-12-03 Thread via GitHub
github-actions[bot] commented on issue #7709: URL: https://github.com/apache/iceberg/issues/7709#issuecomment-1837652801 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Iceberg Schema moving inner column after adding new StructType doesn't work. [iceberg]

2023-12-03 Thread via GitHub
github-actions[bot] commented on issue #7786: URL: https://github.com/apache/iceberg/issues/7786#issuecomment-1837652748 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Docs: Improve catalog documentation [iceberg]

2023-12-03 Thread via GitHub
github-actions[bot] commented on issue #7785: URL: https://github.com/apache/iceberg/issues/7785#issuecomment-1837652761 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

[I] unable to create iceberg table in minio [iceberg]

2023-12-03 Thread via GitHub
ExplorData24 opened a new issue, #9206: URL: https://github.com/apache/iceberg/issues/9206 ### Apache Iceberg version 1.0.0 ### Query engine Spark ### Please describe the bug 🐞 Hello everyone. I tried to run the following code to create a table in Min

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.21.29 to 2.21.37 [iceberg]

2023-12-03 Thread via GitHub
Fokko commented on PR #9204: URL: https://github.com/apache/iceberg/pull/9204#issuecomment-1837542806 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Build: Bump org.apache.httpcomponents.client5:httpclient5 from 5.2.1 to 5.2.3 [iceberg]

2023-12-03 Thread via GitHub
Fokko merged PR #9202: URL: https://github.com/apache/iceberg/pull/9202 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [I] to_pandas() API which converts iceberg table scan to a pd.DataFrame will lost datetime data type and row order [iceberg-python]

2023-12-03 Thread via GitHub
mapleFU commented on issue #132: URL: https://github.com/apache/iceberg-python/issues/132#issuecomment-1837528576 I think that it's separate into multiple problem. 1. In iceberg, maybe we can maintaining some statistics for multiple columns min-max for a file 2. In Reading SDK, ma

[I] How to connect apache iceberg to minio [iceberg]

2023-12-03 Thread via GitHub
ExplorData24 opened a new issue, #9205: URL: https://github.com/apache/iceberg/issues/9205 ### Apache Iceberg version 1.0.0 ### Query engine Spark ### Please describe the bug 🐞 Hello everyone. I tried to run the following code to create a table in

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.21.2 to 1.23.1 [iceberg-go]

2023-12-03 Thread via GitHub
dependabot[bot] commented on PR #39: URL: https://github.com/apache/iceberg-go/pull/39#issuecomment-1837404728 Superseded by #43. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.21.2 to 1.23.5 [iceberg-go]

2023-12-03 Thread via GitHub
dependabot[bot] opened a new pull request, #43: URL: https://github.com/apache/iceberg-go/pull/43 Bumps [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) from 1.21.2 to 1.23.5. Commits https://github.com/aws/aws-sdk-go-v2/commit/ce842a7ed4981d404542e66d3ccd71

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.21.2 to 1.23.1 [iceberg-go]

2023-12-03 Thread via GitHub
dependabot[bot] closed pull request #39: build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.21.2 to 1.23.1 URL: https://github.com/apache/iceberg-go/pull/39 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.19.1 to 1.25.5 [iceberg-go]

2023-12-03 Thread via GitHub
dependabot[bot] commented on PR #40: URL: https://github.com/apache/iceberg-go/pull/40#issuecomment-1837404219 Superseded by #42. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.19.1 to 1.25.5 [iceberg-go]

2023-12-03 Thread via GitHub
dependabot[bot] closed pull request #40: build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.19.1 to 1.25.5 URL: https://github.com/apache/iceberg-go/pull/40 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.19.1 to 1.25.11 [iceberg-go]

2023-12-03 Thread via GitHub
dependabot[bot] opened a new pull request, #42: URL: https://github.com/apache/iceberg-go/pull/42 Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.19.1 to 1.25.11. Commits https://github.com/aws/aws-sdk-go-v2/commit/ce842a7ed4981d404542e