Re: [I] Caused by: java.net.SocketException: Connection reset [iceberg]

2024-03-17 Thread via GitHub
shanzi commented on issue #9444: URL: https://github.com/apache/iceberg/issues/9444#issuecomment-2003003511 After applied @javrasya 's fix, we no longer encounter the connection reset error, but sometimes we will get `org.apache.http.conn.ConnectionPoolTimeoutException`. So I guess it's ver

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-17 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1527865180 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -113,6 +124,24 @@ public void initialize(String inputName, Map properties) { this.cl

Re: [PR] Build: Bump orc from 1.9.2 to 2.0.0 [iceberg]

2024-03-17 Thread via GitHub
manuzhang commented on PR #9913: URL: https://github.com/apache/iceberg/pull/9913#issuecomment-2002889406 NVM, I find my mistake in the previous PR and submit a new fix https://github.com/apache/iceberg/pull/9981 -- This is an automated message from the Apache Git Service. To respond to t

[PR] Build: Fix ignoring major version update in dependabot [iceberg]

2024-03-17 Thread via GitHub
manuzhang opened a new pull request, #9981: URL: https://github.com/apache/iceberg/pull/9981 I got the config wrong in the previous attempt #9806. This PR fixes it following the [official example](https://github.blog/changelog/2021-05-21-dependabot-version-updates-can-now-ignore-major-minor

Re: [I] Add OAuth Support for iceberg rest client. [iceberg-rust]

2024-03-17 Thread via GitHub
liurenjie1024 closed issue #238: Add OAuth Support for iceberg rest client. URL: https://github.com/apache/iceberg-rust/issues/238 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] feat: implement OAuth for catalog rest client [iceberg-rust]

2024-03-17 Thread via GitHub
liurenjie1024 merged PR #254: URL: https://github.com/apache/iceberg-rust/pull/254 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] API: Fix default FileIO#newInputFile ManifestFile, DataFile and DeleteFile implementation to pass lengths [iceberg]

2024-03-17 Thread via GitHub
ajantha-bhat commented on PR #9953: URL: https://github.com/apache/iceberg/pull/9953#issuecomment-2002774794 ping @danielcweeks, @rdblue, @nastra -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Spark 3.5: Spark action to compute the partition stats [iceberg]

2024-03-17 Thread via GitHub
ajantha-bhat commented on PR #9437: URL: https://github.com/apache/iceberg/pull/9437#issuecomment-2002773904 3.3 CI is failed due to known flaky issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Spark 3.5: Spark action to compute the partition stats [iceberg]

2024-03-17 Thread via GitHub
ajantha-bhat closed pull request #9437: Spark 3.5: Spark action to compute the partition stats URL: https://github.com/apache/iceberg/pull/9437 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] Migrate Manifest, FormatVersion and LocationProvider files in Core to JUnit5 [iceberg]

2024-03-17 Thread via GitHub
tomtongue commented on PR #9964: URL: https://github.com/apache/iceberg/pull/9964#issuecomment-2002767975 @nastra Migrate additional classes to JUnit5. Could you review when you have time? -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] Build: Bump orc from 1.9.2 to 2.0.0 [iceberg]

2024-03-17 Thread via GitHub
manuzhang commented on PR #9913: URL: https://github.com/apache/iceberg/pull/9913#issuecomment-2002763215 @nastra do you get a chance to check the log? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [I] ValueError: Unrecognized filesystem type in URI: abfss [iceberg-python]

2024-03-17 Thread via GitHub
djouallah closed issue #527: ValueError: Unrecognized filesystem type in URI: abfss URL: https://github.com/apache/iceberg-python/issues/527 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [I] ValueError: Unrecognized filesystem type in URI: abfss [iceberg-python]

2024-03-17 Thread via GitHub
djouallah commented on issue #527: URL: https://github.com/apache/iceberg-python/issues/527#issuecomment-2002696867 Thanks, it works now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [I] Provide full unit tests for flink savepoints workflow [iceberg]

2024-03-17 Thread via GitHub
github-actions[bot] commented on issue #2190: URL: https://github.com/apache/iceberg/issues/2190#issuecomment-2002673733 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Flink: move away from InternalSerializers for cloning RowData [iceberg]

2024-03-17 Thread via GitHub
github-actions[bot] commented on issue #2184: URL: https://github.com/apache/iceberg/issues/2184#issuecomment-2002673716 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Flink: Can't use flink sql to create hive catalog [iceberg]

2024-03-17 Thread via GitHub
github-actions[bot] commented on issue #2181: URL: https://github.com/apache/iceberg/issues/2181#issuecomment-2002673691 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] now it can delete [iceberg]

2024-03-17 Thread via GitHub
github-actions[bot] commented on issue #2183: URL: https://github.com/apache/iceberg/issues/2183#issuecomment-2002673702 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Flink : migrate existing hive table to iceberg table [iceberg]

2024-03-17 Thread via GitHub
github-actions[bot] commented on issue #2162: URL: https://github.com/apache/iceberg/issues/2162#issuecomment-2002673680 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Another hive connection leak issue from spark [iceberg]

2024-03-17 Thread via GitHub
github-actions[bot] commented on issue #2137: URL: https://github.com/apache/iceberg/issues/2137#issuecomment-2002673673 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Set table properties through Hive [iceberg-python]

2024-03-17 Thread via GitHub
zenfenan commented on issue #206: URL: https://github.com/apache/iceberg-python/issues/206#issuecomment-2002613198 @Fokko Could you please share more context on this one? If I understand this correctly, this is about setting table properties through `HiveCatalog`. That seems to be already a

Re: [PR] [WIP] Implement project for Transform. #264 [iceberg-rust]

2024-03-17 Thread via GitHub
sdd commented on code in PR #269: URL: https://github.com/apache/iceberg-rust/pull/269#discussion_r1527606344 ## crates/iceberg/src/spec/transform.rs: ## @@ -261,6 +269,50 @@ impl Transform { _ => self == other, } } +/// Projects predicate to `Tran

Re: [PR] [WIP] Implement project for Transform. #264 [iceberg-rust]

2024-03-17 Thread via GitHub
marvinlanhenke commented on PR #269: URL: https://github.com/apache/iceberg-rust/pull/269#issuecomment-2002599583 @liurenjie1024 @Xuanwo the main design concerns I want to discuss: 1. exposing `op` and `literals` from expressions. For prototyping I did this directly on the express

Re: [PR] Support for REPLACE TABLE operation [iceberg-python]

2024-03-17 Thread via GitHub
anupam-saini commented on code in PR #433: URL: https://github.com/apache/iceberg-python/pull/433#discussion_r1527602013 ## pyiceberg/catalog/__init__.py: ## @@ -710,6 +760,45 @@ def _get_updated_props_and_update_summary( return properties_update_summary, updated_prop

Re: [PR] Fix CommitTableRequest serialisation [iceberg-python]

2024-03-17 Thread via GitHub
Fokko merged PR #525: URL: https://github.com/apache/iceberg-python/pull/525 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Add hive metastore catalog support [iceberg-rust]

2024-03-17 Thread via GitHub
marvinlanhenke commented on issue #113: URL: https://github.com/apache/iceberg-rust/issues/113#issuecomment-2002574619 @liurenjie1024 I'm done with the implementation and ready to push the PR. However, I think it would be great to review #272 first, since it blocked my implementation

Re: [PR] Metadata Serde + default partition_specs and sort_orders [iceberg-rust]

2024-03-17 Thread via GitHub
marvinlanhenke commented on PR #272: URL: https://github.com/apache/iceberg-rust/pull/272#issuecomment-2002565662 cc @liurenjie1024 @Xuanwo @ZENOTME -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] ValueError: Unrecognized filesystem type in URI: abfss [iceberg-python]

2024-03-17 Thread via GitHub
kevinjqliu commented on issue #527: URL: https://github.com/apache/iceberg-python/issues/527#issuecomment-2002552232 oh interesting, this is because of a dependency issue. The actual error shows up when you try to import that class ``` from pyiceberg.io.fsspec import FsspecFileI

Re: [I] Caused by: java.net.SocketException: Connection reset [iceberg]

2024-03-17 Thread via GitHub
shanzi commented on issue #9444: URL: https://github.com/apache/iceberg/issues/9444#issuecomment-2002529412 We are using the new Iceberg 1.5.0 with Flink 1.18.1 and have exactly the same issue. Is it possible that Flink will open more connections (especially with the FLIP-27 sources) or

[PR] Hive: Test to do validation hive content and iceberg table with the same name [iceberg]

2024-03-17 Thread via GitHub
nk1506 opened a new pull request, #9980: URL: https://github.com/apache/iceberg/pull/9980 Hive should not allow creating an iceberg table, if there is already another content available with same name. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] [WIP] Add `PartitionEvaluator` to allow filtering of files in a table scan (Issue #152) [iceberg-rust]

2024-03-17 Thread via GitHub
sdd commented on PR #241: URL: https://github.com/apache/iceberg-rust/pull/241#issuecomment-2002482131 @liurenjie1024 @Xuanwo @ZENOTME: I've updated this PR, details in the updated description at the top. It builds on the `Bind` implementation that recently got merged and is all set to make

Re: [PR] [WIP] Add `PartitionEvaluator` to allow filtering of files in a table scan (Issue #152) [iceberg-rust]

2024-03-17 Thread via GitHub
sdd commented on code in PR #241: URL: https://github.com/apache/iceberg-rust/pull/241#discussion_r1527514205 ## crates/iceberg/src/expr/predicate.rs: ## @@ -599,6 +635,14 @@ impl Display for BoundPredicate { } } +pub(crate) trait PredicateVisitor { Review Comment: I

Re: [I] Update Table Error: UPDATE TABLE is not supported temporarily. [iceberg]

2024-03-17 Thread via GitHub
suisenkotoba commented on issue #9960: URL: https://github.com/apache/iceberg/issues/9960#issuecomment-2002451738 @nastra yes I use full identifier including catalog name. How can I make sure the table is indeed an iceberg table. I checked the table has metadata tables like snapshots, part

Re: [I] Integrate with datafusion [iceberg-rust]

2024-03-17 Thread via GitHub
ZENOTME commented on issue #242: URL: https://github.com/apache/iceberg-rust/issues/242#issuecomment-2002444278 I'm working the draft so that we can have more clear discussion above it. Blocked by #277 now. -- This is an automated message from the Apache Git Service. To respond to the mes

[PR] feat: Implement the conversion from Iceberg Schema to Arrow Schema [iceberg-rust]

2024-03-17 Thread via GitHub
ZENOTME opened a new pull request, #277: URL: https://github.com/apache/iceberg-rust/pull/277 A reverse version of #258. There exists a similar thing(const, test case) with #258. So we can merge #258 first and I will rebase this and remove the duplication. -- This is an automated messag