Re: [PR] AWS: Support S3 DSSE-KMS encryption [iceberg]

2024-05-24 Thread via GitHub
aajisaka commented on PR #8370: URL: https://github.com/apache/iceberg/pull/8370#issuecomment-2130959997 Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

Re: [PR] Bump typing-extensions from 4.11.0 to 4.12.0 [iceberg-python]

2024-05-24 Thread via GitHub
HonahX merged PR #767: URL: https://github.com/apache/iceberg-python/pull/767 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Bump griffe from 0.45.1 to 0.45.2 [iceberg-python]

2024-05-24 Thread via GitHub
HonahX merged PR #765: URL: https://github.com/apache/iceberg-python/pull/765 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Bump duckdb from 0.10.2 to 0.10.3 [iceberg-python]

2024-05-24 Thread via GitHub
HonahX merged PR #764: URL: https://github.com/apache/iceberg-python/pull/764 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Support creating tags by adding `set_ref_snapshot` API [iceberg-python]

2024-05-24 Thread via GitHub
HonahX commented on code in PR #728: URL: https://github.com/apache/iceberg-python/pull/728#discussion_r1611069223 ## tests/table/test_init.py: ## @@ -652,6 +652,58 @@ def test_update_metadata_add_snapshot(table_v2: Table) -> None: assert new_metadata.last_updated_ms == ne

Re: [I] Run RevAPI without Gradle [iceberg]

2024-05-24 Thread via GitHub
jbonofre commented on issue #10368: URL: https://github.com/apache/iceberg/issues/10368#issuecomment-2130810296 After some investigations, I'm doing: 1. I'm replacing `.palantir/revapi.yml` (specific to the revapi gradle plugin) by a simple file where users can add accepted breaks 2. t

[PR] Flink: Iceberg flink multi table sink and runtime table discoverability [iceberg]

2024-05-24 Thread via GitHub
adamyasharma2797 opened a new pull request, #10376: URL: https://github.com/apache/iceberg/pull/10376 IcebergStreamWriter and IcebergFilesCommitter class have support for writing to only one table. And these tables have to be known before instantiating the pipeline, i.e. at compile time its

Re: [PR] Flink: MultiTable Sink Support with Run-time Table Discoverability [iceberg]

2024-05-24 Thread via GitHub
adamyasharma2797 closed pull request #10375: Flink: MultiTable Sink Support with Run-time Table Discoverability URL: https://github.com/apache/iceberg/pull/10375 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

[PR] Flink: MultiTable Sink Support with Run-time Table Discoverability [iceberg]

2024-05-24 Thread via GitHub
adamyasharma2797 opened a new pull request, #10375: URL: https://github.com/apache/iceberg/pull/10375 IcebergStreamWriter and IcebergFilesCommitter class have support for writing to only one table. And these tables have to be known before instantiating the pipeline, i.e. at compile time its

Re: [I] Correct names in the ManifestList [iceberg-rust]

2024-05-24 Thread via GitHub
gupteaj commented on issue #354: URL: https://github.com/apache/iceberg-rust/issues/354#issuecomment-2130517206 PR : https://github.com/apache/iceberg-rust/pull/380 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

[PR] Rename V2 spec names [iceberg-rust]

2024-05-24 Thread via GitHub
gupteaj opened a new pull request, #380: URL: https://github.com/apache/iceberg-rust/pull/380 Issue : https://github.com/apache/iceberg-rust/issues/354 Problem: From V2 spec following names are not matching added_data_files_count, existing_data_files_count & deleted_data_files_coun

[PR] Bump typing-extensions from 4.11.0 to 4.12.0 [iceberg-python]

2024-05-24 Thread via GitHub
dependabot[bot] opened a new pull request, #767: URL: https://github.com/apache/iceberg-python/pull/767 Bumps [typing-extensions](https://github.com/python/typing_extensions) from 4.11.0 to 4.12.0. Release notes Sourced from https://github.com/python/typing_extensions/releases";>ty

Re: [PR] Core: Throw CommitStateUnknownException if RuntimeException that is not marked as cleanable is thrown [iceberg]

2024-05-24 Thread via GitHub
danielcweeks commented on PR #10373: URL: https://github.com/apache/iceberg/pull/10373#issuecomment-2130322881 @amogh-jahagirdar this might cover a few cases where a RuntimeException is thrown, but I think the underlying problem is more that a lot of the implementations throw CommitFailedEx

Re: [PR] feat: add `ExpressionEvaluator` [iceberg-rust]

2024-05-24 Thread via GitHub
sdd commented on code in PR #363: URL: https://github.com/apache/iceberg-rust/pull/363#discussion_r1613965281 ## crates/iceberg/src/expr/visitors/expression_evaluator.rs: ## @@ -0,0 +1,819 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [I] bug: `StructAccessor` should return `Result>` [iceberg-rust]

2024-05-24 Thread via GitHub
sdd commented on issue #379: URL: https://github.com/apache/iceberg-rust/issues/379#issuecomment-2130275538 Of course, I'll pick this up 👍🏼 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] feat: Adding literals [iceberg-go]

2024-05-24 Thread via GitHub
zeroshade commented on code in PR #76: URL: https://github.com/apache/iceberg-go/pull/76#discussion_r1613885354 ## literals.go: ## @@ -0,0 +1,777 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +/

Re: [PR] feat: Adding literals [iceberg-go]

2024-05-24 Thread via GitHub
zeroshade commented on code in PR #76: URL: https://github.com/apache/iceberg-go/pull/76#discussion_r1613884699 ## literals.go: ## @@ -0,0 +1,777 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +/

Re: [PR] AWS: Support S3 DSSE-KMS encryption [iceberg]

2024-05-24 Thread via GitHub
jackye1995 merged PR #8370: URL: https://github.com/apache/iceberg/pull/8370 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] AWS: Support S3 DSSE-KMS encryption [iceberg]

2024-05-24 Thread via GitHub
jackye1995 commented on PR #8370: URL: https://github.com/apache/iceberg/pull/8370#issuecomment-2129937351 Was waiting for @nastra , but I agree this is ready to be merged, I will go ahead to do that, thanks for the contribution! -- This is an automated message from the Apache Git Service

Re: [PR] Spark 3.5: Support camel case session configs and options [iceberg]

2024-05-24 Thread via GitHub
aokolnychyi commented on PR #10310: URL: https://github.com/apache/iceberg/pull/10310#issuecomment-2129886572 Thanks, @szehon-ho @advancedxy! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Spark 3.5: Support camel case session configs and options [iceberg]

2024-05-24 Thread via GitHub
aokolnychyi merged PR #10310: URL: https://github.com/apache/iceberg/pull/10310 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@icebe

Re: [PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]

2024-05-24 Thread via GitHub
stevenzwu commented on code in PR #10370: URL: https://github.com/apache/iceberg/pull/10370#discussion_r1613698660 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java: ## @@ -124,7 +124,7 @@ public void testBoundedWithSavepoint() thr

Re: [PR] Core: Throw CommitStateUnknownException if RuntimeException that is not marked as cleanable is thrown [iceberg]

2024-05-24 Thread via GitHub
amogh-jahagirdar commented on PR #10373: URL: https://github.com/apache/iceberg/pull/10373#issuecomment-2129842921 A few tests are failing now due to always throwing CommitStateUnknownException; I'll step through and see which exceptions should be marked as cleanable and which should be pro

Re: [PR] AWS: Support S3 DSSE-KMS encryption [iceberg]

2024-05-24 Thread via GitHub
aajisaka commented on PR #8370: URL: https://github.com/apache/iceberg/pull/8370#issuecomment-2129806008 I think this patch is ready to merge: - Built Iceberg and ran a Spark job using Glue 4.0. I confirmed the Spark job successfully created an Iceberg table under the S3 prefix which

Re: [PR] Core: Throw CommitStateUnknownException if RuntimeException that is not marked as cleanable is thrown [iceberg]

2024-05-24 Thread via GitHub
amogh-jahagirdar commented on code in PR #10373: URL: https://github.com/apache/iceberg/pull/10373#discussion_r1613637717 ## core/src/main/java/org/apache/iceberg/BaseTransaction.java: ## @@ -436,6 +440,8 @@ private void commitSimpleTransaction() { } catch (RuntimeException

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-05-24 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1613636902 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/committer/SinkCommittable.java: ## @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Core: Throw CommitStateUnknownException if RuntimeException that is not marked as cleanable is thrown [iceberg]

2024-05-24 Thread via GitHub
amogh-jahagirdar commented on code in PR #10373: URL: https://github.com/apache/iceberg/pull/10373#discussion_r1613629622 ## core/src/main/java/org/apache/iceberg/SnapshotProducer.java: ## @@ -415,6 +415,8 @@ public void commit() { } catch (RuntimeException e) { if (

Re: [PR] Core: Throw CommitStateUnknownException if RuntimeException that is not marked as cleanable is thrown [iceberg]

2024-05-24 Thread via GitHub
amogh-jahagirdar commented on code in PR #10373: URL: https://github.com/apache/iceberg/pull/10373#discussion_r1613626590 ## core/src/main/java/org/apache/iceberg/BaseTransaction.java: ## @@ -436,6 +440,8 @@ private void commitSimpleTransaction() { } catch (RuntimeException

[PR] Core: Throw CommitStateUnknownException if RuntimeException that is not marked as cleanable is thrown [iceberg]

2024-05-24 Thread via GitHub
amogh-jahagirdar opened a new pull request, #10373: URL: https://github.com/apache/iceberg/pull/10373 Upstream callers of the transaction/snapshot producer APIs such as engines like Spark currently handle CommitStateUnknown exceptions to avoid cleaning up data files when it's unclear if the

Re: [I] Run RevAPI without Gradle [iceberg]

2024-05-24 Thread via GitHub
jbonofre commented on issue #10368: URL: https://github.com/apache/iceberg/issues/10368#issuecomment-2129504134 I created a script to run revapi: https://github.com/jbonofre/iceberg/blob/GRADLE/dev/revapi I have three things to address: 1. support `acceptedBreaks` from `.palantir/r

Re: [PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]

2024-05-24 Thread via GitHub
manuzhang commented on code in PR #10370: URL: https://github.com/apache/iceberg/pull/10370#discussion_r1613284723 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java: ## @@ -124,7 +124,7 @@ public void testBoundedWithSavepoint() thr

Re: [PR] Flink: Maintenance - MonitorSource [iceberg]

2024-05-24 Thread via GitHub
gyfora commented on code in PR #10308: URL: https://github.com/apache/iceberg/pull/10308#discussion_r1613226726 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/SingleThreadedIteratorSource.java: ## @@ -0,0 +1,196 @@ +/* + * Licensed to the Apache

[PR] Implement Kerberos authentication support for Hive Catalog [iceberg-python]

2024-05-24 Thread via GitHub
yothinix opened a new pull request, #766: URL: https://github.com/apache/iceberg-python/pull/766 ## Change proposed - Introduce support for HMS behind kerberize environment, to enable this feature we need to add catalog config `hive.use-kerberos=true` Related to #135 -- This is

Re: [PR] feat: make file scan task serializable [iceberg-rust]

2024-05-24 Thread via GitHub
liurenjie1024 commented on PR #377: URL: https://github.com/apache/iceberg-rust/pull/377#issuecomment-2129114452 I think the idea looks good to me, but I'm not sure if we should modify `FileScanTask` now. Should we postpone this until we start to integrate with a true engine such as datafus

Re: [I] Empty snapshot ID should be `Null` instead of `-1` [iceberg-rust]

2024-05-24 Thread via GitHub
liurenjie1024 commented on issue #352: URL: https://github.com/apache/iceberg-rust/issues/352#issuecomment-2129102884 I took a look the code and found that maybe we don't need to change anything since we already handled this case: https://github.com/apache/iceberg-rust/blob/6f8545618dbc666b

Re: [PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]

2024-05-24 Thread via GitHub
pvary commented on code in PR #10370: URL: https://github.com/apache/iceberg/pull/10370#discussion_r1613105037 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java: ## @@ -124,7 +124,7 @@ public void testBoundedWithSavepoint() throws

Re: [I] Introduce a parameter to control whether the flink writer is linked with the previous operator [iceberg]

2024-05-24 Thread via GitHub
huyuanfeng2018 commented on issue #10371: URL: https://github.com/apache/iceberg/issues/10371#issuecomment-2128931501 > Could you use one of the ways described [here](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/overview/#task-chaining-and-resource-group

Re: [PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]

2024-05-24 Thread via GitHub
manuzhang commented on code in PR #10370: URL: https://github.com/apache/iceberg/pull/10370#discussion_r1612975023 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java: ## @@ -124,7 +124,7 @@ public void testBoundedWithSavepoint() thr

Re: [PR] Core: rewrite should drop delete files by data sequence number partition wise [iceberg]

2024-05-24 Thread via GitHub
zinking commented on PR #9454: URL: https://github.com/apache/iceberg/pull/9454#issuecomment-2128767668 As is discussed, this approach burdens the write path with all partition checks, the preferred approach is using an async process. -- This is an automated message from the Apache Git Se