Re: [PR] Add detailed debug and warn logging to SparkMicroBatchStream [iceberg]

2025-05-23 Thread via GitHub
bk-mz commented on PR #12856: URL: https://github.com/apache/iceberg/pull/12856#issuecomment-2906515915 @singhpk234 Hello again. Do you think it's worth merging this PR? Anything you'd like to change maybe? -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] fix: add metadata_properties to _construct_parameters when update hive table [iceberg-python]

2025-05-23 Thread via GitHub
kadai0308 commented on PR #2013: URL: https://github.com/apache/iceberg-python/pull/2013#issuecomment-2906508511 Start working on it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[PR] Update hive storage descriptor after commit schema change [iceberg-python]

2025-05-23 Thread via GitHub
frankliee opened a new pull request, #2036: URL: https://github.com/apache/iceberg-python/pull/2036 # Rationale for this change # Are these changes tested? # Are there any user-facing changes? -- This is an automated message from the Apache Git

Re: [PR] Hive: Update hive storage descriptor after commit schema change [iceberg-python]

2025-05-23 Thread via GitHub
frankliee closed pull request #2035: Hive: Update hive storage descriptor after commit schema change URL: https://github.com/apache/iceberg-python/pull/2035 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[PR] Hive: Update hive storage descriptor after commit schema change [iceberg-python]

2025-05-23 Thread via GitHub
frankliee opened a new pull request, #2035: URL: https://github.com/apache/iceberg-python/pull/2035 # Rationale for this change Like Java iceberg, we should also change hive's storage descriptor after commit. See more details in https://github.com/apache/ic

Re: [PR] REST: Avoid table corruption on unhandled RestException [iceberg]

2025-05-23 Thread via GitHub
singhpk234 commented on code in PR #12818: URL: https://github.com/apache/iceberg/pull/12818#discussion_r2105697905 ## core/src/main/java/org/apache/iceberg/rest/ErrorHandlers.java: ## @@ -91,7 +91,19 @@ public void accept(ErrorResponse error) { case 404: thr

Re: [PR] Change `FieldSummary` `{upper,lower}_bound` to `ByteBuf` [iceberg-rust]

2025-05-23 Thread via GitHub
kevinjqliu commented on code in PR #1369: URL: https://github.com/apache/iceberg-rust/pull/1369#discussion_r2105691302 ## crates/iceberg/src/spec/manifest_list.rs: ## @@ -844,83 +812,10 @@ pub(super) mod _serde { pub key_metadata: Option, } -#[derive(Debug, S

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
Guosmilesmile commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2105683593 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/CommittableToTableChangeConverter.java: ## @@ -0,0 +1,181 @@ +/* + * Licensed to the Apache Sof

Re: [PR] Spark 3.5: Update to Spark 3.5.6 [iceberg]

2025-05-23 Thread via GitHub
drexler-sky commented on code in PR #13142: URL: https://github.com/apache/iceberg/pull/13142#discussion_r2105685951 ## build.gradle: ## @@ -120,6 +120,9 @@ allprojects { repositories { mavenCentral() mavenLocal() +maven { + url "https://repository.apache.

[PR] Spark 3.5: Update to Spark 3.5.6 [iceberg]

2025-05-23 Thread via GitHub
drexler-sky opened a new pull request, #13142: URL: https://github.com/apache/iceberg/pull/13142 Update to Spark 3.5.6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [I] OpenAPI spec missing `schema` field for `TableMetadata` [iceberg]

2025-05-23 Thread via GitHub
Bhargavkonidena commented on issue #13103: URL: https://github.com/apache/iceberg/issues/13103#issuecomment-2906346722 @nastra I am interested to contribute for this. Could you please assign this to me. -- This is an automated message from the Apache Git Service. To respond to the mess

Re: [PR] Fix Issue #13064 [iceberg]

2025-05-23 Thread via GitHub
Bhargavkonidena commented on PR #13113: URL: https://github.com/apache/iceberg/pull/13113#issuecomment-2906345603 Hi @manuzhang , Please check screenshots after my changes(rearrangement of time travel sections) https://github.com/user-attachments/assets/be45f6e3-fb07-49e2-9db9-4a6ce8d

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
Guosmilesmile commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2105683593 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/CommittableToTableChangeConverter.java: ## @@ -0,0 +1,181 @@ +/* + * Licensed to the Apache Sof

Re: [I] REST Catalog fixture is particular about query params [iceberg]

2025-05-23 Thread via GitHub
kevinjqliu commented on issue #13119: URL: https://github.com/apache/iceberg/issues/13119#issuecomment-2906306952 I feel like I'm not understanding the improvement here, but feel free to open a PR to clarify the spec! These are the options to consider: ``` http://localhost:818

Re: [PR] Core: Make pageToken query parameter optional [iceberg]

2025-05-23 Thread via GitHub
kevinjqliu commented on code in PR #13129: URL: https://github.com/apache/iceberg/pull/13129#discussion_r2105636264 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -565,13 +568,16 @@ public List listNamespaces(SessionContext context, Namespace names

Re: [I] REST Catalog fixture is particular about query params [iceberg]

2025-05-23 Thread via GitHub
elphastori commented on issue #13119: URL: https://github.com/apache/iceberg/issues/13119#issuecomment-2906092762 @kevinjqliu Thanks for reviewing the PR! We clearly agree that omitting `pageToken` when `pageSize` is specified should work, and the PR will fix that. Although the spec s

Re: [PR] API: Remove deprecated `apply()` [iceberg]

2025-05-23 Thread via GitHub
github-actions[bot] commented on PR #11691: URL: https://github.com/apache/iceberg/pull/11691#issuecomment-2906077053 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Add detailed debug and warn logging to SparkMicroBatchStream [iceberg]

2025-05-23 Thread via GitHub
github-actions[bot] commented on PR #12856: URL: https://github.com/apache/iceberg/pull/12856#issuecomment-2906077159 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] The source artifacts provided with GitHub releases are not buildable. [iceberg]

2025-05-23 Thread via GitHub
github-actions[bot] closed issue #11505: The source artifacts provided with GitHub releases are not buildable. URL: https://github.com/apache/iceberg/issues/11505 -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [I] The source artifacts provided with GitHub releases are not buildable. [iceberg]

2025-05-23 Thread via GitHub
github-actions[bot] commented on issue #11505: URL: https://github.com/apache/iceberg/issues/11505#issuecomment-2906076972 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [PR] Arrow, AWS, Azure, Core, GCP, Hive, Kafka, Snowflake: Rename test classes to use Test as prefix instead of suffix [iceberg]

2025-05-23 Thread via GitHub
github-actions[bot] commented on PR #12879: URL: https://github.com/apache/iceberg/pull/12879#issuecomment-2906077177 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Spark. Fix Procedure Description. [iceberg]

2025-05-23 Thread via GitHub
github-actions[bot] commented on PR #12816: URL: https://github.com/apache/iceberg/pull/12816#issuecomment-2906077117 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Core: Make pageToken query parameter optional [iceberg]

2025-05-23 Thread via GitHub
elphastori commented on code in PR #13129: URL: https://github.com/apache/iceberg/pull/13129#discussion_r2105535540 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -565,13 +568,16 @@ public List listNamespaces(SessionContext context, Namespace names

Re: [PR] Core: Make pageToken query parameter optional [iceberg]

2025-05-23 Thread via GitHub
elphastori commented on code in PR #13129: URL: https://github.com/apache/iceberg/pull/13129#discussion_r2105516701 ## core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java: ## @@ -111,7 +111,8 @@ public CommitFailedException wrapped() { } private static Pair,

Re: [PR] Spark4.0: RC -> official release [iceberg]

2025-05-23 Thread via GitHub
szehon-ho commented on PR #13121: URL: https://github.com/apache/iceberg/pull/13121#issuecomment-2905941897 @huaxingao awesome accomplishment to finally have Spark 4.0 support ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[I] Manifest list written by go-iceberg contains incorrect file lengths [iceberg-go]

2025-05-23 Thread via GitHub
alexjo2144 opened a new issue, #438: URL: https://github.com/apache/iceberg-go/issues/438 ### Apache Iceberg version None ### Please describe the bug 🐞 Hey iceberg-go team, @alexandre-normand and I have been trying out writes to an Iceberg table from Go and have hit a sn

Re: [PR] Spark4.0: RC -> official release [iceberg]

2025-05-23 Thread via GitHub
huaxingao commented on PR #13121: URL: https://github.com/apache/iceberg/pull/13121#issuecomment-2905922830 Thanks @RussellSpitzer @szehon-ho @dongjoon-hyun @nastra -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Spark4.0: RC -> official release [iceberg]

2025-05-23 Thread via GitHub
RussellSpitzer commented on PR #13121: URL: https://github.com/apache/iceberg/pull/13121#issuecomment-2905918797 Thanks @huaxingao ! Thanks @szehon-ho and @dongjoon-hyun for reviewing! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Spark4.0: RC -> official release [iceberg]

2025-05-23 Thread via GitHub
RussellSpitzer merged PR #13121: URL: https://github.com/apache/iceberg/pull/13121 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Core: Interface based DataFile reader and writer API [iceberg]

2025-05-23 Thread via GitHub
stevenzwu commented on code in PR #12298: URL: https://github.com/apache/iceberg/pull/12298#discussion_r2105467425 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkObjectModels.java: ## @@ -0,0 +1,80 @@ +/* + * Licensed to the Apache Software Foundation (AS

Re: [PR] Core: Add basic classes for writing table format-version 4 [iceberg]

2025-05-23 Thread via GitHub
RussellSpitzer commented on code in PR #13123: URL: https://github.com/apache/iceberg/pull/13123#discussion_r2105451519 ## api/src/test/java/org/apache/iceberg/TestHelpers.java: ## @@ -54,7 +54,7 @@ public class TestHelpers { private TestHelpers() {} - public static fina

Re: [PR] feat(catalog/sql): Add views related ops [iceberg-go]

2025-05-23 Thread via GitHub
zeroshade commented on code in PR #414: URL: https://github.com/apache/iceberg-go/pull/414#discussion_r2105433224 ## catalog/sql/sql_test.go: ## @@ -299,8 +303,12 @@ func (s *SqliteCatalogTestSuite) TestCreationAllTablesExist() { "catalog_name" VARCHAR NOT NULL,

Re: [PR] feat(catalog/sql): Add views related ops [iceberg-go]

2025-05-23 Thread via GitHub
zeroshade commented on code in PR #414: URL: https://github.com/apache/iceberg-go/pull/414#discussion_r2105429667 ## catalog/sql/sql.go: ## @@ -298,6 +302,7 @@ func (c *Catalog) CreateTable(ctx context.Context, ident table.Identifier, sc *i TableNamespac

[I] Relative Path Support In Table Spec [iceberg]

2025-05-23 Thread via GitHub
talatuyarer opened a new issue, #13141: URL: https://github.com/apache/iceberg/issues/13141 ### Proposed Change Currently, Iceberg uses absolute URIs for file paths, requiring metadata rewrites when a table is moved. We want to introduce Relative path which is to achieve "Zero

Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

2025-05-23 Thread via GitHub
jbonofre commented on PR #12897: URL: https://github.com/apache/iceberg/pull/12897#issuecomment-2905590955 Finally back from several trips, so resuming work on this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

2025-05-23 Thread via GitHub
jbonofre commented on code in PR #12897: URL: https://github.com/apache/iceberg/pull/12897#discussion_r2105296456 ## api/src/main/java/org/apache/iceberg/UnboundPartitionSpec.java: ## @@ -88,14 +89,23 @@ Builder withSpecId(int newSpecId) { return this; } +Build

Re: [PR] Core: Implement source-ids to deal with multi arguments transforms [iceberg]

2025-05-23 Thread via GitHub
jbonofre commented on code in PR #12897: URL: https://github.com/apache/iceberg/pull/12897#discussion_r2105295782 ## api/src/main/java/org/apache/iceberg/PartitionSpec.java: ## @@ -602,17 +602,26 @@ public Builder alwaysNull(String sourceName) { // add a partition field w

Re: [PR] Spark: support rewrite on specified target branch [iceberg]

2025-05-23 Thread via GitHub
lliangyu-lin commented on PR #12257: URL: https://github.com/apache/iceberg/pull/12257#issuecomment-2905424859 Hello @amitgilad3, are you still actively working on this PR? If not, could I help on rebasing the refactored change based on your current changes to get this in? -- This is an

Re: [PR] feat: Introduce snapshot summary properties [iceberg-rust]

2025-05-23 Thread via GitHub
dentiny commented on PR #1336: URL: https://github.com/apache/iceberg-rust/pull/1336#issuecomment-2905417703 Hi @Xuanwo , I think the linting issue has been fixed, also merged with main branch; could you please take another look when you have some time? Thank you! -- This is an automated

Re: [I] How to get latest partitions values for specific timestamp range [iceberg-python]

2025-05-23 Thread via GitHub
mark-major commented on issue #1988: URL: https://github.com/apache/iceberg-python/issues/1988#issuecomment-2905406034 Hi @dvnageshpatil ! I think for that you will need to read the manifest files and get the information you need from them. This is something that probably won't be direc

Re: [PR] Core: Make pageToken query parameter optional [iceberg]

2025-05-23 Thread via GitHub
kevinjqliu commented on code in PR #13129: URL: https://github.com/apache/iceberg/pull/13129#discussion_r2105182161 ## core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java: ## @@ -111,7 +111,8 @@ public CommitFailedException wrapped() { } private static Pair,

Re: [I] REST Catalog fixture is particular about query params [iceberg]

2025-05-23 Thread via GitHub
kevinjqliu commented on issue #13119: URL: https://github.com/apache/iceberg/issues/13119#issuecomment-2905336288 > The API expects the first request to have an empty pageToken, rather than omitting the parameter entirely. While this aligns with the current spec, it’s not the most intuitive

Re: [PR] Spark4.0: RC -> official release [iceberg]

2025-05-23 Thread via GitHub
huaxingao commented on PR #13121: URL: https://github.com/apache/iceberg/pull/13121#issuecomment-2905327869 cc @amogh-jahagirdar @nastra @RussellSpitzer @szehon-ho I have switched to the official released Spark4.0. -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Spec: Add details on GZIP compressed metadata files [iceberg]

2025-05-23 Thread via GitHub
pan3793 commented on code in PR #12598: URL: https://github.com/apache/iceberg/pull/12598#discussion_r2105074290 ## format/spec.md: ## @@ -1761,6 +1763,10 @@ The reference Java implementation uses a type 4 uuid and XORs the 4 most signifi Java writes `-1` for "no current sna

Re: [PR] Flink: Dynamic Iceberg Sink: Add dynamic writer and committer [iceberg]

2025-05-23 Thread via GitHub
mxm commented on PR #13080: URL: https://github.com/apache/iceberg/pull/13080#issuecomment-2905119534 Thanks for the review @pvary. I'll some more tests for the writer / aggregator / committer. There is more integration / e2e testing coming with the subsequent PRs, but we should also test t

Re: [PR] Flink: Backport fix npe in TaskResultAggregator when job recovery to Flink 1.19 and 1.20 [iceberg]

2025-05-23 Thread via GitHub
Guosmilesmile commented on PR #13140: URL: https://github.com/apache/iceberg/pull/13140#issuecomment-2904903617 @pvary Hey Peter. It is a clearn backport. If you have time, take a look and help to review it. Thanks! -- This is an automated message from the Apache Git Service. To re

Re: [I] REST Catalog fixture is particular about query params [iceberg]

2025-05-23 Thread via GitHub
elphastori commented on issue #13119: URL: https://github.com/apache/iceberg/issues/13119#issuecomment-2904808219 @nastra I've opened the PR ([#13129](https://github.com/apache/iceberg/pull/13129)) with the fix/improvement. Could you please take a look? -- This is an automated message f

Re: [PR] Feature: Write to branches [iceberg-python]

2025-05-23 Thread via GitHub
vinjai commented on PR #941: URL: https://github.com/apache/iceberg-python/pull/941#issuecomment-2904799400 Thanks @SebastienN15 — I’ll review your commit and take it forward from there. I’ll share the working PR tomorrow. -- This is an automated message from the Apache Git Service. T

Re: [I] REST Catalog fixture is particular about query params [iceberg]

2025-05-23 Thread via GitHub
elphastori commented on issue #13119: URL: https://github.com/apache/iceberg/issues/13119#issuecomment-2904798383 Thanks for linking the [rest spec](https://github.com/apache/iceberg/blob/91dff9886e6e6c494f6a970129f4c08487c98a0d/open-api/rest-catalog-open-api.yaml#L2018C1-L2041C21). >

Re: [PR] Core, OpenAPI: Add commit timestamp to CommitReport [iceberg]

2025-05-23 Thread via GitHub
manuzhang commented on PR #12990: URL: https://github.com/apache/iceberg/pull/12990#issuecomment-2904698406 Closed as per discussion in https://lists.apache.org/thread/1jyqtf5zt67hhzksxrrlcrz6htblf7d9 -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] AWS: Close the S3SeekableInputStreamFactory before removing from cache [iceberg]

2025-05-23 Thread via GitHub
SanjayMarreddi commented on code in PR #12891: URL: https://github.com/apache/iceberg/pull/12891#discussion_r2104777316 ## aws/src/main/java/org/apache/iceberg/aws/s3/AnalyticsAcceleratorUtil.java: ## @@ -83,6 +95,16 @@ private static S3SeekableInputStreamFactory createNewFactor

Re: [PR] Flink: Backport fix npe in TaskResultAggregator when job recovery to Flink 1.19 and 1.20 [iceberg]

2025-05-23 Thread via GitHub
Guosmilesmile closed pull request #13140: Flink: Backport fix npe in TaskResultAggregator when job recovery to Flink 1.19 and 1.20 URL: https://github.com/apache/iceberg/pull/13140 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Core, OpenAPI: Add commit timestamp to CommitReport [iceberg]

2025-05-23 Thread via GitHub
manuzhang closed pull request #12990: Core, OpenAPI: Add commit timestamp to CommitReport URL: https://github.com/apache/iceberg/pull/12990 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[PR] Flink: Backport fix npe in TaskResultAggregator when job recovery to Flink 1.19 and 1.20 [iceberg]

2025-05-23 Thread via GitHub
Guosmilesmile opened a new pull request, #13140: URL: https://github.com/apache/iceberg/pull/13140 This is a backport to Flink 1.19 and 1.20 https://github.com/apache/iceberg/pull/13086 -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] [WIP] Spark4.0: RC -> official release [iceberg]

2025-05-23 Thread via GitHub
dongjoon-hyun commented on PR #13121: URL: https://github.com/apache/iceberg/pull/13121#issuecomment-2904574268 Ya, 4.0.0 is now uploaded to the Maven Central. Please update this PR, @huaxingao ~ -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [PR] AWS: Configure Default s3Async credentials the same as s3 [iceberg]

2025-05-23 Thread via GitHub
devinrsmith commented on PR #13132: URL: https://github.com/apache/iceberg/pull/13132#issuecomment-2904530671 @SanjayMarreddi, I think should look at this since you were responsible for https://github.com/apache/iceberg/pull/12503? -- This is an automated message from the Apache Git Servi

Re: [PR] Fix Issue #13064 [iceberg]

2025-05-23 Thread via GitHub
manuzhang commented on PR #13113: URL: https://github.com/apache/iceberg/pull/13113#issuecomment-2904523353 @Bhargavkonidena Can you share a snapshot of the page after your changes? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] Spark: Custom snapshot property from session configuration [iceberg]

2025-05-23 Thread via GitHub
cccs-jory commented on code in PR #12999: URL: https://github.com/apache/iceberg/pull/12999#discussion_r2104652942 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkWriteConf.java: ## @@ -252,6 +252,15 @@ public long targetDeleteFileSize() { public Map extraSnap

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
pvary commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2104644217 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/CommittableToTableChangeConverter.java: ## @@ -0,0 +1,181 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] Flink: Fix npe in TaskResultAggregator when job recovery [iceberg]

2025-05-23 Thread via GitHub
pvary merged PR #13086: URL: https://github.com/apache/iceberg/pull/13086 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Flink: Fix npe in TaskResultAggregator when job recovery [iceberg]

2025-05-23 Thread via GitHub
pvary commented on PR #13086: URL: https://github.com/apache/iceberg/pull/13086#issuecomment-2904473887 Merged to main. Thanks @Guosmilesmile for the PR and @mxm for the review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

[PR] Remove obsolete version attribute to avoid confusion [iceberg]

2025-05-23 Thread via GitHub
omkenge opened a new pull request, #13139: URL: https://github.com/apache/iceberg/pull/13139 The version attribute is no longer in use and is ignored during processing. This PR removes the attribute to eliminate unnecessary clutter and prevent potential confusion for future contributors and

Re: [PR] fix: check leaf column is root column in Parquet schema [iceberg-rust]

2025-05-23 Thread via GitHub
Xuanwo merged PR #1347: URL: https://github.com/apache/iceberg-rust/pull/1347 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] feat: Introduce snapshot summary properties [iceberg-rust]

2025-05-23 Thread via GitHub
Xuanwo commented on PR #1336: URL: https://github.com/apache/iceberg-rust/pull/1336#issuecomment-2904433770 One thing left here is to format the code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] demo: an easy to use catalog loader [iceberg-rust]

2025-05-23 Thread via GitHub
Xuanwo commented on PR #1372: URL: https://github.com/apache/iceberg-rust/pull/1372#issuecomment-2904415943 In pyiceberg, we have: ```python class RestCatalog(Catalog): uri: str _session: Session def __init__(self, name: str, **properties: str): "

[PR] demo: an easy to use catalog loader [iceberg-rust]

2025-05-23 Thread via GitHub
Xuanwo opened a new pull request, #1372: URL: https://github.com/apache/iceberg-rust/pull/1372 ## Which issue does this PR close? Part of https://github.com/apache/iceberg-rust/issues/1254 ## What changes are included in this PR? This PR introduces a catalog loader trait

Re: [PR] Hive: Throw exception for when listing a non-existing namespace [iceberg]

2025-05-23 Thread via GitHub
nastra commented on code in PR #13130: URL: https://github.com/apache/iceberg/pull/13130#discussion_r2104562932 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -1212,7 +1212,6 @@ public void testDatabaseLocationWithSlashInWarehouseDir() {

Re: [PR] AWS: Close the S3SeekableInputStreamFactory before removing from cache [iceberg]

2025-05-23 Thread via GitHub
nastra commented on code in PR #12891: URL: https://github.com/apache/iceberg/pull/12891#discussion_r2104540081 ## aws/src/main/java/org/apache/iceberg/aws/s3/AnalyticsAcceleratorUtil.java: ## @@ -83,6 +95,16 @@ private static S3SeekableInputStreamFactory createNewFactory(

Re: [I] AWS: Closing S3FileIO can result in ClassNotFoundException [iceberg]

2025-05-23 Thread via GitHub
nastra closed issue #13133: AWS: Closing S3FileIO can result in ClassNotFoundException URL: https://github.com/apache/iceberg/issues/13133 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] AWS: Skip cleanup of analytics accelerator when disabled [iceberg]

2025-05-23 Thread via GitHub
nastra merged PR #13134: URL: https://github.com/apache/iceberg/pull/13134 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Docs: Update spec version for write modes [iceberg]

2025-05-23 Thread via GitHub
nastra merged PR #13138: URL: https://github.com/apache/iceberg/pull/13138 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

[PR] Docs: Update spec version for write modes [iceberg]

2025-05-23 Thread via GitHub
ajantha-bhat opened a new pull request, #13138: URL: https://github.com/apache/iceberg/pull/13138 write modes are supported for v2 and above. Not just v2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] AWS: KeyManagementClient implementation that works with AWS KMS [iceberg]

2025-05-23 Thread via GitHub
szlta commented on PR #13136: URL: https://github.com/apache/iceberg/pull/13136#issuecomment-2904218886 cc: @amogh-jahagirdar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
pvary commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2104171688 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/TableMaintenance.java: ## @@ -59,6 +59,12 @@ public class TableMaintenance { static final

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
Guosmilesmile commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2104404729 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/CommittableToTableChangeConverter.java: ## @@ -0,0 +1,181 @@ +/* + * Licensed to the Apache Sof

[PR] API: Follow up on adding Variant data type to implement sanitizing for Variant #11479 [iceberg]

2025-05-23 Thread via GitHub
manirajv06 opened a new pull request, #13137: URL: https://github.com/apache/iceberg/pull/13137 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] API: Follow up on adding Variant data type to implement sanitizing fo… [iceberg]

2025-05-23 Thread via GitHub
manirajv06 commented on PR #12611: URL: https://github.com/apache/iceberg/pull/12611#issuecomment-2904122091 @aihuaxu @RussellSpitzer Sorry for the delay. Have changes ready sitting locally, need to push forward. Can you reopen this? -- This is an automated message from the Apache Gi

Re: [PR] AWS: Skip cleanup of analytics accelerator when disabled [iceberg]

2025-05-23 Thread via GitHub
SanjayMarreddi commented on PR #13134: URL: https://github.com/apache/iceberg/pull/13134#issuecomment-2904105656 Also want to highlight the relevant clean up PR similar to this change here: https://github.com/apache/iceberg/pull/12891 -- This is an automated message from the Apache Git Se

[PR] AWS: KeyManagementClient implementation that works with AWS KMS [iceberg]

2025-05-23 Thread via GitHub
szlta opened a new pull request, #13136: URL: https://github.com/apache/iceberg/pull/13136 To be used in table encryption for: - wrapping/unwrapping encryption keys - generating data keys (available specs: AES_256, AES_128) Added integration test for verification. -- This is a

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
Guosmilesmile commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2104315758 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TaskResultAggregator.java: ## @@ -61,7 +61,7 @@ public class TaskResultAggregat

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
Guosmilesmile commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2104313253 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/CommittableToTableChangeConverter.java: ## @@ -0,0 +1,181 @@ +/* + * Licensed to the Apache Sof

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
Guosmilesmile commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2104295443 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TaskResultAggregator.java: ## @@ -61,7 +61,7 @@ public class TaskResultAggregat

Re: [I] Make FileIO a Trait [iceberg-rust]

2025-05-23 Thread via GitHub
roeap commented on issue #1314: URL: https://github.com/apache/iceberg-rust/issues/1314#issuecomment-2903969557 Just sharing some experiences from the delta world which may not immediately applicable to the question around which trait to use, but maybe be food for thought as to where things

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
pvary commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2104208183 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java: ## @@ -216,7 +235,44 @@ public SimpleVersionedSerializer getCommittableSerializer()

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
pvary commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2104223659 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/TestFlinkTableSinkCompaction.java: ## @@ -0,0 +1,141 @@ +/* + * Licensed to the Apache Software Foundation (

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
pvary commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2104226130 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestZkLockFactoryBuilder.java: ## @@ -0,0 +1,77 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
pvary commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2104221208 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/TestFlinkTableSinkCompaction.java: ## @@ -0,0 +1,141 @@ +/* + * Licensed to the Apache Software Foundation (

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
pvary commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2104219442 ## flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/TestFlinkTableSinkCompaction.java: ## @@ -0,0 +1,141 @@ +/* + * Licensed to the Apache Software Foundation (

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
pvary commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2104217262 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java: ## @@ -641,6 +716,17 @@ private static Map writeProperties( return writeProperti

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
pvary commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2104210075 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java: ## @@ -530,7 +586,26 @@ IcebergSink build() { flinkWriteConf, e

Re: [PR] Docs: add Tinybird to the list of vendors and blog posts [iceberg]

2025-05-23 Thread via GitHub
nastra commented on code in PR #13128: URL: https://github.com/apache/iceberg/pull/13128#discussion_r2104207705 ## site/docs/vendors.md: ## @@ -1,6 +1,7 @@ --- title: "Vendors" --- + Review Comment: these seem all like unrelated changes -- This is an automated message

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
pvary commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2104206088 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java: ## @@ -216,7 +235,44 @@ public SimpleVersionedSerializer getCommittableSerializer()

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
pvary commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2104202165 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/CommittableToTableChangeConverter.java: ## @@ -0,0 +1,181 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] Flink: Remove the MiniClusterWithClientResource dependency [iceberg]

2025-05-23 Thread via GitHub
JeonDaehong commented on PR #13021: URL: https://github.com/apache/iceberg/pull/13021#issuecomment-2903837698 > Merged to main. Thanks @JeonDaehong for the PR and @nastra, @tomtongue, @Guosmilesmile and @mxm for the review! > > @JeonDaehong please proceed with the backport Than

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
pvary commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2104195636 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/CommittableToTableChangeConverter.java: ## @@ -0,0 +1,181 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
pvary commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2104188531 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/CommittableToTableChangeConverter.java: ## @@ -0,0 +1,181 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
pvary commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2104174969 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TaskResultAggregator.java: ## @@ -61,7 +61,7 @@ public class TaskResultAggregator exten

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
pvary commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2104173086 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/TableMaintenanceConfig.java: ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
pvary commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2104170427 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/RewriteDataFilesConfig.java: ## @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-23 Thread via GitHub
pvary commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2104185618 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/CommittableToTableChangeConverter.java: ## @@ -0,0 +1,181 @@ +/* + * Licensed to the Apache Software Fo

  1   2   >