Re: [PR] Data: Add partition stats writer and reader [iceberg]

ajantha-bhat commented on code in PR #11216: URL: https://github.com/apache/iceberg/pull/11216#discussion_r1998023614 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java: ## @@ -0,0 +1,284 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[I] Issue with IsIn Predicate Formatting in Iceberg-Go [iceberg-go]

rameshkanna3 opened a new issue, #335: URL: https://github.com/apache/iceberg-go/issues/335 ### Apache Iceberg version None ### Please describe the bug 🐞 When using the `IsIn` filter in **Iceberg-Go**, filtering on a **single integer value works correctly**, but filterin

Re: [PR] Data: Add partition stats writer and reader [iceberg]

ajantha-bhat commented on code in PR #11216: URL: https://github.com/apache/iceberg/pull/11216#discussion_r1998031053 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java: ## @@ -0,0 +1,284 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Data: Add partition stats writer and reader [iceberg]

pvary commented on code in PR #11216: URL: https://github.com/apache/iceberg/pull/11216#discussion_r1998041969 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java: ## @@ -0,0 +1,284 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [I] catalog table-default and table-override properties are not supported in CREATE_OR_REPLACE operation in IRC [iceberg]

puchengy commented on issue #12506: URL: https://github.com/apache/iceberg/issues/12506#issuecomment-2728358350 @nastra yes, I am happy to. I am using https://github.com/apache/gravitino as my IRC implementation and it uses HiveCatalog as its IRC backend catalog. In the setting, I sim

Re: [I] catalog table-default and table-override properties are not supported in CREATE_OR_REPLACE operation in IRC [iceberg]

nastra commented on issue #12506: URL: https://github.com/apache/iceberg/issues/12506#issuecomment-2728352322 @puchengy can you add some details on how you're setting them on the server side? -- This is an automated message from the Apache Git Service. To respond to the message, please lo

[I] Not able to create Pyiceberg table with Partition Spec using pyarrow schema [iceberg-python]

heman026 opened a new issue, #1797: URL: https://github.com/apache/iceberg-python/issues/1797 ### Question Hi I am reading an existing iceberg table using batch reader. The reader has pyarrow schema. When creating Table using this pyarrow schema, the fields have Field_id = -1 whi

[PR] feat: include spec id in DataFile [iceberg-rust]

ZENOTME opened a new pull request, #1098: URL: https://github.com/apache/iceberg-rust/pull/1098 ## Which issue does this PR close? - Closes #. ## What changes are included in this PR? This PR includes spec id in DataFile which will be used in the future, s

Re: [PR] feat(manifests): Consolidate V1/V2 manifest file objects [iceberg-go]

kevinjqliu commented on code in PR #332: URL: https://github.com/apache/iceberg-go/pull/332#discussion_r1997684054 ## manifest.go: ## @@ -146,25 +158,66 @@ type fallbackManifestFileV1 struct { AddedSnapshotID *int64 `avro:"added_snapshot_id"` } -func (f *fallbackManif

Re: [PR] Added `FsspecFileIO` method for OSS, virtual hosted style default to true, standardized key configurations for OSS [iceberg-python]

Fokko commented on code in PR #1788: URL: https://github.com/apache/iceberg-python/pull/1788#discussion_r1997988907 ## pyiceberg/io/fsspec.py: ## @@ -124,6 +128,22 @@ def _file(_: Properties) -> LocalFileSystem: return LocalFileSystem(auto_mkdir=True) +def _oss(properti

[PR] Handle pagination via `next-page-token` in REST Catalog [iceberg-rust]

phillipleblanc opened a new pull request, #1097: URL: https://github.com/apache/iceberg-rust/pull/1097 ## Which issue does this PR close? - Closes #1096 ## What changes are included in this PR? Implements pagination on the `GET /v1/namespaces` and `GET /v1/namespaces/my_

[I] Error failed to get table info from metastore When Using Kafka Connect Iceberg with Hive SSL [iceberg]

tranhan02 opened a new issue, #12547: URL: https://github.com/apache/iceberg/issues/12547 I am using kafka-connect-iceberg to connect to an Iceberg catalog backed by Hive Metastore (HMS) with SSL enabled. Below is my connector configuration: ``` class: io.tabular.iceberg.connect.Ic

Re: [PR] Spark: Add some tests for variant fixup [iceberg]

XBaith commented on code in PR #12497: URL: https://github.com/apache/iceberg/pull/12497#discussion_r1997955335 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestSparkFixupTypes.java: ## @@ -0,0 +1,163 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

Re: [PR] chore(deps): Bump aws-config from 1.5.16 to 1.5.18 [iceberg-rust]

liurenjie1024 merged PR #1091: URL: https://github.com/apache/iceberg-rust/pull/1091 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Spark: Add some tests for variant fixup [iceberg]

aihuaxu commented on code in PR #12497: URL: https://github.com/apache/iceberg/pull/12497#discussion_r1997882801 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestSparkFixupTypes.java: ## @@ -0,0 +1,163 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

[PR] refactor: Split manifest module [iceberg-rust]

jonathanc-n opened a new pull request, #1095: URL: https://github.com/apache/iceberg-rust/pull/1095 ## Which issue does this PR close? - Closes #1083 . ## What changes are included in this PR? Split manifest module into smaller modules ## Are these changes

Re: [PR] chore(deps): Bump once_cell from 1.20.3 to 1.21.1 [iceberg-rust]

liurenjie1024 merged PR #1089: URL: https://github.com/apache/iceberg-rust/pull/1089 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] chore(deps): Bump uuid from 1.13.2 to 1.16.0 [iceberg-rust]

liurenjie1024 merged PR #1092: URL: https://github.com/apache/iceberg-rust/pull/1092 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] chore(deps): Bump http from 1.2.0 to 1.3.1 [iceberg-rust]

liurenjie1024 merged PR #1090: URL: https://github.com/apache/iceberg-rust/pull/1090 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] chore(deps): Bump tempfile from 3.18.0 to 3.19.0 [iceberg-rust]

liurenjie1024 merged PR #1093: URL: https://github.com/apache/iceberg-rust/pull/1093 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] fix: fix delete files sequence comparison [iceberg-rust]

liurenjie1024 commented on code in PR #1077: URL: https://github.com/apache/iceberg-rust/pull/1077#discussion_r1997841417 ## crates/iceberg/src/delete_file_index.rs: ## @@ -147,21 +147,21 @@ impl PopulatedDeleteFileIndex { self.global_deletes .iter() -

Re: [PR] chore(deps): Bump tokio from 1.43.0 to 1.44.1 [iceberg-rust]

liurenjie1024 merged PR #1094: URL: https://github.com/apache/iceberg-rust/pull/1094 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [I] deletion & purge improvements for undelete feature in REST catalog [iceberg]

github-actions[bot] commented on issue #11023: URL: https://github.com/apache/iceberg/issues/11023#issuecomment-2727740637 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] Core: add variant builder implementation [iceberg]

github-actions[bot] commented on PR #11857: URL: https://github.com/apache/iceberg/pull/11857#issuecomment-2727740712 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] Review new ImmutablesReferenceEquality error-prone check [iceberg]

github-actions[bot] commented on issue #10855: URL: https://github.com/apache/iceberg/issues/10855#issuecomment-2727740607 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] [feature request] Automatically Refreshable AWS credential [iceberg-python]

github-actions[bot] closed issue #1129: [feature request] Automatically Refreshable AWS credential URL: https://github.com/apache/iceberg-python/issues/1129 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] backport #11301(rowconverter) to Flink 1.19 and 1.18 [iceberg]

github-actions[bot] commented on PR #11826: URL: https://github.com/apache/iceberg/pull/11826#issuecomment-2727740690 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] [feature request] Automatically Refreshable AWS credential [iceberg-python]

github-actions[bot] commented on issue #1129: URL: https://github.com/apache/iceberg-python/issues/1129#issuecomment-2727742917 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the A

Re: [I] Iceberg table not able to read data from S3 after few hours using Athena . [iceberg]

github-actions[bot] commented on issue #9684: URL: https://github.com/apache/iceberg/issues/9684#issuecomment-2727740574 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Kafka: runtime integration test failure or flaky [iceberg]

github-actions[bot] commented on issue #11046: URL: https://github.com/apache/iceberg/issues/11046#issuecomment-2727740656 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Table maintenace procedure(expire_snapshots) not work as expceted [iceberg]

github-actions[bot] commented on issue #10907: URL: https://github.com/apache/iceberg/issues/10907#issuecomment-2727740618 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Iceberg table not able to read data from S3 after few hours using Athena . [iceberg]

github-actions[bot] closed issue #9684: Iceberg table not able to read data from S3 after few hours using Athena . URL: https://github.com/apache/iceberg/issues/9684 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] Review new ImmutablesReferenceEquality error-prone check [iceberg]

github-actions[bot] closed issue #10855: Review new ImmutablesReferenceEquality error-prone check URL: https://github.com/apache/iceberg/issues/10855 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Spark: Remove closing of IO in SerializableTable* [iceberg]

github-actions[bot] commented on PR #12129: URL: https://github.com/apache/iceberg/pull/12129#issuecomment-2727740751 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] feat(manifests): Consolidate V1/V2 manifest file objects [iceberg-go]

zeroshade commented on code in PR #332: URL: https://github.com/apache/iceberg-go/pull/332#discussion_r1997687026 ## manifest.go: ## @@ -146,25 +158,66 @@ type fallbackManifestFileV1 struct { AddedSnapshotID *int64 `avro:"added_snapshot_id"` } -func (f *fallbackManife

Re: [PR] feat(manifests): Consolidate V1/V2 manifest file objects [iceberg-go]

kevinjqliu commented on code in PR #332: URL: https://github.com/apache/iceberg-go/pull/332#discussion_r1997687593 ## manifest.go: ## @@ -146,25 +158,66 @@ type fallbackManifestFileV1 struct { AddedSnapshotID *int64 `avro:"added_snapshot_id"` } -func (f *fallbackManif

Re: [PR] feat(manifests): Consolidate V1/V2 manifest file objects [iceberg-go]

kevinjqliu commented on code in PR #332: URL: https://github.com/apache/iceberg-go/pull/332#discussion_r1997687364 ## manifest.go: ## @@ -146,25 +158,66 @@ type fallbackManifestFileV1 struct { AddedSnapshotID *int64 `avro:"added_snapshot_id"` } -func (f *fallbackManif

Re: [PR] feat: (catalog/glue): Fix glue table type [iceberg-go]

zeroshade merged PR #333: URL: https://github.com/apache/iceberg-go/pull/333 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] feat(manifests): Consolidate V1/V2 manifest file objects [iceberg-go]

zeroshade commented on code in PR #332: URL: https://github.com/apache/iceberg-go/pull/332#discussion_r1997688039 ## manifest.go: ## @@ -146,25 +158,66 @@ type fallbackManifestFileV1 struct { AddedSnapshotID *int64 `avro:"added_snapshot_id"` } -func (f *fallbackManife

Re: [PR] Spark: support rewrite on specified target branch [iceberg]

smaheshwar-pltr commented on code in PR #12257: URL: https://github.com/apache/iceberg/pull/12257#discussion_r1997678815 ## core/src/main/java/org/apache/iceberg/actions/RewriteDataFilesCommitManager.java: ## @@ -51,7 +53,12 @@ public RewriteDataFilesCommitManager(Table table, l

[PR] Spark: Call configureTable in ScanTestBase to ensure proper table configuration [iceberg]

drexler-sky opened a new pull request, #12546: URL: https://github.com/apache/iceberg/pull/12546 Seems like `configureTable` is not currently in use. This PR explicitly calls `configureTable` to configure the table. -- This is an automated message from the Apache Git Service. To respond t

Re: [PR] feat(table): Basic Transaction and AddFiles [iceberg-go]

kevinjqliu commented on code in PR #330: URL: https://github.com/apache/iceberg-go/pull/330#discussion_r1997684846 ## table/arrow_utils.go: ## @@ -1250,5 +1656,56 @@ func dataFileStatsFromParquetMetadata(pqmeta *metadata.FileMetaData, statsCols m nullValueCounts

Re: [PR] feat(table): Basic Transaction and AddFiles [iceberg-go]

zeroshade commented on code in PR #330: URL: https://github.com/apache/iceberg-go/pull/330#discussion_r1997678930 ## table/transaction.go: ## @@ -0,0 +1,340 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOT

Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

deniskuzZ commented on code in PR #12528: URL: https://github.com/apache/iceberg/pull/12528#discussion_r1996398329 ## api/src/main/java/org/apache/iceberg/types/Comparators.java: ## @@ -108,6 +109,15 @@ public int compare(StructLike o1, StructLike o2) { return 0;

Re: [PR] feat(table): Basic Transaction and AddFiles [iceberg-go]

zeroshade commented on code in PR #330: URL: https://github.com/apache/iceberg-go/pull/330#discussion_r1997678736 ## table/transaction.go: ## @@ -0,0 +1,340 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOT

Re: [PR] feat(table): Basic Transaction and AddFiles [iceberg-go]

zeroshade commented on code in PR #330: URL: https://github.com/apache/iceberg-go/pull/330#discussion_r1997678156 ## table/transaction.go: ## @@ -0,0 +1,340 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOT

Re: [PR] feat(table): Basic Transaction and AddFiles [iceberg-go]

kevinjqliu commented on code in PR #330: URL: https://github.com/apache/iceberg-go/pull/330#discussion_r1997669352 ## table/transaction.go: ## @@ -0,0 +1,340 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NO

Re: [I] improve pyiceberg CLI [iceberg-python]

iting0321 commented on issue #1784: URL: https://github.com/apache/iceberg-python/issues/1784#issuecomment-2727458081 Hi, I have some questions. If the command is `pyiceberg list`, I need to read the `default` entry in the catalog. However, what if `default` is not set in the catal

[PR] chore(deps): Bump uuid from 1.13.2 to 1.16.0 [iceberg-rust]

dependabot[bot] opened a new pull request, #1092: URL: https://github.com/apache/iceberg-rust/pull/1092 Bumps [uuid](https://github.com/uuid-rs/uuid) from 1.13.2 to 1.16.0. Release notes Sourced from https://github.com/uuid-rs/uuid/releases";>uuid's releases. v1.16.0 What'

[PR] chore(deps): Bump aws-config from 1.5.16 to 1.5.18 [iceberg-rust]

dependabot[bot] opened a new pull request, #1091: URL: https://github.com/apache/iceberg-rust/pull/1091 Bumps [aws-config](https://github.com/smithy-lang/smithy-rs) from 1.5.16 to 1.5.18. Commits See full diff in https://github.com/smithy-lang/smithy-rs/commits";>compare view

[PR] chore(deps): Bump tokio from 1.43.0 to 1.44.1 [iceberg-rust]

dependabot[bot] opened a new pull request, #1094: URL: https://github.com/apache/iceberg-rust/pull/1094 Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.43.0 to 1.44.1. Release notes Sourced from https://github.com/tokio-rs/tokio/releases";>tokio's releases. Tokio v1.4

[PR] chore(deps): Bump tempfile from 3.18.0 to 3.19.0 [iceberg-rust]

dependabot[bot] opened a new pull request, #1093: URL: https://github.com/apache/iceberg-rust/pull/1093 Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.18.0 to 3.19.0. Changelog Sourced from https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md";>tempfile's

[PR] chore(deps): Bump http from 1.2.0 to 1.3.1 [iceberg-rust]

dependabot[bot] opened a new pull request, #1090: URL: https://github.com/apache/iceberg-rust/pull/1090 Bumps [http](https://github.com/hyperium/http) from 1.2.0 to 1.3.1. Release notes Sourced from https://github.com/hyperium/http/releases";>http's releases. v1.3.1 What's

[PR] chore(deps): Bump once_cell from 1.20.3 to 1.21.1 [iceberg-rust]

dependabot[bot] opened a new pull request, #1089: URL: https://github.com/apache/iceberg-rust/pull/1089 Bumps [once_cell](https://github.com/matklad/once_cell) from 1.20.3 to 1.21.1. Changelog Sourced from https://github.com/matklad/once_cell/blob/master/CHANGELOG.md";>once_cell's

[I] Support schema field metadata [iceberg-python]

p1c2u opened a new issue, #1796: URL: https://github.com/apache/iceberg-python/issues/1796 ### Feature Request / Improvement I noticed pyiceberg doesn't support user-defined metadata in schema [fields](https://github.com/apache/iceberg-python/blob/main/pyiceberg/io/pyarrow.py#L609).

Re: [PR] WIP: add view support for Glue Catalog [iceberg]

lawofcycles commented on PR #12544: URL: https://github.com/apache/iceberg/pull/12544#issuecomment-2727276770 I've confirmed that the primary functionality works as expected using both integration tests and an actual AWS environment. I plan to complete the following tasks, and once they're

[PR] Spark: prefix SparkTable with 'iceberg' to clearly identify Iceberg table [iceberg]

cgpoh opened a new pull request, #12543: URL: https://github.com/apache/iceberg/pull/12543 This PR prefixes SparkTable with `iceberg` to clearly identify Iceberg tables and enable metadata collectors (e.g., DataHub) to correctly extract information from Spark's `ProgressReporter`. -- Thi

[PR] build(deps): bump the gomod_updates group with 4 updates [iceberg-go]

dependabot[bot] opened a new pull request, #334: URL: https://github.com/apache/iceberg-go/pull/334 Bumps the gomod_updates group with 4 updates: [github.com/apache/arrow-go/v18](https://github.com/apache/arrow-go), [github.com/aws/aws-sdk-go-v2/service/glue](https://github.com/aws/aws-sdk-