Re: [I] --output json not supported? [iceberg-go]

2025-03-07 Thread via GitHub
ayush1009208 commented on issue #325: URL: https://github.com/apache/iceberg-go/issues/325#issuecomment-2708074624 @knoguchi can you please explaiin a bit more? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] Feature: Write to branches [iceberg-python]

2025-03-07 Thread via GitHub
vinjai commented on PR #941: URL: https://github.com/apache/iceberg-python/pull/941#issuecomment-2708059145 Hey @kevinjqliu Did you get a chance to look at the PR again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Arrow: Infer the types when reading [iceberg-python]

2025-03-07 Thread via GitHub
sungwy commented on code in PR #1669: URL: https://github.com/apache/iceberg-python/pull/1669#discussion_r1985968756 ## pyiceberg/io/pyarrow.py: ## @@ -1655,19 +1646,16 @@ class ArrowProjectionVisitor(SchemaWithPartnerVisitor[pa.Array, Optional[pa.Arra _file_schema: Schema

[PR] Docs: fix typo in rest-catalog-open-api.yaml. [iceberg]

2025-03-07 Thread via GitHub
slfan1989 opened a new pull request, #12480: URL: https://github.com/apache/iceberg/pull/12480 Docs: fix typo in rest-catalog-open-api.yaml. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] GCP: Add Iceberg Catalog for GCP BigQuery Metastore [iceberg]

2025-03-07 Thread via GitHub
hesham-medhat commented on PR #11039: URL: https://github.com/apache/iceberg/pull/11039#issuecomment-2707861898 Thanks for finding the time, Ryan. Unironically, now **I** am on a long medical leave, but Google is committed to this, and we will find the time to finalize this review soon. -

[I] RUSTSEC-2024-0436: paste - no longer maintained [iceberg-rust]

2025-03-07 Thread via GitHub
github-actions[bot] opened a new issue, #1053: URL: https://github.com/apache/iceberg-rust/issues/1053 > paste - no longer maintained | Details || | --- | -- |

Re: [PR] Spark 3.5: Add Comet tests [iceberg]

2025-03-07 Thread via GitHub
github-actions[bot] commented on PR #12176: URL: https://github.com/apache/iceberg/pull/12176#issuecomment-2707763180 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] GCP: Add Iceberg Catalog for GCP BigQuery Metastore [iceberg]

2025-03-07 Thread via GitHub
github-actions[bot] commented on PR #11039: URL: https://github.com/apache/iceberg/pull/11039#issuecomment-2707763058 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Core: Make totalRecordCount optional in PartitionStats [iceberg]

2025-03-07 Thread via GitHub
jbonofre commented on PR #12226: URL: https://github.com/apache/iceberg/pull/12226#issuecomment-2703264435 @aokolnychyi @Fokko wdyt ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Core: Use InternalData with avro and common DataIterable for readers. [iceberg]

2025-03-07 Thread via GitHub
rdblue commented on code in PR #12476: URL: https://github.com/apache/iceberg/pull/12476#discussion_r1985859647 ## core/src/main/java/org/apache/iceberg/ManifestReader.java: ## @@ -133,12 +131,18 @@ private > PartitionSpec readPartitionSpec(InputFile inp private static > Map

Re: [PR] Core: Use InternalData with avro and common DataIterable for readers. [iceberg]

2025-03-07 Thread via GitHub
rdblue commented on code in PR #12476: URL: https://github.com/apache/iceberg/pull/12476#discussion_r1985848527 ## core/src/main/java/org/apache/iceberg/InternalData.java: ## @@ -163,6 +180,11 @@ public interface ReadBuilder { /** Set a custom class for in-memory objects at

Re: [PR] Handling no coordinator and data loss in ICR mode [iceberg]

2025-03-07 Thread via GitHub
bryanck commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1985410403 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/KafkaUtils.java: ## @@ -50,6 +57,29 @@ static ConsumerGroupMetadata consumerGroupMeta

Re: [PR] Core: Use InternalData with avro and common DataIterable for readers. [iceberg]

2025-03-07 Thread via GitHub
rdblue commented on code in PR #12476: URL: https://github.com/apache/iceberg/pull/12476#discussion_r1985843434 ## core/src/main/java/org/apache/iceberg/AllManifestsTable.java: ## @@ -192,13 +191,11 @@ public List deletes() { @Override public CloseableIterable rows() {

Re: [PR] Handling no coordinator and data loss in ICR mode [iceberg]

2025-03-07 Thread via GitHub
kumarpritam863 commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1985439705 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/KafkaUtils.java: ## @@ -50,6 +57,29 @@ static ConsumerGroupMetadata consumerGr

Re: [PR] feat(table): Adds updateSnapshotSummary internal function [iceberg-go]

2025-03-07 Thread via GitHub
zeroshade commented on code in PR #317: URL: https://github.com/apache/iceberg-go/pull/317#discussion_r1985827437 ## manifest.go: ## @@ -1519,7 +1492,13 @@ func (f *fallbackManifestEntryV1) toEntry() *manifestEntryV1 { return &f.manifestEntryV1 } -func (m *manifestEn

Re: [I] Data Integrity Issue with DELETE Operation Using Copy-on-Write (COW) and Equality Deletes [iceberg]

2025-03-07 Thread via GitHub
RussellSpitzer commented on issue #12467: URL: https://github.com/apache/iceberg/issues/12467#issuecomment-2707670589 Wrote a smaller reproduction here https://github.com/apache/iceberg/pull/12479/files Working on a fix -- This is an automated message from the Apache Git Service.

[PR] Spark 3.5: Add Reproduction [iceberg]

2025-03-07 Thread via GitHub
RussellSpitzer opened a new pull request, #12479: URL: https://github.com/apache/iceberg/pull/12479 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

Re: [I] Broken link on the iceberg web page. [iceberg]

2025-03-07 Thread via GitHub
RussellSpitzer closed issue #12475: Broken link on the iceberg web page. URL: https://github.com/apache/iceberg/issues/12475 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Site: Fix Footer Link [iceberg]

2025-03-07 Thread via GitHub
RussellSpitzer commented on PR #12478: URL: https://github.com/apache/iceberg/pull/12478#issuecomment-2707550965 Thanks @flyrain ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] Site: Fix Footer Link [iceberg]

2025-03-07 Thread via GitHub
RussellSpitzer merged PR #12478: URL: https://github.com/apache/iceberg/pull/12478 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [I] Increase concurrency during rewriting position delete files in Iceberg Tables. [iceberg]

2025-03-07 Thread via GitHub
singhpk234 commented on issue #12477: URL: https://github.com/apache/iceberg/issues/12477#issuecomment-2707438401 Have you enabled partial progress ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Handling no coordinator and data loss in ICR mode [iceberg]

2025-03-07 Thread via GitHub
bryanck commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1985470376 ## kafka-connect/kafka-connect/src/test/java/org/apache/iceberg/connect/MockIcebergSinkTask.java: ## @@ -0,0 +1,69 @@ +/* + * + * * Licensed to the Apache Software F

Re: [I] Broken link on the iceberg web page. [iceberg]

2025-03-07 Thread via GitHub
RussellSpitzer commented on issue #12475: URL: https://github.com/apache/iceberg/issues/12475#issuecomment-2707347974 https://github.com/apache/iceberg/pull/12478 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] Support for S3 catalog to work with S3 Tables [iceberg-python]

2025-03-07 Thread via GitHub
madsenwattiq commented on issue #1404: URL: https://github.com/apache/iceberg-python/issues/1404#issuecomment-2707310389 Is there a timeline for getting this feature merged into a release? I know you were working on getting the testing framework on the moto side completed, but that looks

Re: [I] There is a bug with the new version of boto used by s3fs that prevents writes to non-AWS S3 buckets [iceberg-python]

2025-03-07 Thread via GitHub
steveloughran commented on issue #1546: URL: https://github.com/apache/iceberg-python/issues/1546#issuecomment-2707201608 > I might also suggest pyiceberg consider a library that doesn't rely on AWS controlled boto and uses something more like s2cmd that is more S3-agnostic for those of us

[I] Increase concurrency during rewriting position delete files in Iceberg Tables. [iceberg]

2025-03-07 Thread via GitHub
vburenin opened a new issue, #12477: URL: https://github.com/apache/iceberg/issues/12477 ### Feature Request / Improvement I recently run into an issue where Iceberg tables exploded with the large number of delete files. While getting rid of them is not a problem, the problem is how

Re: [PR] fix: kleene logic bug [iceberg-rust]

2025-03-07 Thread via GitHub
Fokko commented on code in PR #1045: URL: https://github.com/apache/iceberg-rust/pull/1045#discussion_r1982961967 ## crates/iceberg/src/arrow/reader.rs: ## @@ -1336,4 +1347,138 @@ message schema { .expect("Some ProjectionMask"); assert_eq!(mask, Project

Re: [PR] OpenAPI: Use more clear language in recommending error responses [iceberg]

2025-03-07 Thread via GitHub
flyrain commented on code in PR #12376: URL: https://github.com/apache/iceberg/pull/12376#discussion_r1985522848 ## open-api/rest-catalog-open-api.yaml: ## @@ -4566,7 +4568,9 @@ components: AuthenticationTimeoutResponse: description: -Credentials have timed

Re: [I] Data Integrity Issue with DELETE Operation Using Copy-on-Write (COW) and Equality Deletes [iceberg]

2025-03-07 Thread via GitHub
sfc-gh-ygu commented on issue #12467: URL: https://github.com/apache/iceberg/issues/12467#issuecomment-2707132156 It's fine if the eq delete files will still be picked up at next read. But I guess we did filter it out based on the sequence number, so that the previous eq delete files won't

Re: [PR] Adds AWS to vendors page [iceberg]

2025-03-07 Thread via GitHub
rbowen commented on PR #12468: URL: https://github.com/apache/iceberg/pull/12468#issuecomment-2707121703 Thanks for all the feedback and suggestions. I think this is probably ready to go now. -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] Handling no coordinator and data loss in ICR mode [iceberg]

2025-03-07 Thread via GitHub
kumarpritam863 commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1985498105 ## kafka-connect/kafka-connect/src/test/java/org/apache/iceberg/connect/MockIcebergSinkTask.java: ## @@ -0,0 +1,69 @@ +/* + * + * * Licensed to the Apache Sof

Re: [PR] Handling no coordinator and data loss in ICR mode [iceberg]

2025-03-07 Thread via GitHub
kumarpritam863 commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1985495529 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/CommitterFactory.java: ## @@ -18,11 +18,13 @@ */ package org.apache.iceberg.connect;

Re: [I] Data Integrity Issue with DELETE Operation Using Copy-on-Write (COW) and Equality Deletes [iceberg]

2025-03-07 Thread via GitHub
RussellSpitzer commented on issue #12467: URL: https://github.com/apache/iceberg/issues/12467#issuecomment-2707097097 @sfc-gh-ygu It has to apply both in Copy on Write. So imagine I have a Data File *data.parquet* |X| |--| |1 | |2| |3| |4| And

Re: [PR] Adds AWS to vendors page [iceberg]

2025-03-07 Thread via GitHub
rbowen commented on code in PR #12468: URL: https://github.com/apache/iceberg/pull/12468#discussion_r1985481595 ## site/docs/vendors.md: ## @@ -22,6 +22,10 @@ title: "Vendors" This page contains some of the vendors who are shipping and supporting Apache Iceberg in their prod

Re: [PR] Core: Use InternalData with avro and common DataIterable for readers. [iceberg]

2025-03-07 Thread via GitHub
danielcweeks commented on code in PR #12476: URL: https://github.com/apache/iceberg/pull/12476#discussion_r1985473714 ## core/src/main/java/org/apache/iceberg/ManifestReader.java: ## @@ -133,12 +131,18 @@ private > PartitionSpec readPartitionSpec(InputFile inp private static

Re: [PR] Core: Use InternalData with avro and common DataIterable for readers. [iceberg]

2025-03-07 Thread via GitHub
danielcweeks commented on code in PR #12476: URL: https://github.com/apache/iceberg/pull/12476#discussion_r1985465684 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetReader.java: ## @@ -34,7 +35,7 @@ import org.apache.parquet.hadoop.ParquetFileReader; import org.apa

Re: [PR] Handling no coordinator and data loss in ICR mode [iceberg]

2025-03-07 Thread via GitHub
bryanck commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1985468217 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/CommitterFactory.java: ## @@ -18,11 +18,13 @@ */ package org.apache.iceberg.connect; +imp

Re: [PR] Handling no coordinator and data loss in ICR mode [iceberg]

2025-03-07 Thread via GitHub
kumarpritam863 commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1985439435 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/CommitterFactory.java: ## @@ -18,11 +18,13 @@ */ package org.apache.iceberg.connect;

[I] Transactions do not support Upsert [iceberg-python]

2025-03-07 Thread via GitHub
suarez-agu opened a new issue, #1776: URL: https://github.com/apache/iceberg-python/issues/1776 ### Feature Request / Improvement Given the issue #1759 I tried instead writing smaller amounts of rows but also wanted a guarantee that all rows would be upserted. Tried wra

Re: [PR] Handling no coordinator and data loss in ICR mode [iceberg]

2025-03-07 Thread via GitHub
bryanck commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1985415982 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/CommitterFactory.java: ## @@ -18,11 +18,13 @@ */ package org.apache.iceberg.connect; +imp

Re: [PR] Adds AWS to vendors page [iceberg]

2025-03-07 Thread via GitHub
ajantha-bhat commented on code in PR #12468: URL: https://github.com/apache/iceberg/pull/12468#discussion_r1985423329 ## site/docs/vendors.md: ## @@ -22,6 +22,10 @@ title: "Vendors" This page contains some of the vendors who are shipping and supporting Apache Iceberg in thei

Re: [PR] Handling no coordinator and data loss in ICR mode [iceberg]

2025-03-07 Thread via GitHub
kumarpritam863 commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1985440317 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitterImpl.java: ## @@ -56,51 +74,57 @@ public int compare(TopicPartition o1

Re: [PR] Handling no coordinator and data loss in ICR mode [iceberg]

2025-03-07 Thread via GitHub
kumarpritam863 commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1985440976 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/IcebergSinkTask.java: ## @@ -46,26 +45,28 @@ public String version() { @Override

Re: [PR] Handling no coordinator and data loss in ICR mode [iceberg]

2025-03-07 Thread via GitHub
kumarpritam863 commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1985440582 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitterImpl.java: ## @@ -56,51 +74,57 @@ public int compare(TopicPartition o1

Re: [PR] Handling no coordinator and data loss in ICR mode [iceberg]

2025-03-07 Thread via GitHub
kumarpritam863 commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1985439435 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/CommitterFactory.java: ## @@ -18,11 +18,13 @@ */ package org.apache.iceberg.connect;

Re: [PR] Core: Use InternalData with avro with common DataIterable for readers. [iceberg]

2025-03-07 Thread via GitHub
danielcweeks commented on code in PR #12476: URL: https://github.com/apache/iceberg/pull/12476#discussion_r1985436472 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetReader.java: ## @@ -34,7 +35,7 @@ import org.apache.parquet.hadoop.ParquetFileReader; import org.apa

Re: [I] Issue in Reading Iceberg tables in Nessie + Minio using Pyiceberg [iceberg-python]

2025-03-07 Thread via GitHub
aliSadegh commented on issue #1560: URL: https://github.com/apache/iceberg-python/issues/1560#issuecomment-2707011014 @HungYangChang did you resolve that? i have same issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

[PR] chore(deps): Bump ring from 0.17.9 to 0.17.13 in /bindings/python [iceberg-rust]

2025-03-07 Thread via GitHub
dependabot[bot] opened a new pull request, #1052: URL: https://github.com/apache/iceberg-rust/pull/1052 Bumps [ring](https://github.com/briansmith/ring) from 0.17.9 to 0.17.13. Changelog Sourced from https://github.com/briansmith/ring/blob/main/RELEASES.md";>ring's changelog.

Re: [PR] Adds AWS to vendors page [iceberg]

2025-03-07 Thread via GitHub
ajantha-bhat commented on code in PR #12468: URL: https://github.com/apache/iceberg/pull/12468#discussion_r1985423329 ## site/docs/vendors.md: ## @@ -22,6 +22,10 @@ title: "Vendors" This page contains some of the vendors who are shipping and supporting Apache Iceberg in thei

Re: [PR] Core: Use InternalData with avro with common DataIterable for readers. [iceberg]

2025-03-07 Thread via GitHub
ajantha-bhat commented on code in PR #12476: URL: https://github.com/apache/iceberg/pull/12476#discussion_r1985386525 ## core/src/main/java/org/apache/iceberg/InternalData.java: ## @@ -34,6 +34,21 @@ import org.slf4j.LoggerFactory; public class InternalData { + /** + * An

Re: [PR] Handling no coordinator and data loss in ICR mode [iceberg]

2025-03-07 Thread via GitHub
bryanck commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1985407865 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitterImpl.java: ## @@ -56,51 +74,57 @@ public int compare(TopicPartition o1, Topic

Re: [PR] Handling no coordinator and data loss in ICR mode [iceberg]

2025-03-07 Thread via GitHub
bryanck commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1985409093 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitterImpl.java: ## @@ -56,51 +74,57 @@ public int compare(TopicPartition o1, Topic

Re: [PR] feat(table): Adds updateSnapshotSummary internal function [iceberg-go]

2025-03-07 Thread via GitHub
zeroshade commented on code in PR #317: URL: https://github.com/apache/iceberg-go/pull/317#discussion_r1985407723 ## manifest.go: ## @@ -1519,7 +1492,13 @@ func (f *fallbackManifestEntryV1) toEntry() *manifestEntryV1 { return &f.manifestEntryV1 } -func (m *manifestEn

Re: [PR] Handling no coordinator and data loss in ICR mode [iceberg]

2025-03-07 Thread via GitHub
bryanck commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1985405348 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/IcebergSinkTask.java: ## @@ -46,26 +45,28 @@ public String version() { @Override public

Re: [PR] Handling no coordinator and data loss in ICR mode [iceberg]

2025-03-07 Thread via GitHub
bryanck commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1985403832 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/Committer.java: ## @@ -20,13 +20,70 @@ import java.util.Collection; import org.apache.iceb

Re: [PR] Handling no coordinator and data loss in ICR mode [iceberg]

2025-03-07 Thread via GitHub
bryanck commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1985402337 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/Committer.java: ## @@ -20,13 +20,70 @@ import java.util.Collection; import org.apache.iceb

Re: [PR] feat(table): Adds updateSnapshotSummary internal function [iceberg-go]

2025-03-07 Thread via GitHub
Fokko commented on code in PR #317: URL: https://github.com/apache/iceberg-go/pull/317#discussion_r1984823063 ## manifest.go: ## @@ -1519,7 +1492,13 @@ func (f *fallbackManifestEntryV1) toEntry() *manifestEntryV1 { return &f.manifestEntryV1 } -func (m *manifestEntryV

[PR] Core: Use InternalData with avro with common DataIterable for readers. [iceberg]

2025-03-07 Thread via GitHub
danielcweeks opened a new pull request, #12476: URL: https://github.com/apache/iceberg/pull/12476 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [I] Spark can't get information from metadata tables [iceberg]

2025-03-07 Thread via GitHub
singhpk234 commented on issue #12466: URL: https://github.com/apache/iceberg/issues/12466#issuecomment-2706908600 Update : We do have tests to read metadata tables https://github.com/apache/iceberg/blob/main/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/Te

Re: [PR] feat: implement display trait for Ident [iceberg-rust]

2025-03-07 Thread via GitHub
ZENOTME commented on PR #1049: URL: https://github.com/apache/iceberg-rust/pull/1049#issuecomment-2705758951 cc @Xuanwo @liurenjie1024 @Fokko @sdd -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Adds AWS to vendors page [iceberg]

2025-03-07 Thread via GitHub
tomtongue commented on code in PR #12468: URL: https://github.com/apache/iceberg/pull/12468#discussion_r1985326690 ## site/docs/vendors.md: ## @@ -22,6 +22,10 @@ title: "Vendors" This page contains some of the vendors who are shipping and supporting Apache Iceberg in their p

Re: [PR] Adds AWS to vendors page [iceberg]

2025-03-07 Thread via GitHub
tomtongue commented on code in PR #12468: URL: https://github.com/apache/iceberg/pull/12468#discussion_r1985326690 ## site/docs/vendors.md: ## @@ -22,6 +22,10 @@ title: "Vendors" This page contains some of the vendors who are shipping and supporting Apache Iceberg in their p

Re: [PR] Adds AWS to vendors page [iceberg]

2025-03-07 Thread via GitHub
tomtongue commented on code in PR #12468: URL: https://github.com/apache/iceberg/pull/12468#discussion_r1985326690 ## site/docs/vendors.md: ## @@ -22,6 +22,10 @@ title: "Vendors" This page contains some of the vendors who are shipping and supporting Apache Iceberg in their p

[I] pyiceberg with hive and S3 fails even when providing creds [iceberg-python]

2025-03-07 Thread via GitHub
lozbrown opened a new issue, #1775: URL: https://github.com/apache/iceberg-python/issues/1775 I'm trying to use pyiceberg within a pod that has access via a role I've configured PYICEBERG_CATALOG__DEFAULT__S3__ROLE_ARN and AWS_ROLE_ARN environment variables but that fails with a Head

Re: [PR] HIVE-28801 Iceberg: Refactor HMS table parameter setting to be able to reuse [iceberg]

2025-03-07 Thread via GitHub
gaborkaszab commented on code in PR #12461: URL: https://github.com/apache/iceberg/pull/12461#discussion_r1985254162 ## hive-metastore/src/main/java/org/apache/iceberg/hive/IcebergTableConverter.java: ## @@ -0,0 +1,197 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [I] Issue during Upsert [iceberg-python]

2025-03-07 Thread via GitHub
suarez-agu commented on issue #1759: URL: https://github.com/apache/iceberg-python/issues/1759#issuecomment-2706773605 I also have the same issue trying to upsert 10k rows. Starting from an empty Table btw -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

2025-03-07 Thread via GitHub
pvary commented on PR #11513: URL: https://github.com/apache/iceberg/pull/11513#issuecomment-2706765443 Not stale -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

Re: [I] Spark can't get information from metadata tables [iceberg]

2025-03-07 Thread via GitHub
singhpk234 commented on issue #12466: URL: https://github.com/apache/iceberg/issues/12466#issuecomment-2706734909 This is a valid seems like metadata identifiers are not being propagated correctly, though which might be a bit tricky as we there is something called supportNamespace with dots

Re: [PR] Arrow: Infer the types when reading [iceberg-python]

2025-03-07 Thread via GitHub
Fokko commented on code in PR #1669: URL: https://github.com/apache/iceberg-python/pull/1669#discussion_r1985246483 ## pyiceberg/io/pyarrow.py: ## @@ -1655,19 +1646,16 @@ class ArrowProjectionVisitor(SchemaWithPartnerVisitor[pa.Array, Optional[pa.Arra _file_schema: Schema

Re: [PR] Core: Make totalRecordCount optional in PartitionStats [iceberg]

2025-03-07 Thread via GitHub
ajantha-bhat commented on PR #12226: URL: https://github.com/apache/iceberg/pull/12226#issuecomment-2706657194 We have already finalized on this change by discussing on https://github.com/apache/iceberg/pull/12098#discussion_r1940551771 and https://lists.apache.org/thread/1dzcv7989q0hchjy

Re: [I] Broken link on the iceberg web page. [iceberg]

2025-03-07 Thread via GitHub
alokwatve commented on issue #12475: URL: https://github.com/apache/iceberg/issues/12475#issuecomment-2706580143 FTR you can reach that link by going to https://iceberg.apache.org/docs/nightly/#user-experience and then clicking "REST Catalog" at the bottom left in the "Features" se

Re: [PR] Spec: Add implementation note on `current-snapshot-id` [iceberg]

2025-03-07 Thread via GitHub
Fokko merged PR #12334: URL: https://github.com/apache/iceberg/pull/12334 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] fix: upgrade ring to v0.17.13 fix Security audit [iceberg-rust]

2025-03-07 Thread via GitHub
Fokko merged PR #1050: URL: https://github.com/apache/iceberg-rust/pull/1050 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]

2025-03-07 Thread via GitHub
lk-1984 commented on issue #1771: URL: https://github.com/apache/iceberg-python/issues/1771#issuecomment-2706132436 There is actually one combination that works, and which is documented in the click help. ``` Usage: pyiceberg [OPTIONS] COMMAND [ARGS]... ``` ``` pyiceb

Re: [I] Upsert with list type not supported [iceberg-python]

2025-03-07 Thread via GitHub
marcoaanogueira commented on issue #1711: URL: https://github.com/apache/iceberg-python/issues/1711#issuecomment-2706495488 Hi @Fokko, I’d like to add that upsert operations also do not work with struct types. When trying to perform an upsert with a struct field, I encounter the foll

Re: [PR] feat(table): Simple location provider [iceberg-go]

2025-03-07 Thread via GitHub
Fokko merged PR #319: URL: https://github.com/apache/iceberg-go/pull/319 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] feat: add construct_ref for table_metadata [iceberg-rust]

2025-03-07 Thread via GitHub
Fokko commented on PR #1043: URL: https://github.com/apache/iceberg-rust/pull/1043#issuecomment-2706390034 Thanks @ZENOTME 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] feat: add construct_ref for table_metadata [iceberg-rust]

2025-03-07 Thread via GitHub
Fokko merged PR #1043: URL: https://github.com/apache/iceberg-rust/pull/1043 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Arrow: Infer the types when reading [iceberg-python]

2025-03-07 Thread via GitHub
sungwy commented on code in PR #1669: URL: https://github.com/apache/iceberg-python/pull/1669#discussion_r1985005690 ## pyiceberg/io/pyarrow.py: ## @@ -1655,19 +1646,16 @@ class ArrowProjectionVisitor(SchemaWithPartnerVisitor[pa.Array, Optional[pa.Arra _file_schema: Schema

Re: [PR] feat(table): Adds updateSnapshotSummary internal function [iceberg-go]

2025-03-07 Thread via GitHub
Fokko commented on code in PR #317: URL: https://github.com/apache/iceberg-go/pull/317#discussion_r1984961840 ## table/snapshots.go: ## @@ -197,3 +315,156 @@ type SnapshotLogEntry struct { SnapshotID int64 `json:"snapshot-id"` TimestampMs int64 `json:"timestamp-

Re: [PR] feat(table): Adds updateSnapshotSummary internal function [iceberg-go]

2025-03-07 Thread via GitHub
Fokko commented on code in PR #317: URL: https://github.com/apache/iceberg-go/pull/317#discussion_r1984950982 ## table/snapshots.go: ## @@ -53,14 +55,130 @@ func ValidOperation(s string) (Operation, error) { return "", fmt.Errorf("%w: found '%s'", ErrInvalidOperation, s)

Re: [PR] Spark-3.5: Add procedure to compute partition stats [iceberg]

2025-03-07 Thread via GitHub
ajantha-bhat commented on code in PR #12451: URL: https://github.com/apache/iceberg/pull/12451#discussion_r1984926963 ## api/src/main/java/org/apache/iceberg/actions/ComputePartitionStats.java: ## @@ -0,0 +1,40 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [I] URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME [iceberg-python]

2025-03-07 Thread via GitHub
lk-1984 closed issue #1771: URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME URL: https://github.com/apache/iceberg-python/issues/1771 -- This is an automated message from the Apache Git Se

Re: [PR] feat: implement display trait for Ident [iceberg-rust]

2025-03-07 Thread via GitHub
Fokko merged PR #1049: URL: https://github.com/apache/iceberg-rust/pull/1049 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Writing to Iceberg Tables in Polaris not working [iceberg]

2025-03-07 Thread via GitHub
nastra commented on issue #12469: URL: https://github.com/apache/iceberg/issues/12469#issuecomment-2706231209 Please open the issue at https://github.com/apache/polaris as the Iceberg community doesn't maintain Polaris -- This is an automated message from the Apache Git Service. To respon

Re: [I] Writing to Iceberg Tables in Polaris not working [iceberg]

2025-03-07 Thread via GitHub
nastra closed issue #12469: Writing to Iceberg Tables in Polaris not working URL: https://github.com/apache/iceberg/issues/12469 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] feat: add construct_ref for table_metadata [iceberg-rust]

2025-03-07 Thread via GitHub
ZENOTME commented on PR #1043: URL: https://github.com/apache/iceberg-rust/pull/1043#issuecomment-2705997518 > LGTM, should we throw in a test? Have added the test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [I] RUSTSEC-2025-0009: Some AES functions may panic when overflow checking is enabled. [iceberg-rust]

2025-03-07 Thread via GitHub
Xuanwo closed issue #1046: RUSTSEC-2025-0009: Some AES functions may panic when overflow checking is enabled. URL: https://github.com/apache/iceberg-rust/issues/1046 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Spec: Update partition stats for V3 [iceberg]

2025-03-07 Thread via GitHub
ajantha-bhat commented on PR #12098: URL: https://github.com/apache/iceberg/pull/12098#issuecomment-2706118893 Looks like we have everyone's approval. Can we please merge this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] feat(table): Adds updateSnapshotSummary internal function [iceberg-go]

2025-03-07 Thread via GitHub
Fokko commented on code in PR #317: URL: https://github.com/apache/iceberg-go/pull/317#discussion_r1984821694 ## manifest.go: ## @@ -1519,7 +1492,13 @@ func (f *fallbackManifestEntryV1) toEntry() *manifestEntryV1 { return &f.manifestEntryV1 } -func (m *manifestEntryV

Re: [I] [Feature] Add Support for Distributed Write [iceberg-python]

2025-03-07 Thread via GitHub
andormarkus commented on issue #1751: URL: https://github.com/apache/iceberg-python/issues/1751#issuecomment-2706056065 Hi @Fokko, I'd like to share a working example that demonstrates how to serialize and deserialize both partition and non-partitioned tables: ```python output

Re: [PR] feat(table): make Table aware of its catalog [iceberg-go]

2025-03-07 Thread via GitHub
Fokko merged PR #323: URL: https://github.com/apache/iceberg-go/pull/323 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] API, Core: Add geometry and geography types support [iceberg]

2025-03-07 Thread via GitHub
szehon-ho commented on code in PR #12346: URL: https://github.com/apache/iceberg/pull/12346#discussion_r1984779883 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -543,6 +561,120 @@ public int hashCode() { } } + public static class GeometryType extends

Re: [PR] 0.9.0 post release [iceberg-python]

2025-03-07 Thread via GitHub
Fokko merged PR #1773: URL: https://github.com/apache/iceberg-python/pull/1773 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] feat: support strict projection [iceberg-rust]

2025-03-07 Thread via GitHub
Fokko commented on code in PR #946: URL: https://github.com/apache/iceberg-rust/pull/946#discussion_r1984748281 ## crates/iceberg/src/spec/transform.rs: ## @@ -296,6 +297,165 @@ impl Transform { } } +/// Strictly projects a given predicate according to the tr

Re: [PR] API, Core: Add geometry and geography types support [iceberg]

2025-03-07 Thread via GitHub
Kontinuation commented on code in PR #12346: URL: https://github.com/apache/iceberg/pull/12346#discussion_r1984738489 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -543,6 +561,120 @@ public int hashCode() { } } + public static class GeometryType exte

Re: [PR] fix: upgrade ring to v0.17.13 fix Security audit [iceberg-rust]

2025-03-07 Thread via GitHub
Fokko commented on PR #1050: URL: https://github.com/apache/iceberg-rust/pull/1050#issuecomment-2705951130 Thanks for fixing this @ZENOTME -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] fix: upgrade ring to v0.17.13 fix Security audit [iceberg-rust]

2025-03-07 Thread via GitHub
ZENOTME commented on PR #1050: URL: https://github.com/apache/iceberg-rust/pull/1050#issuecomment-2705910656 cc @Xuanwo @liurenjie1024 @Fokko @sdd -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[I] Flink/Spark: add visitor support for variant [iceberg]

2025-03-07 Thread via GitHub
aihuaxu opened a new issue, #12473: URL: https://github.com/apache/iceberg/issues/12473 ### Feature Request / Improvement Follow up #11831, some Variant support needs to be added in Flink and Spark modules: FlinkFixupTypes SparkFixupTypes PruneColumnsWithoutReordering

Re: [I] Provide differert read interface for reader [iceberg-rust]

2025-03-07 Thread via GitHub
sdd commented on issue #1047: URL: https://github.com/apache/iceberg-rust/issues/1047#issuecomment-2705918721 Seems like a reasonable idea to me. If my 5 open PRs for delete file read support get reviewed and merged then implementing what you need would be pretty trivial on top of them :-)

  1   2   >