Re: [PR] Data: Add partition stats writer and reader [iceberg]

2025-02-24 Thread via GitHub
gaborkaszab commented on code in PR #11216: URL: https://github.com/apache/iceberg/pull/11216#discussion_r1969140073 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java: ## @@ -0,0 +1,286 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Spec additions for encryption [iceberg]

2025-02-24 Thread via GitHub
ggershinsky commented on code in PR #12162: URL: https://github.com/apache/iceberg/pull/12162#discussion_r1969113128 ## format/spec.md: ## @@ -975,6 +978,23 @@ The unified partition type looks like `Struct`. and then the table has evolved into `spec#1` which has just one field

Re: [PR] Arrow, Parquet, Spark 3.5, Flink 1.20: Avoid deprecated method [iceberg]

2025-02-24 Thread via GitHub
nastra merged PR #11874: URL: https://github.com/apache/iceberg/pull/11874 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Core: Add "volatile" to HadoopFileIO#hadoopConf [iceberg]

2025-02-24 Thread via GitHub
nastra merged PR #12388: URL: https://github.com/apache/iceberg/pull/12388 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

[PR] fix: fix version of mechete [iceberg-rust]

2025-02-24 Thread via GitHub
ZENOTME opened a new pull request, #1006: URL: https://github.com/apache/iceberg-rust/pull/1006 cargo-machete just released 0.8.0 and it needs new rust version. This PR fix it. ``` error: failed to download `cargo-machete v0.8.0` Caused by: unable to get packages from s

Re: [I] Add public accessors to `BinaryExpression` and `SetExpression` [iceberg-rust]

2025-02-24 Thread via GitHub
liurenjie1024 closed issue #1003: Add public accessors to `BinaryExpression` and `SetExpression` URL: https://github.com/apache/iceberg-rust/issues/1003 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] make predicate accessor functions public [iceberg-rust]

2025-02-24 Thread via GitHub
liurenjie1024 merged PR #1005: URL: https://github.com/apache/iceberg-rust/pull/1005 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Arrow, Parquet, Spark 3.5, Flink 1.20: Avoid deprecated method [iceberg]

2025-02-24 Thread via GitHub
ebyhr commented on code in PR #11874: URL: https://github.com/apache/iceberg/pull/11874#discussion_r1969078113 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/data/ParquetWithFlinkSchemaVisitor.java: ## @@ -51,106 +51,100 @@ public static T visit( } else {

Re: [PR] Handling no coordinator and data loss in ICR mode [iceberg]

2025-02-24 Thread via GitHub
kumarpritam863 commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1969030650 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/Committer.java: ## @@ -19,14 +19,21 @@ package org.apache.iceberg.connect; import j

Re: [PR] Build: Bump junit to v5.12.0 [iceberg]

2025-02-24 Thread via GitHub
ajantha-bhat commented on PR #12391: URL: https://github.com/apache/iceberg/pull/12391#issuecomment-2680732936 @nastra: We probably need this PR to get merged first at Nessie side. https://github.com/projectnessie/nessie/pull/10442 Looks like Junit 5.12.0 has some breaking changes

Re: [PR] Spark: Bump Spark 3.5 to 3.5.5 [iceberg]

2025-02-24 Thread via GitHub
pan3793 commented on code in PR #12396: URL: https://github.com/apache/iceberg/pull/12396#discussion_r1968950998 ## gradle/libs.versions.toml: ## @@ -79,7 +79,7 @@ scala-collection-compat = "2.13.0" slf4j = "2.0.16" snowflake-jdbc = "3.22.0" spark-hive34 = "3.4.4" -spark-hive

Re: [PR] feat: Add existing parquet files [iceberg-rust]

2025-02-24 Thread via GitHub
jonathanc-n commented on code in PR #960: URL: https://github.com/apache/iceberg-rust/pull/960#discussion_r1968937841 ## crates/iceberg/src/writer/file_writer/parquet_writer.rs: ## @@ -391,6 +404,79 @@ impl ParquetWriter { ); Ok(builder) } + +/// `

Re: [PR] Core: Add Variant logical type for Avro [iceberg]

2025-02-24 Thread via GitHub
aihuaxu commented on code in PR #12238: URL: https://github.com/apache/iceberg/pull/12238#discussion_r1968822976 ## api/src/main/java/org/apache/iceberg/variants/Variant.java: ## @@ -22,6 +22,10 @@ /** A variant metadata and value pair. */ public interface Variant { + Strin

Re: [PR] Spec: Add implementation note on `current-snapshot-id` [iceberg]

2025-02-24 Thread via GitHub
danielcweeks commented on code in PR #12334: URL: https://github.com/apache/iceberg/pull/12334#discussion_r1968935239 ## format/spec.md: ## @@ -1754,6 +1754,14 @@ Snapshot summary can include metrics fields to track numeric stats of the snapsh | **`engine-name`**| "spa

Re: [PR] fix: refine doc for write support [iceberg-rust]

2025-02-24 Thread via GitHub
ZENOTME commented on code in PR #999: URL: https://github.com/apache/iceberg-rust/pull/999#discussion_r1968914615 ## crates/iceberg/src/lib.rs: ## @@ -50,6 +50,87 @@ //! Ok(()) //! } //! ``` +//! +//! ## Fast append data to table +//! +//! ```rust, no_run Review Comment:

Re: [PR] feat: support position delete writer [iceberg-rust]

2025-02-24 Thread via GitHub
ZENOTME commented on code in PR #704: URL: https://github.com/apache/iceberg-rust/pull/704#discussion_r1968910011 ## crates/iceberg/src/writer/base_writer/position_delete_file_writer.rs: ## @@ -0,0 +1,277 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or m

[PR] make predicate accessor functions public [iceberg-rust]

2025-02-24 Thread via GitHub
Nathan-Fenner opened a new pull request, #1005: URL: https://github.com/apache/iceberg-rust/pull/1005 Makes the accessor functions on `UnaryExpression`, `BinaryExpression`, and `SetExpression` public, making it easier to programmatically transform `iceberg::expr::Predicate` expressions from

Re: [I] org.apache.iceberg.spark.source.SerializableTableWithSize cannot be cast to org.apache.iceberg.Table [iceberg]

2025-02-24 Thread via GitHub
zryogi commented on issue #8978: URL: https://github.com/apache/iceberg/issues/8978#issuecomment-2680474170 Can confirm this is still an issue in `iceberg-spark-runtime-3.5_2.12-1.8.0` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] support pyarrow recordbatch as a valid data source for writing Iceberg table [iceberg-python]

2025-02-24 Thread via GitHub
djouallah commented on issue #1004: URL: https://github.com/apache/iceberg-python/issues/1004#issuecomment-2680407058 > [@djouallah](https://github.com/djouallah) do you have the stack trace? Im curious which part of `append` is causing the OOM. BTW there's an [`to_arrow_batch_reader()`](h

Re: [I] Add public accessors to `BinaryExpression` and `SetExpression` [iceberg-rust]

2025-02-24 Thread via GitHub
Nathan-Fenner commented on issue #1003: URL: https://github.com/apache/iceberg-rust/issues/1003#issuecomment-2680383378 Yeah, I am happy to contribute the change as a PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] feat: support position delete writer [iceberg-rust]

2025-02-24 Thread via GitHub
liurenjie1024 commented on code in PR #704: URL: https://github.com/apache/iceberg-rust/pull/704#discussion_r1968813999 ## crates/iceberg/src/writer/base_writer/position_delete_file_writer.rs: ## @@ -0,0 +1,277 @@ +// Licensed to the Apache Software Foundation (ASF) under one +/

Re: [I] support pyarrow recordbatch as a valid data source for writing Iceberg table [iceberg-python]

2025-02-24 Thread via GitHub
kevinjqliu commented on issue #1004: URL: https://github.com/apache/iceberg-python/issues/1004#issuecomment-2680363742 @djouallah do you have the stack trace? Im curious which part of `append` is causing the OOM. BTW there's an [`to_arrow_batch_reader()`](https://github.com/apache/iceb

Re: [PR] feat: Add existing parquet files [iceberg-rust]

2025-02-24 Thread via GitHub
liurenjie1024 commented on code in PR #960: URL: https://github.com/apache/iceberg-rust/pull/960#discussion_r1968805045 ## crates/iceberg/src/writer/file_writer/parquet_writer.rs: ## @@ -391,6 +404,79 @@ impl ParquetWriter { ); Ok(builder) } + +///

[I] Replace `FileMetadata` in parquet writer with in memory representation. [iceberg-rust]

2025-02-24 Thread via GitHub
liurenjie1024 opened a new issue, #1004: URL: https://github.com/apache/iceberg-rust/issues/1004 Currently in [parquet writer](https://github.com/apache/iceberg-rust/blob/1c632b869a0b9076f49df3b266c7d9fe1a648125/crates/iceberg/src/writer/file_writer/parquet_writer.rs#L312) the `FileMetadata

Re: [PR] feat: Add existing parquet files [iceberg-rust]

2025-02-24 Thread via GitHub
liurenjie1024 commented on code in PR #960: URL: https://github.com/apache/iceberg-rust/pull/960#discussion_r1968794863 ## crates/iceberg/src/writer/file_writer/parquet_writer.rs: ## @@ -391,6 +404,79 @@ impl ParquetWriter { ); Ok(builder) } + +///

Re: [PR] Core: Apply correct metric configs in GenericAppenderFactory [iceberg]

2025-02-24 Thread via GitHub
XBaith commented on code in PR #12366: URL: https://github.com/apache/iceberg/pull/12366#discussion_r1968759987 ## data/src/main/java/org/apache/iceberg/data/GenericAppenderFactory.java: ## @@ -44,41 +46,85 @@ /** Factory to create a new {@link FileAppender} to write {@link R

Re: [PR] feat: support `arrow_struct_to_iceberg_struct` [iceberg-rust]

2025-02-24 Thread via GitHub
liurenjie1024 commented on code in PR #731: URL: https://github.com/apache/iceberg-rust/pull/731#discussion_r1968754297 ## crates/iceberg/src/spec/schema/visitor.rs: ## @@ -121,3 +121,162 @@ pub fn visit_schema(schema: &Schema, visitor: &mut V) -> Resul let result = visit_

Re: [I] Add public accessors to `BinaryExpression` and `SetExpression` [iceberg-rust]

2025-02-24 Thread via GitHub
liurenjie1024 commented on issue #1003: URL: https://github.com/apache/iceberg-rust/issues/1003#issuecomment-2680281126 Thanks @Nathan-Fenner , are you interested in contributing this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Handling no coordinator and data loss in ICR mode [iceberg]

2025-02-24 Thread via GitHub
kumarpritam863 commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1968751068 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/Committer.java: ## @@ -19,14 +19,21 @@ package org.apache.iceberg.connect; import j

Re: [PR] Handling no coordinator and data loss in ICR mode [iceberg]

2025-02-24 Thread via GitHub
kumarpritam863 commented on code in PR #12372: URL: https://github.com/apache/iceberg/pull/12372#discussion_r1968750575 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/Committer.java: ## @@ -19,14 +19,21 @@ package org.apache.iceberg.connect; import j

Re: [PR] Spec: Add implementation note on `current-snapshot-id` [iceberg]

2025-02-24 Thread via GitHub
liurenjie1024 commented on code in PR #12334: URL: https://github.com/apache/iceberg/pull/12334#discussion_r1968725930 ## format/spec.md: ## @@ -1754,6 +1754,14 @@ Snapshot summary can include metrics fields to track numeric stats of the snapsh | **`engine-name`**| "sp

Re: [PR] HIVE: release expired hive lock [iceberg]

2025-02-24 Thread via GitHub
xicm closed pull request #12392: HIVE: release expired hive lock URL: https://github.com/apache/iceberg/pull/12392 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [PR] HIVE: release expired hive lock [iceberg]

2025-02-24 Thread via GitHub
xicm commented on PR #12392: URL: https://github.com/apache/iceberg/pull/12392#issuecomment-2680207999 > HMS should have a service to release the expired locks. It is not the responsibility of the client to do so. Okay, got it. -- This is an automated message from the Apache Git Se

Re: [I] support pyarrow recordbatch as a valid data source for writing Iceberg table [iceberg-python]

2025-02-24 Thread via GitHub
djouallah commented on issue #1004: URL: https://github.com/apache/iceberg-python/issues/1004#issuecomment-2680180934 any news :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] Retry on NoSuchNamespaceException not found in rename table for rest catalog [iceberg]

2025-02-24 Thread via GitHub
huan233usc commented on PR #12159: URL: https://github.com/apache/iceberg/pull/12159#issuecomment-2680119199 Updated PR with comments, working on resolving conflict -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] Core: Add Variant logical type for Avro [iceberg]

2025-02-24 Thread via GitHub
rdblue commented on PR #12238: URL: https://github.com/apache/iceberg/pull/12238#issuecomment-2680076226 This is close. There are just two blockers for me: 1. This should not expose unnecessary constants in `Variant`. Cross-project constant use is unnecessary API surface and can cause iss

[PR] parallelize `add_files` [iceberg-python]

2025-02-24 Thread via GitHub
vtk9 opened a new pull request, #1717: URL: https://github.com/apache/iceberg-python/pull/1717 - `parquet_files_to_data_files` changed to `parquet_file_to_data_files` which processes a single parquet file and returns a `DataFile` - `_parquet_files_to_data_files` uses internal ExecutorFact

Re: [PR] Core: Add Variant logical type for Avro [iceberg]

2025-02-24 Thread via GitHub
rdblue commented on code in PR #12238: URL: https://github.com/apache/iceberg/pull/12238#discussion_r1968643588 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetVariantVisitor.java: ## @@ -21,17 +21,14 @@ import java.util.List; import org.apache.iceberg.relocated.com

Re: [PR] Spark: Infer partition spec in ADD_FILES procedure for FileTables than taking latest table spec [iceberg]

2025-02-24 Thread via GitHub
bharos commented on code in PR #12327: URL: https://github.com/apache/iceberg/pull/12327#discussion_r1968579723 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAddFilesProcedure.java: ## @@ -635,6 +635,67 @@ public void addFilteredPartitions

Re: [PR] Build: Bump polars from 1.22.0 to 1.23.0 [iceberg-python]

2025-02-24 Thread via GitHub
kevinjqliu merged PR #1715: URL: https://github.com/apache/iceberg-python/pull/1715 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Core: Add Variant logical type for Avro [iceberg]

2025-02-24 Thread via GitHub
rdblue commented on code in PR #12238: URL: https://github.com/apache/iceberg/pull/12238#discussion_r1968639335 ## core/src/main/java/org/apache/iceberg/avro/AvroSchemaVisitor.java: ## @@ -46,7 +47,18 @@ public static T visit(Schema schema, AvroSchemaVisitor visitor) {

Re: [PR] Core: Add Variant logical type for Avro [iceberg]

2025-02-24 Thread via GitHub
rdblue commented on code in PR #12238: URL: https://github.com/apache/iceberg/pull/12238#discussion_r1968636242 ## core/src/main/java/org/apache/iceberg/avro/AvroCustomOrderSchemaVisitor.java: ## @@ -47,7 +47,13 @@ public static T visit(Schema schema, AvroCustomOrderSchemaVisi

Re: [PR] Core: Add Variant logical type for Avro [iceberg]

2025-02-24 Thread via GitHub
rdblue commented on code in PR #12238: URL: https://github.com/apache/iceberg/pull/12238#discussion_r1968634436 ## api/src/main/java/org/apache/iceberg/variants/Variant.java: ## @@ -22,6 +22,10 @@ /** A variant metadata and value pair. */ public interface Variant { + String

Re: [I] Refactor TestIcebergCommitter state recovery unit tests to use checkpointId=1 [iceberg]

2025-02-24 Thread via GitHub
github-actions[bot] closed issue #10942: Refactor TestIcebergCommitter state recovery unit tests to use checkpointId=1 URL: https://github.com/apache/iceberg/issues/10942 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] Core: Properly detect metadata tables [iceberg]

2025-02-24 Thread via GitHub
github-actions[bot] commented on PR #11963: URL: https://github.com/apache/iceberg/pull/11963#issuecomment-2680003701 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Core: Set missing table-default property in RESTSessionCatalog [iceberg]

2025-02-24 Thread via GitHub
github-actions[bot] closed pull request #11646: Core: Set missing table-default property in RESTSessionCatalog URL: https://github.com/apache/iceberg/pull/11646 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] Use SupportsPrefixOperations for Remove OrphanFile Procedure [iceberg]

2025-02-24 Thread via GitHub
github-actions[bot] commented on PR #11906: URL: https://github.com/apache/iceberg/pull/11906#issuecomment-2680003629 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] S3: Disable strong integrity checksums [iceberg]

2025-02-24 Thread via GitHub
mmgaggle commented on PR #12264: URL: https://github.com/apache/iceberg/pull/12264#issuecomment-2680003867 > > How hard would it be for us to get some integration tests with one of these systems into the Iceberg project? Seems like we should have some coverage for these other S3-Compat syst

Re: [PR] S3: Disable strong integrity checksums [iceberg]

2025-02-24 Thread via GitHub
mmgaggle commented on code in PR #12264: URL: https://github.com/apache/iceberg/pull/12264#discussion_r1968606398 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3RequestUtil.java: ## @@ -149,4 +151,10 @@ static void configurePermission( Function aclSetter) { aclSett

Re: [PR] Core: Properly detect metadata tables [iceberg]

2025-02-24 Thread via GitHub
github-actions[bot] closed pull request #11963: Core: Properly detect metadata tables URL: https://github.com/apache/iceberg/pull/11963 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Purge RCK test entries in `afterEach` instead of `beforeEach` [iceberg]

2025-02-24 Thread via GitHub
github-actions[bot] commented on PR #11699: URL: https://github.com/apache/iceberg/pull/11699#issuecomment-2680003533 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Fix comment on `WRITE_OBJECT_STORE_PARTITIONED_PATHS` table property [iceberg]

2025-02-24 Thread via GitHub
github-actions[bot] commented on PR #11798: URL: https://github.com/apache/iceberg/pull/11798#issuecomment-2680003598 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [I] Support committed callback [iceberg]

2025-02-24 Thread via GitHub
github-actions[bot] closed issue #10936: Support committed callback URL: https://github.com/apache/iceberg/issues/10936 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] Fix comment on `WRITE_OBJECT_STORE_PARTITIONED_PATHS` table property [iceberg]

2025-02-24 Thread via GitHub
github-actions[bot] closed pull request #11798: Fix comment on `WRITE_OBJECT_STORE_PARTITIONED_PATHS` table property URL: https://github.com/apache/iceberg/pull/11798 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Core: Expose `added_rows_count`, `existing_rows_count` and `deleted_rows_count` fields in all_manifests and manifests tables [iceberg]

2025-02-24 Thread via GitHub
github-actions[bot] closed pull request #11679: Core: Expose `added_rows_count`, `existing_rows_count` and `deleted_rows_count` fields in all_manifests and manifests tables URL: https://github.com/apache/iceberg/pull/11679 -- This is an automated message from the Apache Git Service. To respo

Re: [PR] Core: Expose `added_rows_count`, `existing_rows_count` and `deleted_rows_count` fields in all_manifests and manifests tables [iceberg]

2025-02-24 Thread via GitHub
github-actions[bot] commented on PR #11679: URL: https://github.com/apache/iceberg/pull/11679#issuecomment-2680003502 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Core: Set missing table-default property in RESTSessionCatalog [iceberg]

2025-02-24 Thread via GitHub
github-actions[bot] commented on PR #11646: URL: https://github.com/apache/iceberg/pull/11646#issuecomment-2680003457 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Purge RCK test entries in `afterEach` instead of `beforeEach` [iceberg]

2025-02-24 Thread via GitHub
github-actions[bot] closed pull request #11699: Purge RCK test entries in `afterEach` instead of `beforeEach` URL: https://github.com/apache/iceberg/pull/11699 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] Support custom spark procedure in plugin mode for iceberg [iceberg]

2025-02-24 Thread via GitHub
github-actions[bot] commented on issue #10906: URL: https://github.com/apache/iceberg/issues/10906#issuecomment-2680003318 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Support committed callback [iceberg]

2025-02-24 Thread via GitHub
github-actions[bot] commented on issue #10936: URL: https://github.com/apache/iceberg/issues/10936#issuecomment-2680003348 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Iceberg materialized view [iceberg]

2025-02-24 Thread via GitHub
github-actions[bot] closed issue #10890: Iceberg materialized view URL: https://github.com/apache/iceberg/issues/10890 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [I] Support custom spark procedure in plugin mode for iceberg [iceberg]

2025-02-24 Thread via GitHub
github-actions[bot] closed issue #10906: Support custom spark procedure in plugin mode for iceberg URL: https://github.com/apache/iceberg/issues/10906 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [I] Iceberg materialized view [iceberg]

2025-02-24 Thread via GitHub
github-actions[bot] commented on issue #10890: URL: https://github.com/apache/iceberg/issues/10890#issuecomment-2680003283 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [PR] S3: Disable strong integrity checksums [iceberg]

2025-02-24 Thread via GitHub
mmgaggle commented on code in PR #12264: URL: https://github.com/apache/iceberg/pull/12264#discussion_r1968606398 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3RequestUtil.java: ## @@ -149,4 +151,10 @@ static void configurePermission( Function aclSetter) { aclSett

Re: [I] Push upsert changes detection to Arrow [iceberg-python]

2025-02-24 Thread via GitHub
kevinjqliu commented on issue #1679: URL: https://github.com/apache/iceberg-python/issues/1679#issuecomment-2679985290 I think this is done in #1685, right? @Fokko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Spark: Infer partition spec in ADD_FILES procedure for FileTables than taking latest table spec [iceberg]

2025-02-24 Thread via GitHub
RussellSpitzer commented on code in PR #12327: URL: https://github.com/apache/iceberg/pull/12327#discussion_r1968561130 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/Spark3Util.java: ## @@ -858,6 +858,23 @@ public static String quotedFullIdentifier(String catalogNa

Re: [I] Push upsert changes detection to Arrow [iceberg-python]

2025-02-24 Thread via GitHub
mattmartin14 commented on issue #1679: URL: https://github.com/apache/iceberg-python/issues/1679#issuecomment-2679973969 @Fokko - good stuff here :). I see you all are starting to ramp up the efforts with working on the upsert stuff. -- This is an automated message from the Apache Git

Re: [PR] Build: Bump datafusion from 44.0.0 to 45.2.0 [iceberg-python]

2025-02-24 Thread via GitHub
kevinjqliu merged PR #1714: URL: https://github.com/apache/iceberg-python/pull/1714 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Build: Bump datafusion from 44.0.0 to 45.2.0 [iceberg-python]

2025-02-24 Thread via GitHub
kevinjqliu commented on PR #1714: URL: https://github.com/apache/iceberg-python/pull/1714#issuecomment-2679937398 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Build: Bump mypy-boto3-glue from 1.36.4 to 1.37.0 [iceberg-python]

2025-02-24 Thread via GitHub
kevinjqliu merged PR #1716: URL: https://github.com/apache/iceberg-python/pull/1716 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Build: Bump mkdocs-autorefs from 1.3.1 to 1.4.0 [iceberg-python]

2025-02-24 Thread via GitHub
kevinjqliu merged PR #1712: URL: https://github.com/apache/iceberg-python/pull/1712 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Build: Bump moto from 5.0.28 to 5.1.0 [iceberg-python]

2025-02-24 Thread via GitHub
kevinjqliu merged PR #1713: URL: https://github.com/apache/iceberg-python/pull/1713 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [I] don't show warning about bucket [iceberg-python]

2025-02-24 Thread via GitHub
kevinjqliu commented on issue #1705: URL: https://github.com/apache/iceberg-python/issues/1705#issuecomment-2679934898 Thanks @djouallah The new 0.9.0 RC is out (https://pypi.org/project/pyiceberg/0.9.0rc2/) Let us know if you run into any issue :) You can also [verify the relea

Re: [PR] Spark: Infer partition spec in ADD_FILES procedure for FileTables than taking latest table spec [iceberg]

2025-02-24 Thread via GitHub
RussellSpitzer commented on code in PR #12327: URL: https://github.com/apache/iceberg/pull/12327#discussion_r1968558602 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/AddFilesProcedure.java: ## @@ -265,55 +271,16 @@ public String description() { retur

Re: [PR] Spark: Infer partition spec in ADD_FILES procedure for FileTables than taking latest table spec [iceberg]

2025-02-24 Thread via GitHub
RussellSpitzer commented on code in PR #12327: URL: https://github.com/apache/iceberg/pull/12327#discussion_r1968556347 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAddFilesProcedure.java: ## @@ -635,6 +635,67 @@ public void addFilteredPa

Re: [PR] Spark: Infer partition spec in ADD_FILES procedure for FileTables than taking latest table spec [iceberg]

2025-02-24 Thread via GitHub
RussellSpitzer commented on code in PR #12327: URL: https://github.com/apache/iceberg/pull/12327#discussion_r1968555311 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/AddFilesProcedure.java: ## @@ -265,55 +271,16 @@ public String description() { retur

Re: [PR] Spark: Infer partition spec in ADD_FILES procedure for FileTables than taking latest table spec [iceberg]

2025-02-24 Thread via GitHub
RussellSpitzer commented on code in PR #12327: URL: https://github.com/apache/iceberg/pull/12327#discussion_r1968554090 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/AddFilesProcedure.java: ## @@ -196,11 +194,23 @@ private void importFileTable( Str

[PR] Build: Bump datafusion from 44.0.0 to 45.2.0 [iceberg-python]

2025-02-24 Thread via GitHub
dependabot[bot] opened a new pull request, #1714: URL: https://github.com/apache/iceberg-python/pull/1714 Bumps [datafusion](https://github.com/apache/datafusion-python) from 44.0.0 to 45.2.0. Commits https://github.com/apache/datafusion-python/commit/639e2abc9ce1e7753f9779c749

Re: [PR] Spark: Infer partition spec in ADD_FILES procedure for FileTables than taking latest table spec [iceberg]

2025-02-24 Thread via GitHub
RussellSpitzer commented on code in PR #12327: URL: https://github.com/apache/iceberg/pull/12327#discussion_r1968547373 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAddFilesProcedure.java: ## @@ -635,6 +635,67 @@ public void addFilteredPa

Re: [PR] Spark: Infer partition spec in ADD_FILES procedure for FileTables than taking latest table spec [iceberg]

2025-02-24 Thread via GitHub
RussellSpitzer commented on code in PR #12327: URL: https://github.com/apache/iceberg/pull/12327#discussion_r1968552113 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java: ## @@ -1124,7 +1115,67 @@ private static PartitionSpec findCompatibleSpec(

Re: [PR] Spark: Infer partition spec in ADD_FILES procedure for FileTables than taking latest table spec [iceberg]

2025-02-24 Thread via GitHub
RussellSpitzer commented on code in PR #12327: URL: https://github.com/apache/iceberg/pull/12327#discussion_r1968550486 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java: ## @@ -1091,21 +1094,9 @@ private ExecutorService getService() { /** *

Re: [PR] Spark: Infer partition spec in ADD_FILES procedure for FileTables than taking latest table spec [iceberg]

2025-02-24 Thread via GitHub
RussellSpitzer commented on code in PR #12327: URL: https://github.com/apache/iceberg/pull/12327#discussion_r1968551184 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java: ## @@ -1124,7 +1115,67 @@ private static PartitionSpec findCompatibleSpec(

Re: [PR] Spark: Infer partition spec in ADD_FILES procedure for FileTables than taking latest table spec [iceberg]

2025-02-24 Thread via GitHub
RussellSpitzer commented on code in PR #12327: URL: https://github.com/apache/iceberg/pull/12327#discussion_r1968545924 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAddFilesProcedure.java: ## @@ -635,6 +635,67 @@ public void addFilteredPa

[PR] Build: Bump mypy-boto3-glue from 1.36.4 to 1.37.0 [iceberg-python]

2025-02-24 Thread via GitHub
dependabot[bot] opened a new pull request, #1716: URL: https://github.com/apache/iceberg-python/pull/1716 Bumps [mypy-boto3-glue](https://github.com/youtype/mypy_boto3_builder) from 1.36.4 to 1.37.0. Release notes Sourced from https://github.com/youtype/mypy_boto3_builder/releases"

[PR] Build: Bump polars from 1.22.0 to 1.23.0 [iceberg-python]

2025-02-24 Thread via GitHub
dependabot[bot] opened a new pull request, #1715: URL: https://github.com/apache/iceberg-python/pull/1715 Bumps [polars](https://github.com/pola-rs/polars) from 1.22.0 to 1.23.0. Release notes Sourced from https://github.com/pola-rs/polars/releases";>polars's releases. Pytho

[PR] Build: Bump moto from 5.0.28 to 5.1.0 [iceberg-python]

2025-02-24 Thread via GitHub
dependabot[bot] opened a new pull request, #1713: URL: https://github.com/apache/iceberg-python/pull/1713 Bumps [moto](https://github.com/getmoto/moto) from 5.0.28 to 5.1.0. Changelog Sourced from https://github.com/getmoto/moto/blob/master/CHANGELOG.md";>moto's changelog. 5

[PR] Build: Bump mkdocs-autorefs from 1.3.1 to 1.4.0 [iceberg-python]

2025-02-24 Thread via GitHub
dependabot[bot] opened a new pull request, #1712: URL: https://github.com/apache/iceberg-python/pull/1712 Bumps [mkdocs-autorefs](https://github.com/mkdocstrings/autorefs) from 1.3.1 to 1.4.0. Release notes Sourced from https://github.com/mkdocstrings/autorefs/releases";>mkdocs-aut

Re: [PR] Spec: Add implementation note on `current-snapshot-id` [iceberg]

2025-02-24 Thread via GitHub
szehon-ho commented on code in PR #12334: URL: https://github.com/apache/iceberg/pull/12334#discussion_r1968535875 ## format/spec.md: ## @@ -1754,6 +1754,14 @@ Snapshot summary can include metrics fields to track numeric stats of the snapsh | **`engine-name`**| "spark"

Re: [PR] Spec: Add implementation note on `current-snapshot-id` [iceberg]

2025-02-24 Thread via GitHub
rdblue commented on code in PR #12334: URL: https://github.com/apache/iceberg/pull/12334#discussion_r1968529807 ## format/spec.md: ## @@ -1754,6 +1754,14 @@ Snapshot summary can include metrics fields to track numeric stats of the snapsh | **`engine-name`**| "spark"

Re: [I] don't show warning about bucket [iceberg-python]

2025-02-24 Thread via GitHub
djouallah commented on issue #1705: URL: https://github.com/apache/iceberg-python/issues/1705#issuecomment-2679835469 thank you guys, pyiceberg 0.9 is a great release already !!! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [PR] fix(manifest): ManifestEntry partition field schema should be dynamically generated [iceberg-go]

2025-02-24 Thread via GitHub
zeroshade merged PR #307: URL: https://github.com/apache/iceberg-go/pull/307 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] build(deps): bump the gomod_updates group with 16 updates [iceberg-go]

2025-02-24 Thread via GitHub
zeroshade merged PR #314: URL: https://github.com/apache/iceberg-go/pull/314 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Add example for document [iceberg-rust]

2025-02-24 Thread via GitHub
jonathanc-n commented on issue #986: URL: https://github.com/apache/iceberg-rust/issues/986#issuecomment-2679638759 Is the example [here](https://github.com/apache/iceberg-rust/blob/main/crates/integrations/datafusion/src/table/table_provider_factory.rs#L35) insufficient? what more do you t

Re: [I] Support for list types? [iceberg-go]

2025-02-24 Thread via GitHub
zeroshade commented on issue #309: URL: https://github.com/apache/iceberg-go/issues/309#issuecomment-2679587701 Yea i'm seeing the same warnings, i think it's related to https://github.com/apache/iceberg/pull/12264 and I might have to disable the strong integrity checksum -- This is an a

Re: [PR] feat: support update for memory catalog [iceberg-rust]

2025-02-24 Thread via GitHub
jonathanc-n commented on code in PR #1002: URL: https://github.com/apache/iceberg-rust/pull/1002#discussion_r1968342633 ## crates/catalog/memory/src/catalog.rs: ## @@ -53,6 +53,30 @@ impl MemoryCatalog { warehouse_location, } } + +fn new_metadata_l

Re: [PR] API: Define RepairManifests action interface [iceberg]

2025-02-24 Thread via GitHub
flyrain commented on PR #10784: URL: https://github.com/apache/iceberg/pull/10784#issuecomment-2679510929 @danielcweeks thanks for the confirmation. One of use cases is to check the integrity of a copied table from different places. The use cases are increasing as `rewriteTablePath` was out

Re: [PR] AWS: Integrate S3 analytics accelerator library [iceberg]

2025-02-24 Thread via GitHub
SanjayMarreddi commented on code in PR #12299: URL: https://github.com/apache/iceberg/pull/12299#discussion_r1968300773 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java: ## @@ -640,12 +663,21 @@ public S3FileIOProperties(Map properties) { prop

Re: [PR] AWS: Integrate S3 analytics accelerator library [iceberg]

2025-02-24 Thread via GitHub
SanjayMarreddi commented on code in PR #12299: URL: https://github.com/apache/iceberg/pull/12299#discussion_r1968199157 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3InputFile.java: ## @@ -82,6 +97,27 @@ public long getLength() { @Override public SeekableInputStream n

Re: [PR] Spec: Allow Equality Deletes with Row Lineage and Define Behavior [iceberg]

2025-02-24 Thread via GitHub
RussellSpitzer commented on PR #12230: URL: https://github.com/apache/iceberg/pull/12230#issuecomment-2679381223 Merged! Thanks everyone - @aokolnychyi , @rdblue , @huaxingao , @Fokko , @pvary , @stevenzwu , @singhpk234 for reviewing! -- This is an automated message from the Apache Git Se

Re: [PR] Spec: Allow Equality Deletes with Row Lineage and Define Behavior [iceberg]

2025-02-24 Thread via GitHub
RussellSpitzer merged PR #12230: URL: https://github.com/apache/iceberg/pull/12230 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

  1   2   3   >