Re: [PR] Fix predicates not matching the Arrow type of columns read from parquet files [iceberg-rust]

2025-05-11 Thread via GitHub
phillipleblanc commented on code in PR #1308: URL: https://github.com/apache/iceberg-rust/pull/1308#discussion_r2083959224 ## crates/iceberg/src/arrow/reader.rs: ## @@ -1103,6 +1104,7 @@ impl BoundPredicateVisitor for PredicateConverter<'_> { Ok(Box::new(move |bat

Re: [PR] Fix predicates not matching the Arrow type of columns read from parquet files [iceberg-rust]

2025-05-11 Thread via GitHub
sdd commented on code in PR #1308: URL: https://github.com/apache/iceberg-rust/pull/1308#discussion_r2083937157 ## crates/iceberg/src/arrow/reader.rs: ## @@ -1103,6 +1104,7 @@ impl BoundPredicateVisitor for PredicateConverter<'_> { Ok(Box::new(move |batch| {

Re: [PR] feat: add manifest related structures [iceberg-cpp]

2025-05-11 Thread via GitHub
wgtmac commented on code in PR #91: URL: https://github.com/apache/iceberg-cpp/pull/91#discussion_r2083942933 ## src/iceberg/manifest_entry.cc: ## @@ -0,0 +1,66 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See

Re: [PR] Support retrieving the latest Iceberg table on table scan [iceberg-rust]

2025-05-11 Thread via GitHub
sdd commented on code in PR #1297: URL: https://github.com/apache/iceberg-rust/pull/1297#discussion_r2083901259 ## crates/integrations/datafusion/src/table/mod.rs: ## @@ -38,55 +39,63 @@ use crate::physical_plan::scan::IcebergTableScan; #[derive(Debug, Clone)] pub struct Icebe

Re: [PR] Support retrieving the latest Iceberg table on table scan [iceberg-rust]

2025-05-11 Thread via GitHub
sdd commented on code in PR #1297: URL: https://github.com/apache/iceberg-rust/pull/1297#discussion_r2083901259 ## crates/integrations/datafusion/src/table/mod.rs: ## @@ -38,55 +39,63 @@ use crate::physical_plan::scan::IcebergTableScan; #[derive(Debug, Clone)] pub struct Icebe

Re: [PR] Support retrieving the latest Iceberg table on table scan [iceberg-rust]

2025-05-11 Thread via GitHub
sdd commented on code in PR #1297: URL: https://github.com/apache/iceberg-rust/pull/1297#discussion_r2083900311 ## crates/integrations/datafusion/src/table/mod.rs: ## @@ -38,55 +39,63 @@ use crate::physical_plan::scan::IcebergTableScan; #[derive(Debug, Clone)] pub struct Icebe

Re: [PR] Support retrieving the latest Iceberg table on table scan [iceberg-rust]

2025-05-11 Thread via GitHub
sdd commented on code in PR #1297: URL: https://github.com/apache/iceberg-rust/pull/1297#discussion_r2083898923 ## crates/integrations/datafusion/src/table/mod.rs: ## @@ -38,55 +39,63 @@ use crate::physical_plan::scan::IcebergTableScan; #[derive(Debug, Clone)] pub struct Icebe

Re: [PR] Build: Don't ignore changes to open-api/ in Spark CI [iceberg]

2025-05-11 Thread via GitHub
nastra commented on PR #13018: URL: https://github.com/apache/iceberg/pull/13018#issuecomment-2870971885 I'm not convinced that this is a good idea to always run Spark CI whenever something in open-api changes. We also do actually run the tests inside open-api, so those should be enough imo

Re: [I] Feature request: make `blob` struct public [iceberg-rust]

2025-05-11 Thread via GitHub
sdd commented on issue #1311: URL: https://github.com/apache/iceberg-rust/issues/1311#issuecomment-2870969029 I think the fact that it is `puffin::Blob`, is quite clear. In ambiguous contexts, someone could always just `use puffin::Blob as PuffinBlob` - we already have similar situations wi

Re: [PR] feat: Support `PartitionStats` [iceberg-rust]

2025-05-11 Thread via GitHub
sdd commented on code in PR #: URL: https://github.com/apache/iceberg-rust/pull/#discussion_r2083886173 ## crates/iceberg/src/spec/statistic_file.rs: ## @@ -69,6 +72,137 @@ pub struct PartitionStatisticsFile { pub file_size_in_bytes: i64, } +#[derive(Clone, Debug

Re: [PR] Flink 2.0: Remove the JUnit4 dependency [iceberg]

2025-05-11 Thread via GitHub
tomtongue commented on code in PR #13021: URL: https://github.com/apache/iceberg/pull/13021#discussion_r2083831358 ## flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/sink/TestIcebergSink.java: ## @@ -421,8 +421,10 @@ void testOperatorsUidNameWitUidSuffix() throws Excepti

Re: [PR] Flink 2.0: Remove the JUnit4 dependency [iceberg]

2025-05-11 Thread via GitHub
tomtongue commented on PR #13021: URL: https://github.com/apache/iceberg/pull/13021#issuecomment-2870850642 > @tomtongue > > Hello, Apologies for the repeated question. > > I successfully ran `./gradlew spotlessApply` and `./gradlew build -x test -x integrationTest`, and th

Re: [PR] feat: implement initial MemoryCatalog functionality with namespace and table support [iceberg-cpp]

2025-05-11 Thread via GitHub
wgtmac commented on code in PR #80: URL: https://github.com/apache/iceberg-cpp/pull/80#discussion_r2083769895 ## src/iceberg/catalog/memory_catalog.h: ## @@ -0,0 +1,204 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreemen

Re: [PR] SPARK: Remove dependency on hadoop's filesystem class from remove orphan files [iceberg]

2025-05-11 Thread via GitHub
liziyan-lzy commented on code in PR #12254: URL: https://github.com/apache/iceberg/pull/12254#discussion_r2083767905 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java: ## @@ -303,39 +310,88 @@ private Dataset listedFileDS() {

Re: [PR] feat: add visit type support [iceberg-cpp]

2025-05-11 Thread via GitHub
wgtmac commented on code in PR #94: URL: https://github.com/apache/iceberg-cpp/pull/94#discussion_r2083767328 ## src/iceberg/util/unreachable.h: ## @@ -0,0 +1,33 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. Se

Re: [PR] feat: add visit type support [iceberg-cpp]

2025-05-11 Thread via GitHub
zhjwpku commented on code in PR #94: URL: https://github.com/apache/iceberg-cpp/pull/94#discussion_r2083761948 ## src/iceberg/util/unreachable.h: ## @@ -0,0 +1,33 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. S

Re: [PR] feat: add visit type support [iceberg-cpp]

2025-05-11 Thread via GitHub
zhjwpku commented on code in PR #94: URL: https://github.com/apache/iceberg-cpp/pull/94#discussion_r2083762155 ## src/iceberg/util/checked_cast.h: ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements.

Re: [I] Error creating table from pyarrow schema with pa.uuid() [iceberg-python]

2025-05-11 Thread via GitHub
jim-ngoo commented on issue #1986: URL: https://github.com/apache/iceberg-python/issues/1986#issuecomment-2870545428 we have the [UUIDType](https://py.iceberg.apache.org/reference/pyiceberg/types/#pyiceberg.types.UUIDType) type already, I think what we missed is the `visit_pyarrow` decorat

Re: [PR] feat: add visit type support [iceberg-cpp]

2025-05-11 Thread via GitHub
wgtmac commented on code in PR #94: URL: https://github.com/apache/iceberg-cpp/pull/94#discussion_r2083718131 ## src/iceberg/util/checked_cast.h: ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. S

Re: [PR] chore(deps): Bump tokio from 1.44.2 to 1.45.0 [iceberg-rust]

2025-05-11 Thread via GitHub
liurenjie1024 merged PR #1312: URL: https://github.com/apache/iceberg-rust/pull/1312 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] chore: better error handling of nlohmann json lib apis [iceberg-cpp]

2025-05-11 Thread via GitHub
zhjwpku commented on code in PR #95: URL: https://github.com/apache/iceberg-cpp/pull/95#discussion_r2083671354 ## src/iceberg/json_internal.cc: ## @@ -1210,11 +1216,11 @@ Result> TableMetadataFromJson(const nlohmann::jso } Result FromJsonString(const std::string& json_strin

Re: [PR] chore: better error handling of nlohmann json lib apis [iceberg-cpp]

2025-05-11 Thread via GitHub
zhjwpku commented on code in PR #95: URL: https://github.com/apache/iceberg-cpp/pull/95#discussion_r2083671213 ## src/iceberg/json_internal.cc: ## @@ -173,12 +171,17 @@ void SetOptionalField(nlohmann::json& json, std::string_view key, } } +std::string DumpJsonNoExcept(con

Re: [PR] Spark: Introducing a New Hive to Iceberg Table Migration Method: In-Place Upgrade [iceberg]

2025-05-11 Thread via GitHub
github-actions[bot] commented on PR #12769: URL: https://github.com/apache/iceberg/pull/12769#issuecomment-2870385835 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Spark3.4: Enable Native execution if ParquetReaderType is Comet [iceberg]

2025-05-11 Thread via GitHub
github-actions[bot] commented on PR #12709: URL: https://github.com/apache/iceberg/pull/12709#issuecomment-2870385774 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] How to run streaming upserts and maintenance simultaneously? [iceberg]

2025-05-11 Thread via GitHub
github-actions[bot] commented on issue #11530: URL: https://github.com/apache/iceberg/issues/11530#issuecomment-2870385613 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] flink autoscaler: how set write-parallelism ? [iceberg]

2025-05-11 Thread via GitHub
github-actions[bot] commented on issue #10147: URL: https://github.com/apache/iceberg/issues/10147#issuecomment-2870385478 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] flink autoscaler: how set write-parallelism ? [iceberg]

2025-05-11 Thread via GitHub
github-actions[bot] closed issue #10147: flink autoscaler: how set write-parallelism ? URL: https://github.com/apache/iceberg/issues/10147 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] Core: ability to inject an AuthManager in RESTCatalog [iceberg]

2025-05-11 Thread via GitHub
github-actions[bot] closed pull request #12655: Core: ability to inject an AuthManager in RESTCatalog URL: https://github.com/apache/iceberg/pull/12655 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Core: ability to inject an AuthManager in RESTCatalog [iceberg]

2025-05-11 Thread via GitHub
github-actions[bot] commented on PR #12655: URL: https://github.com/apache/iceberg/pull/12655#issuecomment-2870385745 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Prototyping Spark 3.4 row lineage [iceberg]

2025-05-11 Thread via GitHub
github-actions[bot] commented on PR #12592: URL: https://github.com/apache/iceberg/pull/12592#issuecomment-2870385695 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Flink: add snapshot expiration reset strategy [iceberg]

2025-05-11 Thread via GitHub
github-actions[bot] commented on PR #12639: URL: https://github.com/apache/iceberg/pull/12639#issuecomment-2870385714 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] Prototyping Spark 3.4 row lineage [iceberg]

2025-05-11 Thread via GitHub
github-actions[bot] closed pull request #12592: Prototyping Spark 3.4 row lineage URL: https://github.com/apache/iceberg/pull/12592 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Core: Simplify AuthManager API [iceberg]

2025-05-11 Thread via GitHub
github-actions[bot] closed pull request #12555: Core: Simplify AuthManager API URL: https://github.com/apache/iceberg/pull/12555 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Core: Simplify AuthManager API [iceberg]

2025-05-11 Thread via GitHub
github-actions[bot] commented on PR #12555: URL: https://github.com/apache/iceberg/pull/12555#issuecomment-2870385672 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [I] ADLSFileIO cache DefaultAzureCredentials? [iceberg]

2025-05-11 Thread via GitHub
github-actions[bot] commented on issue #11523: URL: https://github.com/apache/iceberg/issues/11523#issuecomment-2870385595 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] feat: add manifest related structures [iceberg-cpp]

2025-05-11 Thread via GitHub
lidavidm commented on code in PR #91: URL: https://github.com/apache/iceberg-cpp/pull/91#discussion_r2083647799 ## src/iceberg/manifest_entry.cc: ## @@ -0,0 +1,66 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. S

Re: [PR] Flink 2.0: Remove the JUnit4 dependency [iceberg]

2025-05-11 Thread via GitHub
JeonDaehong commented on PR #13021: URL: https://github.com/apache/iceberg/pull/13021#issuecomment-2870265991 @tomtongue Hello, Apologies for the repeated question. I successfully ran `./gradlew spotlessApply` and `./gradlew build -x test -x integrationTest`, and then

[I] Dependency on pyiceberg_core while still marked as Optional [iceberg-python]

2025-05-11 Thread via GitHub
andersbogsnes opened a new issue, #1987: URL: https://github.com/apache/iceberg-python/issues/1987 ### Apache Iceberg version 0.9.0 (latest release) ### Please describe the bug 🐞 (I'm on 0.9.1 but the dropdown is missing that one) Hi, I was trying to add partition

Re: [PR] chore: better error handling of nlohmann json lib apis [iceberg-cpp]

2025-05-11 Thread via GitHub
wgtmac commented on code in PR #95: URL: https://github.com/apache/iceberg-cpp/pull/95#discussion_r2083558800 ## src/iceberg/json_internal.cc: ## @@ -1210,11 +1216,11 @@ Result> TableMetadataFromJson(const nlohmann::jso } Result FromJsonString(const std::string& json_string

Re: [PR] feat: add manifest related structures [iceberg-cpp]

2025-05-11 Thread via GitHub
wgtmac commented on code in PR #91: URL: https://github.com/apache/iceberg-cpp/pull/91#discussion_r2083555468 ## src/iceberg/manifest_entry.h: ## @@ -0,0 +1,256 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See

Re: [PR] Website: Add PyIceberg, IcebergRust, and IcebergGo to top nav bar [iceberg]

2025-05-11 Thread via GitHub
petern48 commented on PR #12950: URL: https://github.com/apache/iceberg/pull/12950#issuecomment-2869936552 I experimented with trying to achieve what the Spark docs do (as you requested [here](https://github.com/apache/iceberg/issues/12910#issuecomment-2855028749)) by nesting the docs insi

Re: [I] While inserting decimal: ValueError: Unexpected physical type FIXED_LEN_BYTE_ARRAY [iceberg-python]

2025-05-11 Thread via GitHub
kevinjqliu closed issue #1979: While inserting decimal: ValueError: Unexpected physical type FIXED_LEN_BYTE_ARRAY URL: https://github.com/apache/iceberg-python/issues/1979 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Write small decimals as INTs [iceberg-python]

2025-05-11 Thread via GitHub
kevinjqliu merged PR #1983: URL: https://github.com/apache/iceberg-python/pull/1983 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [PR] Add table property to disable/enable parquet column statistics #12770 [iceberg]

2025-05-11 Thread via GitHub
huaxiangsun commented on code in PR #12771: URL: https://github.com/apache/iceberg/pull/12771#discussion_r2083549046 ## parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java: ## @@ -401,6 +405,22 @@ public FileAppender build() throws IOException { Preconditions

Re: [PR] Add table property to disable/enable parquet column statistics #12770 [iceberg]

2025-05-11 Thread via GitHub
huaxiangsun commented on code in PR #12771: URL: https://github.com/apache/iceberg/pull/12771#discussion_r2083543648 ## parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java: ## @@ -401,6 +405,22 @@ public FileAppender build() throws IOException { Preconditions

Re: [PR] feat: add name mapping [iceberg-cpp]

2025-05-11 Thread via GitHub
wgtmac commented on code in PR #93: URL: https://github.com/apache/iceberg-cpp/pull/93#discussion_r2083538516 ## src/iceberg/name_mapping.cc: ## @@ -0,0 +1,267 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See

Re: [PR] chore: hms/glue catalog create table should respect default location [iceberg-rust]

2025-05-11 Thread via GitHub
sundy-li commented on PR #1302: URL: https://github.com/apache/iceberg-rust/pull/1302#issuecomment-2869892024 review cc @Xuanwo -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

[PR] chore(deps): Bump tokio from 1.44.2 to 1.45.0 [iceberg-rust]

2025-05-11 Thread via GitHub
dependabot[bot] opened a new pull request, #1312: URL: https://github.com/apache/iceberg-rust/pull/1312 Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.44.2 to 1.45.0. Release notes Sourced from https://github.com/tokio-rs/tokio/releases";>tokio's releases. Tokio v1.4

[PR] chore: better error handling of nlohmann json lib apis [iceberg-cpp]

2025-05-11 Thread via GitHub
zhjwpku opened a new pull request, #95: URL: https://github.com/apache/iceberg-cpp/pull/95 1. switch off exceptions for nlohmann::json::parse 2. add a no except wrapper around json.dump, since we don't handle exceptions in Error messages. This closes issue #87 References

Re: [PR] [WIP] Add precommit hook for linting and formatting [iceberg-rust]

2025-05-11 Thread via GitHub
dentiny closed pull request #1267: [WIP] Add precommit hook for linting and formatting URL: https://github.com/apache/iceberg-rust/pull/1267 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] feat: add base config implementation [iceberg-cpp]

2025-05-11 Thread via GitHub
zhjwpku commented on code in PR #92: URL: https://github.com/apache/iceberg-cpp/pull/92#discussion_r2083481119 ## test/config_test.cc: ## @@ -0,0 +1,110 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOT

[PR] build(deps): bump the gomod_updates group with 4 updates [iceberg-go]

2025-05-11 Thread via GitHub
dependabot[bot] opened a new pull request, #428: URL: https://github.com/apache/iceberg-go/pull/428 Bumps the gomod_updates group with 4 updates: [github.com/apache/arrow-go/v18](https://github.com/apache/arrow-go), [github.com/aws/aws-sdk-go-v2/service/glue](https://github.com/aws/aws-sdk-

Re: [PR] Spark4.0 rc5 integration with Iceberg [iceberg]

2025-05-11 Thread via GitHub
ajantha-bhat commented on code in PR #13022: URL: https://github.com/apache/iceberg/pull/13022#discussion_r2083437664 ## build.gradle: ## @@ -120,6 +120,9 @@ allprojects { repositories { mavenCentral() mavenLocal() +maven { + url "https://repository.apache

Re: [I] Consider Using object_store as IO Abstraction [iceberg-rust]

2025-05-11 Thread via GitHub
linhr commented on issue #172: URL: https://github.com/apache/iceberg-rust/issues/172#issuecomment-2869546190 Having object_store as an OpenDAL layer seems a good idea! Besides supporting object_store, it seems `FileIO` currently only supports a few specific OpenDAL layers listed as `