Re: [PR] feat: Introduce C FFI for iceberg rust [iceberg-rust]

2025-02-13 Thread via GitHub
liurenjie1024 commented on code in PR #966: URL: https://github.com/apache/iceberg-rust/pull/966#discussion_r1953933561 ## bindings/python/Cargo.lock: ## Review Comment: This seems unrelated? ## bindings/c/CMakeLists.txt: ## Review Comment: I'm not c

Re: [I] TestS3FileIO fails locally [iceberg]

2025-02-13 Thread via GitHub
Fokko commented on issue #12237: URL: https://github.com/apache/iceberg/issues/12237#issuecomment-2655813668 I'm also on orbstack, and works fine on my end as well: ``` ./gradlew :iceberg-aws:test Configuration on demand is an incubating feature. > Task :iceberg-api:testJ

Re: [PR] Docs: Add 1.8.0 javadocs [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar merged PR #12246: URL: https://github.com/apache/iceberg/pull/12246 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Docs: Add 1.8.0 javadocs [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar commented on PR #12246: URL: https://github.com/apache/iceberg/pull/12246#issuecomment-2655820101 Thanks @nastra ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] Docs: Site update for 1.8.0 release [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar commented on PR #12242: URL: https://github.com/apache/iceberg/pull/12242#issuecomment-2655820855 Thank you @nastra! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Docs: Site update for 1.8.0 release [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar merged PR #12242: URL: https://github.com/apache/iceberg/pull/12242 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [I] Support commit retrie [iceberg-rust]

2025-02-13 Thread via GitHub
liurenjie1024 commented on issue #964: URL: https://github.com/apache/iceberg-rust/issues/964#issuecomment-2655890945 > For now, we only support FastAppend. So we can complete the whole process based on the FastAppend first and complete conflict detection when we introduce other update acti

Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]

2025-02-13 Thread via GitHub
joshua-yeung-mox commented on issue #10524: URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2655917729 > Finally, I noticed the problem was the session catalog (the default one) was not configured to connect to glue catalog. So it was just an empty local "default" catalog.

Re: [PR] Add support for `write.metadata.path` [iceberg-python]

2025-02-13 Thread via GitHub
kevinjqliu commented on code in PR #1642: URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1955395642 ## pyiceberg/table/locations.py: ## @@ -64,6 +71,35 @@ def new_data_location(self, data_file_name: str, partition_key: Optional[Partiti str: A f

Re: [PR] Add support for `write.metadata.path` [iceberg-python]

2025-02-13 Thread via GitHub
kevinjqliu commented on code in PR #1642: URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1955380186 ## pyiceberg/table/locations.py: ## @@ -64,6 +71,35 @@ def new_data_location(self, data_file_name: str, partition_key: Optional[Partiti str: A f

Re: [PR] feat: support merge append action [iceberg-rust]

2025-02-13 Thread via GitHub
jonathanc-n commented on code in PR #902: URL: https://github.com/apache/iceberg-rust/pull/902#discussion_r1955391542 ## crates/integration_tests/tests/merge_append_test.rs: ## @@ -0,0 +1,183 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contribut

Re: [PR] add Status data structure [iceberg-cpp]

2025-02-13 Thread via GitHub
zhjwpku commented on PR #8: URL: https://github.com/apache/iceberg-cpp/pull/8#issuecomment-2658229119 This is obsoleted by #40, though we haven't decided to adopt expected or not, this PR can be closed. Thanks for all the inputs. -- This is an automated message from the Apache Git Service

Re: [PR] add Status data structure [iceberg-cpp]

2025-02-13 Thread via GitHub
zhjwpku closed pull request #8: add Status data structure URL: https://github.com/apache/iceberg-cpp/pull/8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-ma

Re: [PR] Spark: Remove unused PruneColumnsWithReordering class [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar commented on PR #12258: URL: https://github.com/apache/iceberg/pull/12258#issuecomment-2658231400 I'll go ahead and merge, thanks @Fokko @dramaticlly for reviewing! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [PR] Core: Fix divide by zero when adjust split size [iceberg]

2025-02-13 Thread via GitHub
hantangwangd commented on PR #12201: URL: https://github.com/apache/iceberg/pull/12201#issuecomment-2657960091 @RussellSpitzer my pleasure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Add table upsert support [iceberg-python]

2025-02-13 Thread via GitHub
kevinjqliu commented on PR #1660: URL: https://github.com/apache/iceberg-python/pull/1660#issuecomment-2657959865 Thanks everyone for getting this over the finish line! Upsert has been a long awaited feature. I'm excited to include this as part of the upcoming 0.9.0 release. This is

Re: [PR] Spark: Remove unused PruneColumnsWithReordering class [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar merged PR #12258: URL: https://github.com/apache/iceberg/pull/12258 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] S3: Disable strong integrity checksums [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar commented on code in PR #12264: URL: https://github.com/apache/iceberg/pull/12264#discussion_r1955595821 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3RequestUtil.java: ## @@ -149,4 +151,10 @@ static void configurePermission( Function aclSetter) {

Re: [PR] S3: Disable strong integrity checksums [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar commented on code in PR #12264: URL: https://github.com/apache/iceberg/pull/12264#discussion_r1955595821 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3RequestUtil.java: ## @@ -149,4 +151,10 @@ static void configurePermission( Function aclSetter) {

Re: [PR] Spark: Structured Streaming read limit support follow-up [iceberg]

2025-02-13 Thread via GitHub
singhpk234 commented on code in PR #12260: URL: https://github.com/apache/iceberg/pull/12260#discussion_r1955584774 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java: ## @@ -309,6 +312,49 @@ private static StreamingOffset determineStar

Re: [PR] S3: Disable strong integrity checksums [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar commented on code in PR #12264: URL: https://github.com/apache/iceberg/pull/12264#discussion_r1955600722 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3RequestUtil.java: ## @@ -149,4 +151,10 @@ static void configurePermission( Function aclSetter) {

Re: [I] Spark streaming (merge into) iceberg table concurrent write with compaction job [iceberg]

2025-02-13 Thread via GitHub
singhpk234 commented on issue #12187: URL: https://github.com/apache/iceberg/issues/12187#issuecomment-2658363561 We should look into the combinations of A2 (remove old snapshot, rewrite manifest , compaction (binpack and sometimes z-order), rewrite position delete files, delete orphan file

Re: [PR] Implementation of version metadata table for view [iceberg]

2025-02-13 Thread via GitHub
huan233usc commented on PR #12014: URL: https://github.com/apache/iceberg/pull/12014#issuecomment-2658363474 Thank you both for the review, will update this pr accordingly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] backport #11301(rowconverter) to Flink 1.19 and 1.18 [iceberg]

2025-02-13 Thread via GitHub
pvary commented on PR #11826: URL: https://github.com/apache/iceberg/pull/11826#issuecomment-2658364228 What is the state of this backport @abharath9? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Core: add variant type support [iceberg]

2025-02-13 Thread via GitHub
aihuaxu commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r192361 ## core/src/test/java/org/apache/iceberg/avro/TestSchemaConversions.java: ## @@ -370,4 +370,22 @@ public void testFieldDocsArePreserved() { Lists.newArrayLis

[I] Link LEARN MORE vom https://iceberg.apache.org/about/ runs into Not Found [iceberg]

2025-02-13 Thread via GitHub
alexandertodorovic opened a new issue, #12265: URL: https://github.com/apache/iceberg/issues/12265 Dear All, The LEARN MORE button on the [Apache Iceberg About](https://iceberg.apache.org/about/) page leads to a "Not Found" error: _Not Found The requested URL was not found o

[PR] Docker: Pin QEMU version temporarily [iceberg]

2025-02-13 Thread via GitHub
ebyhr opened a new pull request, #12262: URL: https://github.com/apache/iceberg/pull/12262 Fix failing job: https://github.com/apache/iceberg/actions/workflows/publish-iceberg-rest-fixture-docker.yml -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [PR] Add support for `write.metadata.path` [iceberg-python]

2025-02-13 Thread via GitHub
kevinjqliu commented on code in PR #1642: URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1955383770 ## pyiceberg/table/locations.py: ## @@ -64,6 +71,35 @@ def new_data_location(self, data_file_name: str, partition_key: Optional[Partiti str: A f

Re: [PR] Spark: Structured Streaming read limit support follow-up [iceberg]

2025-02-13 Thread via GitHub
wypoon commented on code in PR #12260: URL: https://github.com/apache/iceberg/pull/12260#discussion_r1955405894 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreamingRead3.java: ## @@ -154,8 +154,7 @@ public void testReadStreamOnIcebergTableWi

Re: [PR] S3: Disable strong integrity checksums [iceberg]

2025-02-13 Thread via GitHub
ebyhr commented on code in PR #12264: URL: https://github.com/apache/iceberg/pull/12264#discussion_r1955642546 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3RequestUtil.java: ## @@ -149,4 +151,10 @@ static void configurePermission( Function aclSetter) { aclSetter.

Re: [PR] S3: Disable strong integrity checksums [iceberg]

2025-02-13 Thread via GitHub
ebyhr commented on code in PR #12264: URL: https://github.com/apache/iceberg/pull/12264#discussion_r1955642546 ## aws/src/main/java/org/apache/iceberg/aws/s3/S3RequestUtil.java: ## @@ -149,4 +151,10 @@ static void configurePermission( Function aclSetter) { aclSetter.

Re: [PR] spec: Remove `source-ids` for `V{1,2}` tables [iceberg]

2025-02-13 Thread via GitHub
Fokko merged PR #12161: URL: https://github.com/apache/iceberg/pull/12161 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] feat(datafusion): Treat timestamp conversion functions like a cast. [iceberg-rust]

2025-02-13 Thread via GitHub
liurenjie1024 commented on code in PR #945: URL: https://github.com/apache/iceberg-rust/pull/945#discussion_r1954046266 ## crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs: ## @@ -119,7 +122,53 @@ fn to_iceberg_predicate(expr: &Expr) -> TransformedResult {

Re: [PR] feat(datafusion): Treat timestamp conversion functions like a cast. [iceberg-rust]

2025-02-13 Thread via GitHub
liurenjie1024 commented on code in PR #945: URL: https://github.com/apache/iceberg-rust/pull/945#discussion_r1954049149 ## crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs: ## @@ -119,7 +122,53 @@ fn to_iceberg_predicate(expr: &Expr) -> TransformedResult {

Re: [PR] feat: Add existing parquet files [iceberg-rust]

2025-02-13 Thread via GitHub
liurenjie1024 commented on PR #960: URL: https://github.com/apache/iceberg-rust/pull/960#issuecomment-2655880166 > So in summary, for this feature, what we would like to provide is a parquet_files_to_data_files in the arrow module(or create a parquet module). The parquet_files_to_data_files

Re: [PR] Core: Adjust Jackson settings to handle large metadata json [iceberg]

2025-02-13 Thread via GitHub
nastra merged PR #12224: URL: https://github.com/apache/iceberg/pull/12224 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Build: Bump Hive to 2.3.10 [iceberg]

2025-02-13 Thread via GitHub
nastra merged PR #12253: URL: https://github.com/apache/iceberg/pull/12253 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Run dependabot daily [iceberg]

2025-02-13 Thread via GitHub
nastra commented on code in PR #12184: URL: https://github.com/apache/iceberg/pull/12184#discussion_r1954429091 ## .github/dependabot.yml: ## @@ -22,25 +22,22 @@ updates: - package-ecosystem: "github-actions" directory: "/" schedule: - interval: "weekly" -

Re: [PR] feat: Add `StrictMetricsEvaluator` [iceberg-rust]

2025-02-13 Thread via GitHub
Fokko commented on code in PR #963: URL: https://github.com/apache/iceberg-rust/pull/963#discussion_r1954322029 ## crates/iceberg/src/expr/visitors/strict_metrics_evaluator.rs: ## @@ -0,0 +1,1854 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contr

Re: [PR] feat: Add existing parquet files [iceberg-rust]

2025-02-13 Thread via GitHub
Fokko commented on code in PR #960: URL: https://github.com/apache/iceberg-rust/pull/960#discussion_r1954451652 ## crates/iceberg/src/transaction.rs: ## @@ -169,6 +175,172 @@ impl<'a> Transaction<'a> { catalog.update_table(table_commit).await } + +/// Adds ex

Re: [PR] Fix: `SqlCatalog` list_namespaces() should return only sub-namespaces [iceberg-python]

2025-02-13 Thread via GitHub
alessandro-nori commented on code in PR #1629: URL: https://github.com/apache/iceberg-python/pull/1629#discussion_r1954139234 ## pyiceberg/catalog/sql.py: ## @@ -610,15 +610,26 @@ def list_namespaces(self, namespace: Union[str, Identifier] = ()) -> List[Identi table_st

Re: [PR] Fix: `SqlCatalog` list_namespaces() should return only sub-namespaces [iceberg-python]

2025-02-13 Thread via GitHub
alessandro-nori commented on code in PR #1629: URL: https://github.com/apache/iceberg-python/pull/1629#discussion_r1954145359 ## tests/catalog/test_sql.py: ## @@ -1158,13 +1171,13 @@ def test_list_non_existing_namespaces(catalog: SqlCatalog) -> None: def test_drop_namespace(ca

Re: [PR] feat: Add existing parquet files [iceberg-rust]

2025-02-13 Thread via GitHub
Fokko commented on code in PR #960: URL: https://github.com/apache/iceberg-rust/pull/960#discussion_r1954458103 ## crates/iceberg/src/transaction.rs: ## @@ -169,6 +175,172 @@ impl<'a> Transaction<'a> { catalog.update_table(table_commit).await } + +/// Adds ex

Re: [PR] feat: Add existing parquet files [iceberg-rust]

2025-02-13 Thread via GitHub
Fokko commented on code in PR #960: URL: https://github.com/apache/iceberg-rust/pull/960#discussion_r1954470977 ## crates/iceberg/src/transaction.rs: ## @@ -169,6 +175,172 @@ impl<'a> Transaction<'a> { catalog.update_table(table_commit).await } + +/// Adds ex

Re: [PR] Spec additions for encryption [iceberg]

2025-02-13 Thread via GitHub
ggershinsky commented on code in PR #12162: URL: https://github.com/apache/iceberg/pull/12162#discussion_r1954484869 ## format/spec.md: ## @@ -685,6 +686,7 @@ A snapshot consists of the following fields: | _optional_ | _optional_ | _optional_ | **`schema-id`** | ID

Re: [PR] [WIP] Ignore UnknownType in General Parquet Writer [iceberg]

2025-02-13 Thread via GitHub
Fokko commented on code in PR #12177: URL: https://github.com/apache/iceberg/pull/12177#discussion_r1954508077 ## parquet/src/main/java/org/apache/iceberg/parquet/TypeToMessageType.java: ## @@ -56,6 +56,10 @@ public class TypeToMessageType { LogicalTypeAnnotation.timestam

Re: [PR] Spark: Fix assertion checks [iceberg]

2025-02-13 Thread via GitHub
nastra commented on code in PR #12255: URL: https://github.com/apache/iceberg/pull/12255#discussion_r1954587640 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewritePositionDeleteFilesAction.java: ## @@ -1075,49 +1075,49 @@ private void checkResult(

Re: [PR] Build: Clean up dependencies [iceberg]

2025-02-13 Thread via GitHub
Fokko merged PR #12252: URL: https://github.com/apache/iceberg/pull/12252 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Core,Api: Add overwrite option when register external table to catalog [iceberg]

2025-02-13 Thread via GitHub
gaborkaszab commented on code in PR #12228: URL: https://github.com/apache/iceberg/pull/12228#discussion_r1954442294 ## api/src/main/java/org/apache/iceberg/catalog/Catalog.java: ## @@ -344,7 +344,23 @@ default void invalidateTable(TableIdentifier identifier) {} * @throws Al

[PR] [WIP] CORE: Remove dependency on hadoop's filesystem class from remove orphan files [iceberg]

2025-02-13 Thread via GitHub
liziyan-lzy opened a new pull request, #12254: URL: https://github.com/apache/iceberg/pull/12254 For: #11541 Not ready for review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Add support for `write.metadata.path` [iceberg-python]

2025-02-13 Thread via GitHub
smaheshwar-pltr commented on code in PR #1642: URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1954666388 ## pyiceberg/table/locations.py: ## @@ -64,6 +71,35 @@ def new_data_location(self, data_file_name: str, partition_key: Optional[Partiti str

Re: [PR] Add support for `write.metadata.path` [iceberg-python]

2025-02-13 Thread via GitHub
smaheshwar-pltr commented on code in PR #1642: URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1954666388 ## pyiceberg/table/locations.py: ## @@ -64,6 +71,35 @@ def new_data_location(self, data_file_name: str, partition_key: Optional[Partiti str

Re: [PR] Add support for `write.metadata.path` [iceberg-python]

2025-02-13 Thread via GitHub
smaheshwar-pltr commented on code in PR #1642: URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1954666388 ## pyiceberg/table/locations.py: ## @@ -64,6 +71,35 @@ def new_data_location(self, data_file_name: str, partition_key: Optional[Partiti str

Re: [PR] feat(datafusion): Treat timestamp conversion functions like a cast. [iceberg-rust]

2025-02-13 Thread via GitHub
omerhadari commented on code in PR #945: URL: https://github.com/apache/iceberg-rust/pull/945#discussion_r1954103936 ## crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs: ## @@ -403,4 +452,55 @@ mod tests { Reference::new("ts").greater_than_or_e

Re: [I] How to query NON Iceberg format tables (created in aws Athena) from spark? [iceberg]

2025-02-13 Thread via GitHub
salah-djb commented on issue #10524: URL: https://github.com/apache/iceberg/issues/10524#issuecomment-2655977524 Hi guys, To read hive tables from glue catalog you have to patch/build the catalog client like [explained here](https://github.com/awslabs/aws-glue-data-catalog-client-for-ap

[I] Add properties support for HadoopTables.load() [iceberg]

2025-02-13 Thread via GitHub
qqchang2nd opened a new issue, #12251: URL: https://github.com/apache/iceberg/issues/12251 ### Feature Request / Improvement Description: Currently, HadoopTables.load() doesn't support passing custom properties when loading tables. While HiveCatalog and HadoopCatalog support manife

Re: [PR] Update release version to 1.8.0 in doap.rdf [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar merged PR #12247: URL: https://github.com/apache/iceberg/pull/12247 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] feat(datafusion): Treat timestamp conversion functions like a cast. [iceberg-rust]

2025-02-13 Thread via GitHub
omerhadari commented on code in PR #945: URL: https://github.com/apache/iceberg-rust/pull/945#discussion_r1954101136 ## crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs: ## @@ -119,7 +122,53 @@ fn to_iceberg_predicate(expr: &Expr) -> TransformedResult {

Re: [PR] Build: Update revAPI to compare against 1.8.0 [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar merged PR #12244: URL: https://github.com/apache/iceberg/pull/12244 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Infra: Update Iceberg bug report template for 1.8.0 [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar commented on PR #12248: URL: https://github.com/apache/iceberg/pull/12248#issuecomment-2655950784 Thanks @nastra! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] feat: add apply in transaction to support stack action [iceberg-rust]

2025-02-13 Thread via GitHub
ZENOTME commented on PR #949: URL: https://github.com/apache/iceberg-rust/pull/949#issuecomment-2655981130 Hi @liurenjie1024, I think we should resolve this PR first before working on #964. It will affect the interface in transactions. -- This is an automated message from the Apache

Re: [PR] Docs: Fix formatting of 1.8.0 release notes [iceberg]

2025-02-13 Thread via GitHub
nastra merged PR #12249: URL: https://github.com/apache/iceberg/pull/12249 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Update release version to 1.8.0 in doap.rdf [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar commented on PR #12247: URL: https://github.com/apache/iceberg/pull/12247#issuecomment-2655949773 Thanks @nastra ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] feat(datafusion): Treat timestamp conversion functions like a cast. [iceberg-rust]

2025-02-13 Thread via GitHub
omerhadari commented on code in PR #945: URL: https://github.com/apache/iceberg-rust/pull/945#discussion_r1954099783 ## crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs: ## @@ -119,7 +122,53 @@ fn to_iceberg_predicate(expr: &Expr) -> TransformedResult {

Re: [PR] Infra: Update Iceberg bug report template for 1.8.0 [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar merged PR #12248: URL: https://github.com/apache/iceberg/pull/12248 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [I] Support commit retrie [iceberg-rust]

2025-02-13 Thread via GitHub
ZENOTME commented on issue #964: URL: https://github.com/apache/iceberg-rust/issues/964#issuecomment-2655980835 > > For now, we only support FastAppend. So we can complete the whole process based on the FastAppend first and complete conflict detection when we introduce other update actions.

[PR] Docs: Fix formatting of 1.8.0 release notes [iceberg]

2025-02-13 Thread via GitHub
amogh-jahagirdar opened a new pull request, #12249: URL: https://github.com/apache/iceberg/pull/12249 Fix formatting for 1.8.0 release notes, was missing a newline and it messed up the list structure below. Before: https://github.com/user-attachments/assets/00d5f9f8-eebf-417a-a48f-

Re: [PR] Fix: `SqlCatalog` list_namespaces() should return only sub-namespaces [iceberg-python]

2025-02-13 Thread via GitHub
alessandro-nori commented on code in PR #1629: URL: https://github.com/apache/iceberg-python/pull/1629#discussion_r1954151981 ## pyiceberg/catalog/sql.py: ## @@ -610,15 +610,26 @@ def list_namespaces(self, namespace: Union[str, Identifier] = ()) -> List[Identi table_st

Re: [PR] feat: Introduce C FFI for iceberg rust [iceberg-rust]

2025-02-13 Thread via GitHub
Xuanwo commented on code in PR #966: URL: https://github.com/apache/iceberg-rust/pull/966#discussion_r1954355077 ## bindings/c/CMakeLists.txt: ## @@ -0,0 +1,77 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NO

Re: [PR] feat: Introduce C FFI for iceberg rust [iceberg-rust]

2025-02-13 Thread via GitHub
Xuanwo commented on code in PR #966: URL: https://github.com/apache/iceberg-rust/pull/966#discussion_r1954356055 ## bindings/c/CMakeLists.txt: ## Review Comment: `CMakeLists` will generate Makefile and served for us to build `libiceberg`. -- This is an automated message

Re: [PR] feat: Add `StrictMetricsEvaluator` [iceberg-rust]

2025-02-13 Thread via GitHub
jonathanc-n commented on code in PR #963: URL: https://github.com/apache/iceberg-rust/pull/963#discussion_r1955029747 ## crates/iceberg/src/expr/visitors/strict_metrics_evaluator.rs: ## @@ -0,0 +1,1854 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] Core: BugFix: PartitionStatsUtil#computeStats returns incomplete stats in case of partition evolution [iceberg]

2025-02-13 Thread via GitHub
deniskuzZ commented on code in PR #12137: URL: https://github.com/apache/iceberg/pull/12137#discussion_r1955044758 ## core/src/test/java/org/apache/iceberg/TestPartitionStatsUtil.java: ## @@ -370,16 +371,167 @@ public void testPartitionStatsWithSchemaEvolution() throws Exceptio

Re: [PR] Add Variant custom logical type for Avro [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #12238: URL: https://github.com/apache/iceberg/pull/12238#discussion_r1955086053 ## core/src/main/java/org/apache/iceberg/avro/AvroCustomOrderSchemaVisitor.java: ## @@ -37,6 +37,10 @@ public static T visit(Schema schema, AvroCustomOrderSchemaVisi

Re: [PR] Add Variant custom logical type for Avro [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #12238: URL: https://github.com/apache/iceberg/pull/12238#discussion_r1955080951 ## core/src/test/java/org/apache/iceberg/avro/TestSchemaConversions.java: ## @@ -76,7 +78,8 @@ public void testPrimitiveTypes() { Schema.createFixed("fixe

Re: [PR] Core: BugFix: PartitionStatsUtil#computeStats returns incomplete stats in case of partition evolution [iceberg]

2025-02-13 Thread via GitHub
deniskuzZ commented on code in PR #12137: URL: https://github.com/apache/iceberg/pull/12137#discussion_r1955067425 ## core/src/test/java/org/apache/iceberg/TestPartitionStatsUtil.java: ## @@ -370,16 +371,167 @@ public void testPartitionStatsWithSchemaEvolution() throws Exceptio

Re: [PR] Core: BugFix: PartitionStatsUtil#computeStats returns incomplete stats in case of partition evolution [iceberg]

2025-02-13 Thread via GitHub
deniskuzZ commented on code in PR #12137: URL: https://github.com/apache/iceberg/pull/12137#discussion_r1955044758 ## core/src/test/java/org/apache/iceberg/TestPartitionStatsUtil.java: ## @@ -370,16 +371,167 @@ public void testPartitionStatsWithSchemaEvolution() throws Exceptio

Re: [PR] Core: BugFix: PartitionStatsUtil#computeStats returns incomplete stats in case of partition evolution [iceberg]

2025-02-13 Thread via GitHub
deniskuzZ commented on code in PR #12137: URL: https://github.com/apache/iceberg/pull/12137#discussion_r1955044758 ## core/src/test/java/org/apache/iceberg/TestPartitionStatsUtil.java: ## @@ -370,16 +371,167 @@ public void testPartitionStatsWithSchemaEvolution() throws Exceptio

Re: [PR] Add table upsert support [iceberg-python]

2025-02-13 Thread via GitHub
mattmartin14 commented on PR #1660: URL: https://github.com/apache/iceberg-python/pull/1660#issuecomment-2657519239 > cc reviewers from the other PR (@Fokko / @corleyma / @tscottcoombes1 / @marcoaanogueira) and the original PR author @mattmartin14 Looks great to me. Thanks for getting

Re: [PR] feat: Add `StrictMetricsEvaluator` [iceberg-rust]

2025-02-13 Thread via GitHub
jonathanc-n commented on code in PR #963: URL: https://github.com/apache/iceberg-rust/pull/963#discussion_r1955094766 ## crates/iceberg/src/expr/visitors/strict_metrics_evaluator.rs: ## @@ -0,0 +1,1854 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] API, Core: Support default values in UpdateSchema [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #12211: URL: https://github.com/apache/iceberg/pull/12211#discussion_r1954993890 ## api/src/main/java/org/apache/iceberg/UpdateSchema.java: ## @@ -67,24 +70,52 @@ default UpdateSchema addColumn(String name, Type type) { } /** - * Add a ne

Re: [PR] Add file-io specific documentation [iceberg-python]

2025-02-13 Thread via GitHub
summermousa-vendia commented on code in PR #1635: URL: https://github.com/apache/iceberg-python/pull/1635#discussion_r1954849491 ## mkdocs/docs/configuration.md: ## @@ -287,6 +287,55 @@ class UUIDLocationProvider(LocationProvider): return f"{prefix}/{partition_key.to_pa

Re: [PR] Spark: Fix assertion checks [iceberg]

2025-02-13 Thread via GitHub
RussellSpitzer commented on code in PR #12255: URL: https://github.com/apache/iceberg/pull/12255#discussion_r1954956756 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewritePositionDeleteFilesAction.java: ## @@ -1075,49 +1075,49 @@ private void checkResu

Re: [PR] Core: Fix divide by zero when adjust split size [iceberg]

2025-02-13 Thread via GitHub
hantangwangd commented on code in PR #12201: URL: https://github.com/apache/iceberg/pull/12201#discussion_r1955038689 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestSelect.java: ## @@ -57,7 +60,10 @@ protected static Object[][] parameters() { {

Re: [PR] Core: Fix divide by zero when adjust split size [iceberg]

2025-02-13 Thread via GitHub
hantangwangd commented on code in PR #12201: URL: https://github.com/apache/iceberg/pull/12201#discussion_r1955038335 ## core/src/main/java/org/apache/iceberg/util/TableScanUtil.java: ## @@ -233,6 +233,8 @@ public static List mergeTasks(List tasks) { } public static lo

Re: [PR] Implement update for `remove-snapshot-ref` action [iceberg-python]

2025-02-13 Thread via GitHub
grihabor commented on PR #1598: URL: https://github.com/apache/iceberg-python/pull/1598#issuecomment-2657770025 @kevinjqliu @Fokko Shall we merge this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Core: add variant type support [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1955154197 ## api/src/test/java/org/apache/iceberg/types/TestTypeUtil.java: ## @@ -645,4 +651,70 @@ public void testReassignOrRefreshIdsCaseInsensitive() { requi

Re: [PR] Core: add variant type support [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1955156020 ## core/src/main/java/org/apache/iceberg/schema/SchemaWithPartnerVisitor.java: ## @@ -106,7 +106,8 @@ public static T visit( } return visitor.map(

Re: [PR] Core: add variant type support [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1955155426 ## core/src/main/java/org/apache/iceberg/schema/SchemaWithPartnerVisitor.java: ## @@ -106,7 +106,8 @@ public static T visit( } return visitor.map(

Re: [PR] Core: add variant type support [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1955149743 ## api/src/test/java/org/apache/iceberg/types/TestTypeUtil.java: ## @@ -645,4 +651,70 @@ public void testReassignOrRefreshIdsCaseInsensitive() { requi

Re: [PR] feat: Add existing parquet files [iceberg-rust]

2025-02-13 Thread via GitHub
jonathanc-n commented on code in PR #960: URL: https://github.com/apache/iceberg-rust/pull/960#discussion_r1955169524 ## crates/iceberg/src/transaction.rs: ## @@ -169,6 +175,172 @@ impl<'a> Transaction<'a> { catalog.update_table(table_commit).await } + +/// A

Re: [PR] Add table upsert support [iceberg-python]

2025-02-13 Thread via GitHub
bitsondatadev commented on PR #1660: URL: https://github.com/apache/iceberg-python/pull/1660#issuecomment-2657648101 Hey @kevinjqliu, just a note after all the changes are done, it may be best to manually squash the commits and make sure Matt is the author for that...not sure what GitHub wi

Re: [PR] Core: add variant type support [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #11831: URL: https://github.com/apache/iceberg/pull/11831#discussion_r1955170959 ## core/src/test/java/org/apache/iceberg/avro/TestSchemaConversions.java: ## @@ -370,4 +370,22 @@ public void testFieldDocsArePreserved() { Lists.newArrayList

Re: [PR] Spark: Rewrite V2 deletes to V3 DVs [iceberg]

2025-02-13 Thread via GitHub
danielcweeks commented on code in PR #12250: URL: https://github.com/apache/iceberg/pull/12250#discussion_r1954910410 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeletesRewrite.java: ## @@ -213,45 +217,61 @@ static class PositionDeletesWriterFa

Re: [I] Add Zordering to table specification [iceberg]

2025-02-13 Thread via GitHub
RussellSpitzer commented on issue #12198: URL: https://github.com/apache/iceberg/issues/12198#issuecomment-2657192896 We couldn't do this previously because we didn't have multi arg transforms, but we could theoretically do this in V3 -- This is an automated message from the Apache Git Se

Re: [PR] [WIP] Ignore UnknownType in General Parquet Writer [iceberg]

2025-02-13 Thread via GitHub
Fokko commented on code in PR #12177: URL: https://github.com/apache/iceberg/pull/12177#discussion_r1954923019 ## parquet/src/main/java/org/apache/iceberg/parquet/TypeToMessageType.java: ## @@ -56,6 +56,10 @@ public class TypeToMessageType { LogicalTypeAnnotation.timestam

Re: [PR] Spark: Rewrite V2 deletes to V3 DVs [iceberg]

2025-02-13 Thread via GitHub
danielcweeks commented on code in PR #12250: URL: https://github.com/apache/iceberg/pull/12250#discussion_r1954884816 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewritePositionDeleteFilesSparkAction.java: ## @@ -404,8 +408,31 @@ private void validateAndIn

Re: [PR] Add support for `write.metadata.path` [iceberg-python]

2025-02-13 Thread via GitHub
kevinjqliu commented on code in PR #1642: URL: https://github.com/apache/iceberg-python/pull/1642#discussion_r1954915956 ## pyiceberg/table/locations.py: ## @@ -64,6 +71,35 @@ def new_data_location(self, data_file_name: str, partition_key: Optional[Partiti str: A f

Re: [PR] Core: BugFix: PartitionStatsUtil#computeStats returns incomplete stats in case of partition evolution [iceberg]

2025-02-13 Thread via GitHub
deniskuzZ commented on code in PR #12137: URL: https://github.com/apache/iceberg/pull/12137#discussion_r1955044758 ## core/src/test/java/org/apache/iceberg/TestPartitionStatsUtil.java: ## @@ -370,16 +371,167 @@ public void testPartitionStatsWithSchemaEvolution() throws Exceptio

Re: [PR] Add Variant custom logical type for Avro [iceberg]

2025-02-13 Thread via GitHub
rdblue commented on code in PR #12238: URL: https://github.com/apache/iceberg/pull/12238#discussion_r1955075130 ## core/src/test/java/org/apache/iceberg/avro/TestAvroSchemaProjection.java: ## @@ -150,4 +150,58 @@ public void projectWithMapSchemaChanged() { .as("Result o

Re: [PR] Core: BugFix: PartitionStatsUtil#computeStats returns incomplete stats in case of partition evolution [iceberg]

2025-02-13 Thread via GitHub
deniskuzZ commented on code in PR #12137: URL: https://github.com/apache/iceberg/pull/12137#discussion_r1955044758 ## core/src/test/java/org/apache/iceberg/TestPartitionStatsUtil.java: ## @@ -370,16 +371,167 @@ public void testPartitionStatsWithSchemaEvolution() throws Exceptio

  1   2   >