Re: [PR] Spark 3.5: Refactor scanning changelog table with timestamps [iceberg]

2024-12-12 Thread via GitHub
manuzhang closed pull request #11612: Spark 3.5: Refactor scanning changelog table with timestamps URL: https://github.com/apache/iceberg/pull/11612 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[PR] Spark 3.5: Refactor scanning changelog table with timestamps [iceberg]

2024-12-12 Thread via GitHub
manuzhang opened a new pull request, #11612: URL: https://github.com/apache/iceberg/pull/11612 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Docs: add note for `day` transform [iceberg]

2024-12-12 Thread via GitHub
ZENOTME commented on code in PR #11749: URL: https://github.com/apache/iceberg/pull/11749#discussion_r1881555120 ## format/spec.md: ## @@ -454,7 +454,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ | **`truncate[W]`** | Value truncated to

Re: [PR] Docs: add note for `day` transform [iceberg]

2024-12-12 Thread via GitHub
manuzhang commented on code in PR #11749: URL: https://github.com/apache/iceberg/pull/11749#discussion_r1881543145 ## format/spec.md: ## @@ -454,7 +454,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ | **`truncate[W]`** | Value truncated t

Re: [PR] Hadoop: Log where the missing metadata file is located [iceberg]

2024-12-12 Thread via GitHub
nastra commented on code in PR #11643: URL: https://github.com/apache/iceberg/pull/11643#discussion_r1881580954 ## core/src/test/java/org/apache/iceberg/hadoop/TestHadoopCatalog.java: ## @@ -548,6 +549,32 @@ public void testVersionHintFileMissingMetadata() throws Exception {

Re: [PR] Hadoop: Log where the missing metadata file is located [iceberg]

2024-12-12 Thread via GitHub
nastra commented on code in PR #11643: URL: https://github.com/apache/iceberg/pull/11643#discussion_r1881580954 ## core/src/test/java/org/apache/iceberg/hadoop/TestHadoopCatalog.java: ## @@ -548,6 +549,32 @@ public void testVersionHintFileMissingMetadata() throws Exception {

Re: [PR] Add CMake format [iceberg-cpp]

2024-12-12 Thread via GitHub
pitrou commented on code in PR #5: URL: https://github.com/apache/iceberg-cpp/pull/5#discussion_r1881609652 ## cmake-format.py: ## @@ -0,0 +1,74 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# di

Re: [PR] Docs: add note for `day` transform [iceberg]

2024-12-12 Thread via GitHub
manuzhang commented on code in PR #11749: URL: https://github.com/apache/iceberg/pull/11749#discussion_r1881613465 ## format/spec.md: ## @@ -454,7 +454,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ | **`truncate[W]`** | Value truncated t

Re: [PR] REST: Use apache/iceberg-rest-fixture docker image [iceberg]

2024-12-12 Thread via GitHub
ajantha-bhat commented on PR #11673: URL: https://github.com/apache/iceberg/pull/11673#issuecomment-2538212669 @Fokko: Anything else needed for this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Spark: Remove deprecated SparkAppenderFactory [iceberg]

2024-12-12 Thread via GitHub
ajantha-bhat commented on PR #11727: URL: https://github.com/apache/iceberg/pull/11727#issuecomment-2538213872 Anything else needed for this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [I] Dectect schema evolution or partition evolution for append DataFile [iceberg-rust]

2024-12-12 Thread via GitHub
Fokko commented on issue #777: URL: https://github.com/apache/iceberg-rust/issues/777#issuecomment-2538223320 This is a very interesting question, that I'm happy to elaborate on. > But there are some case it can't detect for this way, e.g. partition spec type reorder to This

[PR] feat: Expose disable_config_load opendal S3 option [iceberg-rust]

2024-12-12 Thread via GitHub
gruuya opened a new pull request, #782: URL: https://github.com/apache/iceberg-rust/pull/782 Closes #781. Parse `S3_DISABLE_CONFIG_LOAD` alongside the other options. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Spark: Remove deprecated SparkAppenderFactory [iceberg]

2024-12-12 Thread via GitHub
nastra merged PR #11727: URL: https://github.com/apache/iceberg/pull/11727 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Spark3.4,3.5: In describe extended view command: fix wrong view catal… [iceberg]

2024-12-12 Thread via GitHub
nastra commented on code in PR #11751: URL: https://github.com/apache/iceberg/pull/11751#discussion_r1881661391 ## spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DescribeV2ViewExec.scala: ## @@ -55,18 +55,20 @@ case class DescribeV2ViewE

Re: [PR] Docs: add note for `day` transform [iceberg]

2024-12-12 Thread via GitHub
xxchan commented on code in PR #11749: URL: https://github.com/apache/iceberg/pull/11749#discussion_r1881671025 ## format/spec.md: ## @@ -454,7 +454,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ | **`truncate[W]`** | Value truncated to w

Re: [PR] Docs: add note for `day` transform [iceberg]

2024-12-12 Thread via GitHub
xxchan commented on code in PR #11749: URL: https://github.com/apache/iceberg/pull/11749#discussion_r1881671025 ## format/spec.md: ## @@ -454,7 +454,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ | **`truncate[W]`** | Value truncated to w

Re: [I] Decouple building and serialization [iceberg-rust]

2024-12-12 Thread via GitHub
Fokko commented on issue #778: URL: https://github.com/apache/iceberg-rust/issues/778#issuecomment-2538270065 Hey @Sl1mb0, Thanks for raising this issue. If I understand correctly if you could provide your own implementation of FileIO, would you be able to make it work? This would avoid the

Re: [PR] Docs: add note for `day` transform [iceberg]

2024-12-12 Thread via GitHub
xxchan commented on code in PR #11749: URL: https://github.com/apache/iceberg/pull/11749#discussion_r1881671025 ## format/spec.md: ## @@ -454,7 +454,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ | **`truncate[W]`** | Value truncated to w

Re: [PR] Azure: Support vended credentials refresh in ADLSFileIO. [iceberg]

2024-12-12 Thread via GitHub
nastra commented on code in PR #11577: URL: https://github.com/apache/iceberg/pull/11577#discussion_r1881530121 ## azure/src/main/java/org/apache/iceberg/azure/adlsv2/AzureSasCredentialRefresher.java: ## @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation (ASF) u

Re: [PR] Docs: add note for `day` transform [iceberg]

2024-12-12 Thread via GitHub
xxchan commented on code in PR #11749: URL: https://github.com/apache/iceberg/pull/11749#discussion_r1881563207 ## format/spec.md: ## @@ -454,7 +454,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ | **`truncate[W]`** | Value truncated to w

Re: [PR] Spec: Support geo type [iceberg]

2024-12-12 Thread via GitHub
jiayuasu commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1881564104 ## format/spec.md: ## @@ -205,13 +205,18 @@ Supported primitive types are defined in the table below. Primitive types added | | **`uuid`**

Re: [PR] Spec: Support geo type [iceberg]

2024-12-12 Thread via GitHub
jiayuasu commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1881568421 ## format/spec.md: ## @@ -584,8 +589,8 @@ The schema of a manifest file is a struct called `manifest_entry` with the follo | _optional_ | _optional_ | _optional_ |

Re: [PR] Spec: Support geo type [iceberg]

2024-12-12 Thread via GitHub
jiayuasu commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1881572306 ## format/spec.md: ## @@ -1331,6 +1345,9 @@ Types are serialized according to this table: |**`struct`**|`JSON object: {`  `"type": "struct",`  `"fields": [ {``"

Re: [PR] Spec: Support geo type [iceberg]

2024-12-12 Thread via GitHub
jiayuasu commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1881575062 ## format/spec.md: ## @@ -1480,6 +1497,9 @@ This serialization scheme is for storing single values as individual binary valu | **`struct`** | Not s

Re: [PR] Hive3: Fix tests not running [iceberg]

2024-12-12 Thread via GitHub
manuzhang closed pull request #11584: Hive3: Fix tests not running URL: https://github.com/apache/iceberg/pull/11584 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

Re: [PR] Core: Unimplement Map from CharSequenceMap to obey contract [iceberg]

2024-12-12 Thread via GitHub
findepi commented on PR #11704: URL: https://github.com/apache/iceberg/pull/11704#issuecomment-2538383679 @nastra PTAL. after this lands i'd like to fix the `CharSequenceSet` class too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [PR] Spark3.4,3.5: In describe extended view command: fix wrong view catal… [iceberg]

2024-12-12 Thread via GitHub
nastra commented on code in PR #11751: URL: https://github.com/apache/iceberg/pull/11751#discussion_r1881747660 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1414,7 +1414,42 @@ public void describeExtendedView() {

Re: [PR] Spark3.4,3.5: In describe extended view command: fix wrong view catal… [iceberg]

2024-12-12 Thread via GitHub
Ppei-Wang commented on code in PR #11751: URL: https://github.com/apache/iceberg/pull/11751#discussion_r1881750426 ## spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DescribeV2ViewExec.scala: ## @@ -55,18 +55,20 @@ case class DescribeV2Vi

Re: [PR] Add CMake format [iceberg-cpp]

2024-12-12 Thread via GitHub
gaborkaszab commented on code in PR #5: URL: https://github.com/apache/iceberg-cpp/pull/5#discussion_r1881752174 ## cmake-format.py: ## @@ -0,0 +1,74 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file

Re: [PR] Core, Rest: Enable useSystemProperties on RESTClient [iceberg]

2024-12-12 Thread via GitHub
nastra commented on PR #11548: URL: https://github.com/apache/iceberg/pull/11548#issuecomment-2538469512 I'm really hesitant to introduce a flag to control whether things get read from system properties or not. Let's wait for feedback from @danielcweeks -- This is an automated message fro

Re: [PR] Core: Unimplement Map from CharSequenceMap to obey contract [iceberg]

2024-12-12 Thread via GitHub
nastra commented on PR #11704: URL: https://github.com/apache/iceberg/pull/11704#issuecomment-2538473468 > @nastra PTAL. after this lands i'd like to fix the `CharSequenceSet` class too. Sorry for the delay. I'm a bit overloaded currently with reviews, so this might me take a while t

Re: [PR] feat(puffin): Add Puffin crate and CompressionCodec [iceberg-rust]

2024-12-12 Thread via GitHub
liurenjie1024 commented on code in PR #745: URL: https://github.com/apache/iceberg-rust/pull/745#discussion_r1881794442 ## crates/puffin/src/compression.rs: ## Review Comment: Do we really need to put this outside of iceberg crate? I'm thinking how will we deal with `Puffi

Re: [PR] Hadoop: Log where the missing metadata file is located [iceberg]

2024-12-12 Thread via GitHub
nastra merged PR #11643: URL: https://github.com/apache/iceberg/pull/11643 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Core: Add missing REST endpoint definitions [iceberg]

2024-12-12 Thread via GitHub
nastra commented on code in PR #11756: URL: https://github.com/apache/iceberg/pull/11756#discussion_r1881884126 ## core/src/main/java/org/apache/iceberg/rest/Endpoint.java: ## @@ -46,6 +46,8 @@ public class Endpoint { Endpoint.create("POST", ResourcePaths.V1_NAMESPACE_PRO

Re: [PR] Core: Add missing REST endpoint definitions [iceberg]

2024-12-12 Thread via GitHub
nastra commented on code in PR #11756: URL: https://github.com/apache/iceberg/pull/11756#discussion_r1881884126 ## core/src/main/java/org/apache/iceberg/rest/Endpoint.java: ## @@ -46,6 +46,8 @@ public class Endpoint { Endpoint.create("POST", ResourcePaths.V1_NAMESPACE_PRO

[PR] Eagerly project the arrow schema to scope out non-selected fields [iceberg-rust]

2024-12-12 Thread via GitHub
gruuya opened a new pull request, #785: URL: https://github.com/apache/iceberg-rust/pull/785 Closes #784. Make use of the projected fields to scope down the arrow schema, and thus potentially avoid some conversions which are not supported yet. -- This is an automated message from t

Re: [I] Discussion: Support conversion of Arrow `Int8` and `Int16` to `PrimitiveType::Int` [iceberg-rust]

2024-12-12 Thread via GitHub
gruuya commented on issue #783: URL: https://github.com/apache/iceberg-rust/issues/783#issuecomment-2539028722 Nice, thanks! I've opened a corresponding PR (also took the liberty of adding support for the new Utf8View as well). -- This is an automated message from the Apache Git Se

Re: [PR] Spec: add variant type [iceberg]

2024-12-12 Thread via GitHub
emkornfield commented on code in PR #10831: URL: https://github.com/apache/iceberg/pull/10831#discussion_r1882513910 ## format/spec.md: ## @@ -182,6 +182,21 @@ A **`list`** is a collection of values with some element type. The element field A **`map`** is a collection of key

Re: [PR] Core: Use HEAD request to check if view exists [iceberg]

2024-12-12 Thread via GitHub
nastra merged PR #11760: URL: https://github.com/apache/iceberg/pull/11760 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Spec: Add cross-region bucket access property to config [iceberg]

2024-12-12 Thread via GitHub
munendrasn commented on PR #11260: URL: https://github.com/apache/iceberg/pull/11260#issuecomment-2538452754 @nastra @Fokko are the changes good to be merged? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] Core, Rest: Read the max connection for rest client from properties [iceberg]

2024-12-12 Thread via GitHub
munendrasn commented on PR #11522: URL: https://github.com/apache/iceberg/pull/11522#issuecomment-2538453726 @danielcweeks could you please review again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Hadoop: Log where the missing metadata file is located [iceberg]

2024-12-12 Thread via GitHub
manuzhang commented on code in PR #11643: URL: https://github.com/apache/iceberg/pull/11643#discussion_r1881711833 ## core/src/test/java/org/apache/iceberg/hadoop/TestHadoopCatalog.java: ## @@ -548,6 +549,32 @@ public void testVersionHintFileMissingMetadata() throws Exception {

Re: [I] Support for S3 catalog to work with S3 Tables [iceberg-python]

2024-12-12 Thread via GitHub
nicor88 commented on issue #1404: URL: https://github.com/apache/iceberg-python/issues/1404#issuecomment-2538292876 It looks like that the warehouse location of those S3 tables doesn't support List operations. I tried to point my local warehouse (using SQLite) to the warehouse location

Re: [PR] Spark3.4,3.5: In describe extended view command: fix wrong view catal… [iceberg]

2024-12-12 Thread via GitHub
nastra commented on code in PR #11751: URL: https://github.com/apache/iceberg/pull/11751#discussion_r1881747660 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1414,7 +1414,42 @@ public void describeExtendedView() {

Re: [PR] Core, Rest: Enable useSystemProperties on RESTClient [iceberg]

2024-12-12 Thread via GitHub
munendrasn commented on PR #11548: URL: https://github.com/apache/iceberg/pull/11548#issuecomment-2538459119 @nastra @danielcweeks could you please review this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] Spec: Add cross-region bucket access property to config [iceberg]

2024-12-12 Thread via GitHub
nastra commented on PR #11260: URL: https://github.com/apache/iceberg/pull/11260#issuecomment-2538464248 the changes LGTM and given this is only a doc change (by stating a new property that's being supported) I don't think this PR would have to go through the normal voting process on the ma

[PR] Core: Use HEAD request to check if view exists [iceberg]

2024-12-12 Thread via GitHub
nastra opened a new pull request, #11760: URL: https://github.com/apache/iceberg/pull/11760 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

Re: [PR] Core: Use HEAD request to check if view exists [iceberg]

2024-12-12 Thread via GitHub
nastra commented on PR #11760: URL: https://github.com/apache/iceberg/pull/11760#issuecomment-2538522361 @Fokko since you reviewed https://github.com/apache/iceberg/pull/10999 already, could you review this one please? -- This is an automated message from the Apache Git Service. To respon

Re: [PR] Add clang format [iceberg-cpp]

2024-12-12 Thread via GitHub
Fokko commented on PR #4: URL: https://github.com/apache/iceberg-cpp/pull/4#issuecomment-2538644674 Let's move this forward, thanks everyone for chiming in here! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[I] Discussion: Support conversion of Arrow `Int8` and `Int16` to `PrimitiveType::Int` [iceberg-rust]

2024-12-12 Thread via GitHub
gruuya opened a new issue, #783: URL: https://github.com/apache/iceberg-rust/issues/783 Presently only `Int32` is converted into the corresponding Iceberg type `Int` https://github.com/apache/iceberg-rust/blob/42aff04658a00b390122260dbbeaf512d11af61f/crates/iceberg/src/arrow/schema.rs#L370

Re: [PR] Spark3.4,3.5: In describe extended view command: fix wrong view catal… [iceberg]

2024-12-12 Thread via GitHub
Ppei-Wang commented on code in PR #11751: URL: https://github.com/apache/iceberg/pull/11751#discussion_r1882185303 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1414,7 +1414,42 @@ public void describeExtendedView() {

Re: [PR] Core: Change Delete granularity to file for new tables [iceberg]

2024-12-12 Thread via GitHub
amogh-jahagirdar commented on code in PR #11478: URL: https://github.com/apache/iceberg/pull/11478#discussion_r1882456464 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java: ## @@ -231,7 +233,6 @@ public void testMergeWithVectorizedRe

Re: [PR] Hive: Add Hive 4 support and remove Hive 3 [iceberg]

2024-12-12 Thread via GitHub
nastra commented on code in PR #11750: URL: https://github.com/apache/iceberg/pull/11750#discussion_r1882459582 ## gradle/libs.versions.toml: ## @@ -139,10 +139,10 @@ hive2-exec = { module = "org.apache.hive:hive-exec", version.ref = "hive2" } hive2-metastore = { module = "org

[I] Rename the partition field and add a field with the same name as the old partition field GOT ERROR [iceberg]

2024-12-12 Thread via GitHub
madeirak opened a new issue, #11762: URL: https://github.com/apache/iceberg/issues/11762 ### Apache Iceberg version 1.4.3 ### Query engine Spark ### Please describe the bug 🐞 ``` CREATE TABLE db03.test_123 ( id INT COMMENT '11', name STRING COM

Re: [I] Discussion: Support conversion of Arrow `Int8` and `Int16` to `PrimitiveType::Int` [iceberg-rust]

2024-12-12 Thread via GitHub
Fokko closed issue #783: Discussion: Support conversion of Arrow `Int8` and `Int16` to `PrimitiveType::Int` URL: https://github.com/apache/iceberg-rust/issues/783 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] Suport conversion of Arrow Int8 and Int16 to Iceberg Int [iceberg-rust]

2024-12-12 Thread via GitHub
Fokko merged PR #787: URL: https://github.com/apache/iceberg-rust/pull/787 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

[I] Misleading use of LoadTableResponse in RESTTableOperations.commit() [iceberg]

2024-12-12 Thread via GitHub
creechy opened a new issue, #11767: URL: https://github.com/apache/iceberg/issues/11767 ### Apache Iceberg version None ### Query engine None ### Please describe the bug 🐞 This is a little nit-picky, but the Iceberg REST spec defines the response of the Upd

[PR] Add more integration tests [iceberg-rust]

2024-12-12 Thread via GitHub
Fokko opened a new pull request, #786: URL: https://github.com/apache/iceberg-rust/pull/786 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

Re: [PR] Core: Add missing REST endpoint definitions [iceberg]

2024-12-12 Thread via GitHub
nastra commented on code in PR #11756: URL: https://github.com/apache/iceberg/pull/11756#discussion_r1882572295 ## core/src/main/java/org/apache/iceberg/rest/Endpoint.java: ## @@ -46,6 +46,8 @@ public class Endpoint { Endpoint.create("POST", ResourcePaths.V1_NAMESPACE_PRO

Re: [PR] Core: Use HEAD request to check if namespace exists [iceberg]

2024-12-12 Thread via GitHub
nastra merged PR #11761: URL: https://github.com/apache/iceberg/pull/11761 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Spec: Support geo type [iceberg]

2024-12-12 Thread via GitHub
paleolimbot commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1882340902 ## format/spec.md: ## @@ -205,13 +205,18 @@ Supported primitive types are defined in the table below. Primitive types added | | **`uuid`**

Re: [PR] Spec: Support geo type [iceberg]

2024-12-12 Thread via GitHub
paleolimbot commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1882346427 ## format/spec.md: ## @@ -1480,6 +1497,9 @@ This serialization scheme is for storing single values as individual binary valu | **`struct`** | No

Re: [PR] Spec: Support geo type [iceberg]

2024-12-12 Thread via GitHub
paleolimbot commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1882353225 ## format/spec.md: ## @@ -603,6 +608,10 @@ Notes: 4. Position delete metadata can use `referenced_data_file` when all deletes tracked by the entry are in a sing

Re: [PR] feat: eagerly project the arrow schema to scope out non-selected fields [iceberg-rust]

2024-12-12 Thread via GitHub
gruuya commented on PR #785: URL: https://github.com/apache/iceberg-rust/pull/785#issuecomment-2539092128 > let's see if others have any concerns. Thanks! I've revised the test (with a slightly contrived example) seeing that Int8 example is now support with #787 (which addresses my im

Re: [PR] Add more integration tests [iceberg-rust]

2024-12-12 Thread via GitHub
Fokko commented on code in PR #786: URL: https://github.com/apache/iceberg-rust/pull/786#discussion_r1882162789 ## crates/integration_tests/tests/read_evolved_schema.rs: ## @@ -0,0 +1,80 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [I] Eagerly project arrow schema when calculating the parquet `ProjectionMask` [iceberg-rust]

2024-12-12 Thread via GitHub
Fokko commented on issue #784: URL: https://github.com/apache/iceberg-rust/issues/784#issuecomment-2539010624 Thanks @gruuya That would be a great addition. I think we should only project the needed fields anyway. I was doing some testing along the same line, and also noticed that we

Re: [PR] Core: Add missing REST endpoint definitions [iceberg]

2024-12-12 Thread via GitHub
ajreid21 commented on code in PR #11756: URL: https://github.com/apache/iceberg/pull/11756#discussion_r1882395425 ## core/src/main/java/org/apache/iceberg/rest/Endpoint.java: ## @@ -61,6 +63,19 @@ public class Endpoint { Endpoint.create("POST", ResourcePaths.V1_TABLE_REGI

Re: [I] Data loss bug in MergeIntoCommand [iceberg]

2024-12-12 Thread via GitHub
RussellSpitzer commented on issue #11765: URL: https://github.com/apache/iceberg/issues/11765#issuecomment-2539320431 We don't have auto merge schema so I don't think we have the same issue as in the Delta issue (at least not yet). Do you have any more details about the data loss? -- Thi

Re: [PR] Add clang format [iceberg-cpp]

2024-12-12 Thread via GitHub
Fokko merged PR #4: URL: https://github.com/apache/iceberg-cpp/pull/4 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.

Re: [I] Expose `disable_config_load` opendal S3config option [iceberg-rust]

2024-12-12 Thread via GitHub
Xuanwo closed issue #781: Expose `disable_config_load` opendal S3config option URL: https://github.com/apache/iceberg-rust/issues/781 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

[I] Data loss bug in MergeIntoCommand [iceberg]

2024-12-12 Thread via GitHub
BsoBird opened a new issue, #11765: URL: https://github.com/apache/iceberg/issues/11765 ### Apache Iceberg version 1.7.1 (latest release) ### Query engine Spark ### Please describe the bug 🐞 Recently, I've noticed that when using the merge into statement in

Re: [PR] feat: Expose disable_config_load opendal S3 option [iceberg-rust]

2024-12-12 Thread via GitHub
Xuanwo merged PR #782: URL: https://github.com/apache/iceberg-rust/pull/782 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Core: Add missing REST endpoint definitions [iceberg]

2024-12-12 Thread via GitHub
nastra commented on code in PR #11756: URL: https://github.com/apache/iceberg/pull/11756#discussion_r1881884126 ## core/src/main/java/org/apache/iceberg/rest/Endpoint.java: ## @@ -46,6 +46,8 @@ public class Endpoint { Endpoint.create("POST", ResourcePaths.V1_NAMESPACE_PRO

Re: [PR] Core: Add missing REST endpoint definitions [iceberg]

2024-12-12 Thread via GitHub
nastra commented on code in PR #11756: URL: https://github.com/apache/iceberg/pull/11756#discussion_r1881888246 ## core/src/main/java/org/apache/iceberg/rest/Endpoint.java: ## @@ -61,6 +63,19 @@ public class Endpoint { Endpoint.create("POST", ResourcePaths.V1_TABLE_REGIST

Re: [PR] IO Implementation using Go CDK [iceberg-go]

2024-12-12 Thread via GitHub
loicalleyne commented on PR #176: URL: https://github.com/apache/iceberg-go/pull/176#issuecomment-2539346597 @zeroshade > In the meantime can you resolve the conflict in the go.mod? Thanks! I tried updating the `go.mod` version and toolchain versions to match `main` and ran `go mod ti

Re: [PR] Core: Use HEAD request to check if view exists [iceberg]

2024-12-12 Thread via GitHub
nastra commented on PR #11760: URL: https://github.com/apache/iceberg/pull/11760#issuecomment-2539361082 @amogh-jahagirdar good point, I've added a test to `TestRESTViewCatalog` as that's the more appropriate place to check this (instead of `TestRESTCatalog`) -- This is an automated messa

Re: [PR] IO Implementation using Go CDK [iceberg-go]

2024-12-12 Thread via GitHub
dwilson1988 commented on PR #176: URL: https://github.com/apache/iceberg-go/pull/176#issuecomment-2539362190 @loicalleyne - you should just be able to manually remove conflicts in `go.mod`, delete `go.sum` and run a `go mod tidy`. Probably best to do this after syncing your fork and rebasin

[I] Performance Regression Caused by Schema Hash in Spark PartitionPruning with Wide Tables [iceberg]

2024-12-12 Thread via GitHub
wzx140 opened a new issue, #11763: URL: https://github.com/apache/iceberg/issues/11763 ### Apache Iceberg version 1.5.0 ### Query engine Spark ### Please describe the bug 🐞 **Description**: In Spark’s optimization rule *PartitionPruning*, the method `Spa

Re: [I] Discussion: Support conversion of Arrow `Int8` and `Int16` to `PrimitiveType::Int` [iceberg-rust]

2024-12-12 Thread via GitHub
Fokko commented on issue #783: URL: https://github.com/apache/iceberg-rust/issues/783#issuecomment-2538992416 @gruuya I think that would be fine to cast those into an int 👍 In PyIceberg we do the same: https://github.com/apache/iceberg-python/blob/547d881948dfe17c92bdde9e5b63a94d095a1

[I] Freshness aware table loading in REST catalog [iceberg]

2024-12-12 Thread via GitHub
gaborkaszab opened a new issue, #11766: URL: https://github.com/apache/iceberg/issues/11766 ### Proposed Change There are clients of the Iceberg table format (e.g. query engines) that cache table metadata. In order to keep the cache up-to-date they implement different mechanisms like

Re: [PR] Fix `release_rc.sh`, use the right artifact file name [iceberg-go]

2024-12-12 Thread via GitHub
zeroshade merged PR #203: URL: https://github.com/apache/iceberg-go/pull/203 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] build(deps): bump golang.org/x/sync from 0.9.0 to 0.10.0 [iceberg-go]

2024-12-12 Thread via GitHub
zeroshade merged PR #223: URL: https://github.com/apache/iceberg-go/pull/223 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] API: add hashcode cache in StructType [iceberg]

2024-12-12 Thread via GitHub
singhpk234 commented on PR #11764: URL: https://github.com/apache/iceberg/pull/11764#issuecomment-2539708719 Q: does it completely mitigate the flatness observed ? can you please attach the flame graph now ? Interesting find @wzx140 -- This is an automated message from the Apache Git

Re: [PR] Auth Manager API part 1: HTTPRequest, HTTPHeader [iceberg]

2024-12-12 Thread via GitHub
adutra commented on code in PR #11769: URL: https://github.com/apache/iceberg/pull/11769#discussion_r1882682585 ## core/src/main/java/org/apache/iceberg/rest/HTTPHeaders.java: ## @@ -0,0 +1,168 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

Re: [PR] Auth Manager API part 1: HTTPRequest, HTTPHeader [iceberg]

2024-12-12 Thread via GitHub
adutra commented on code in PR #11769: URL: https://github.com/apache/iceberg/pull/11769#discussion_r1882683361 ## core/src/main/java/org/apache/iceberg/rest/HTTPHeaders.java: ## @@ -0,0 +1,168 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

Re: [PR] REST: AuthManager API [iceberg]

2024-12-12 Thread via GitHub
adutra commented on PR #10753: URL: https://github.com/apache/iceberg/pull/10753#issuecomment-2539723438 As requested, this PR will be split in many ones. The first one is #11769. I'm going to close this one now. -- This is an automated message from the Apache Git Service. To respond to t

Re: [PR] Impl rest catalog + table updates & requirements [iceberg-go]

2024-12-12 Thread via GitHub
zeroshade commented on code in PR #146: URL: https://github.com/apache/iceberg-go/pull/146#discussion_r1882697400 ## table/metadata.go: ## @@ -80,20 +92,544 @@ type Metadata interface { SnapshotByName(name string) *Snapshot // CurrentSnapshot returns the table's

Re: [I] Decouple building and serialization [iceberg-rust]

2024-12-12 Thread via GitHub
Sl1mb0 commented on issue #778: URL: https://github.com/apache/iceberg-rust/issues/778#issuecomment-2539813154 > If I understand correctly if you could provide your own implementation of FileIO, would you be able to make it work? This would avoid the copy. Hmm - this may work, but it'

[PR] Auth Manager API part 1: HTTPRequest, HTTPHeader [iceberg]

2024-12-12 Thread via GitHub
adutra opened a new pull request, #11769: URL: https://github.com/apache/iceberg/pull/11769 As requested, I'm splitting #10753 in many PRs. This one is the first one. It introduces `HTTPRequest` which is a prerequisite for the `AuthManager` API. -- This is an automated message from the Ap

[I] Wrong name for parquet page row count min and max stats [iceberg]

2024-12-12 Thread via GitHub
namrathamyske opened a new issue, #11770: URL: https://github.com/apache/iceberg/issues/11770 ### Apache Iceberg version 1.7.1 (latest release) ### Query engine None ### Please describe the bug 🐞 In TableProperties, the properties PARQUET_ROW_GROUP_CHECK_MAX

Re: [PR] Feat: support aliyun oss backend. [iceberg-go]

2024-12-12 Thread via GitHub
zeroshade commented on code in PR #216: URL: https://github.com/apache/iceberg-go/pull/216#discussion_r1882673169 ## catalog/catalog.go: ## @@ -32,6 +32,12 @@ type CatalogType string type AwsProperties map[string]string +type OSSConfig struct { Review Comment: Can we ad

Re: [PR] build(deps): bump github.com/stretchr/testify from 1.9.0 to 1.10.0 [iceberg-go]

2024-12-12 Thread via GitHub
zeroshade merged PR #218: URL: https://github.com/apache/iceberg-go/pull/218 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Hive: Optimize tableExists API in hive catalog [iceberg]

2024-12-12 Thread via GitHub
dramaticlly commented on PR #11597: URL: https://github.com/apache/iceberg/pull/11597#issuecomment-2539699512 Thank you @danielcweeks @szehon-ho @pvary @kevinjqliu @gaborkaszab @haizhou-zhao for the review! I will look into similar change for hive view existence check -- This is an autom

Re: [PR] Hive: Optimize tableExists API in hive catalog [iceberg]

2024-12-12 Thread via GitHub
danielcweeks merged PR #11597: URL: https://github.com/apache/iceberg/pull/11597 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceb

Re: [PR] Hive: Optimize tableExists API in hive catalog [iceberg]

2024-12-12 Thread via GitHub
danielcweeks commented on PR #11597: URL: https://github.com/apache/iceberg/pull/11597#issuecomment-2539677480 Thanks @dramaticlly !! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.67.1 to 1.71.0 [iceberg-go]

2024-12-12 Thread via GitHub
zeroshade merged PR #225: URL: https://github.com/apache/iceberg-go/pull/225 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] IO Implementation using Go CDK [iceberg-go]

2024-12-12 Thread via GitHub
zeroshade commented on PR #176: URL: https://github.com/apache/iceberg-go/pull/176#issuecomment-2539692409 @loicalleyne looks like the integration tests are failing, unable to read the manifest files from the minio instance. -- This is an automated message from the Apache Git Service. To

Re: [PR] REST: AuthManager API [iceberg]

2024-12-12 Thread via GitHub
adutra closed pull request #10753: REST: AuthManager API URL: https://github.com/apache/iceberg/pull/10753 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Re: [PR] Spec: Support geo type [iceberg]

2024-12-12 Thread via GitHub
szehon-ho commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1882776271 ## format/spec.md: ## @@ -1480,6 +1497,9 @@ This serialization scheme is for storing single values as individual binary valu | **`struct`** | Not

Re: [I] Wrong name for parquet page row count min and max stats [iceberg]

2024-12-12 Thread via GitHub
namrathamyske commented on issue #11770: URL: https://github.com/apache/iceberg/issues/11770#issuecomment-2539987558 cc: @stevenzwu @rdblue @nastra -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

  1   2   >