Re: [PR] Spark3.4,3.5: Fix the BUG of iceberg views when resolved "group/order… [iceberg]

nastra commented on code in PR #11729: URL: https://github.com/apache/iceberg/pull/11729#discussion_r1879872034 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -562,6 +562,64 @@ public void readFromViewWithCTE() throws NoSu

Re: [PR] Spark3.4,3.5: Fix the BUG of iceberg views when resolved "group/order… [iceberg]

nastra commented on code in PR #11729: URL: https://github.com/apache/iceberg/pull/11729#discussion_r1879874793 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -562,6 +562,64 @@ public void readFromViewWithCTE() throws NoSu

Re: [PR] Spark3.4,3.5: Fix the BUG of iceberg views when resolved "group/order… [iceberg]

nastra commented on code in PR #11729: URL: https://github.com/apache/iceberg/pull/11729#discussion_r1879875614 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -562,6 +562,64 @@ public void readFromViewWithCTE() throws NoSu

Re: [PR] Add Support for Dynamic Overwrite [iceberg-python]

jqin61 commented on PR #931: URL: https://github.com/apache/iceberg-python/pull/931#issuecomment-2537415525 Thanks for fixing the jar issue, shall we rerun CI and merge? @Fokko Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] Spark: add property to disable client-side purging in spark [iceberg]

rdblue commented on code in PR #11317: URL: https://github.com/apache/iceberg/pull/11317#discussion_r1881174754 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java: ## @@ -365,6 +370,14 @@ public boolean purgeTable(Identifier ident) { String metad

Re: [PR] Core: Add missing REST endpoint definitions [iceberg]

ebyhr commented on code in PR #11756: URL: https://github.com/apache/iceberg/pull/11756#discussion_r1881226290 ## core/src/main/java/org/apache/iceberg/rest/Endpoint.java: ## @@ -61,6 +63,19 @@ public class Endpoint { Endpoint.create("POST", ResourcePaths.V1_TABLE_REGISTE

Re: [PR] doc: add note for `day` transform [iceberg]

manuzhang commented on code in PR #11749: URL: https://github.com/apache/iceberg/pull/11749#discussion_r1881244151 ## format/spec.md: ## @@ -454,7 +454,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ | **`truncate[W]`** | Value truncated t

Re: [PR] Spark 3.5: Add ignore-invalid-options to RewriteDataFilesSparkAction and RewritePositionDeleteFilesSparkAction [iceberg]

manuzhang commented on PR #11737: URL: https://github.com/apache/iceberg/pull/11737#issuecomment-2537563264 @ajantha-bhat @RussellSpitzer @nastra Please help review whether this is a valid request. -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [PR] Spec: Support geo type [iceberg]

mkaravel commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1879219879 ## format/spec.md: ## @@ -205,13 +205,18 @@ Supported primitive types are defined in the table below. Primitive types added | | **`uuid`**

[PR] Fix: Resolve IDENTIFIER FIELDS with merge-on-read bug [iceberg]

601madman opened a new pull request, #11757: URL: https://github.com/apache/iceberg/pull/11757 ### Problem When IDENTIFIER FIELDS are set, and `merge-on-read` mode is used, a validation error occurs due to incorrect metadata schema checks. ### Solution Adjusted the `calculateMet

Re: [PR] [WIP][Core] Restrict adding column of StructType with Empty Fields [iceberg]

ebyhr commented on code in PR #11755: URL: https://github.com/apache/iceberg/pull/11755#discussion_r1881237568 ## core/src/test/java/org/apache/iceberg/TestSchemaUpdate.java: ## @@ -731,6 +731,17 @@ public void testAmbiguousAdd() { .hasMessageStartingWith("Cannot add co

Re: [PR] Add clang format [iceberg-cpp]

zhjwpku commented on code in PR #4: URL: https://github.com/apache/iceberg-cpp/pull/4#discussion_r1881257096 ## .clang-format: ## @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# dis

Re: [PR] Spark: add property to disable client-side purging in spark [iceberg]

rdblue commented on code in PR #11317: URL: https://github.com/apache/iceberg/pull/11317#discussion_r1881170392 ## core/src/main/java/org/apache/iceberg/CatalogProperties.java: ## @@ -78,6 +78,14 @@ private CatalogProperties() {} public static final boolean IO_MANIFEST_CACH

Re: [PR] Core: Add Variant implementation to read serialized objects [iceberg]

rdblue commented on code in PR #11415: URL: https://github.com/apache/iceberg/pull/11415#discussion_r1881180413 ## core/src/main/java/org/apache/iceberg/variants/PrimitiveWrapper.java: ## @@ -0,0 +1,206 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [I] Create iceberg table from existsing parquet files with slightly different schemas (schemas merge is possible). [iceberg-python]

github-actions[bot] commented on issue #601: URL: https://github.com/apache/iceberg-python/issues/601#issuecomment-2537469223 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apac

Re: [I] Create iceberg table from existsing parquet files with slightly different schemas (schemas merge is possible). [iceberg-python]

github-actions[bot] closed issue #601: Create iceberg table from existsing parquet files with slightly different schemas (schemas merge is possible). URL: https://github.com/apache/iceberg-python/issues/601 -- This is an automated message from the Apache Git Service. To respond to the message

Re: [I] BUG: Bug: partition name stored in partition data in data file contains special character [iceberg-python]

github-actions[bot] commented on issue #175: URL: https://github.com/apache/iceberg-python/issues/175#issuecomment-2537469241 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity oc

Re: [PR] Add CMake format [iceberg-cpp]

zhjwpku commented on code in PR #5: URL: https://github.com/apache/iceberg-cpp/pull/5#discussion_r1881282329 ## cmake-format.py: ## @@ -0,0 +1,74 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# d

Re: [PR] Spec: Support geo type [iceberg]

mkaravel commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1881291822 ## format/spec.md: ## @@ -603,6 +608,10 @@ Notes: 4. Position delete metadata can use `referenced_data_file` when all deletes tracked by the entry are in a single

Re: [PR] Spec: Support geo type [iceberg]

mkaravel commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1881291822 ## format/spec.md: ## @@ -603,6 +608,10 @@ Notes: 4. Position delete metadata can use `referenced_data_file` when all deletes tracked by the entry are in a single

[PR] infra: Dismiss stale reviews [iceberg-rust]

liurenjie1024 opened a new pull request, #779: URL: https://github.com/apache/iceberg-rust/pull/779 Add a protected branch rule to dismiss stale reviews when new commits are added. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [I] Merge into using the exactly dataset copy the entire data [iceberg]

github-actions[bot] commented on issue #9736: URL: https://github.com/apache/iceberg/issues/9736#issuecomment-2537466213 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] No module named 'pyiceberg.table.partitioning' [iceberg]

github-actions[bot] commented on issue #10491: URL: https://github.com/apache/iceberg/issues/10491#issuecomment-2537466332 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] Core: Store schema and spec in TaskContext to avoid unnecessary deserialization (#11235) [iceberg]

github-actions[bot] closed pull request #11280: Core: Store schema and spec in TaskContext to avoid unnecessary deserialization (#11235) URL: https://github.com/apache/iceberg/pull/11280 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [PR] Workers gets stuck as there is no-coordinator for emitting Start_Commit request in Incremental Cooperative Rebalancing[ICR] Mode [iceberg]

github-actions[bot] commented on PR #11288: URL: https://github.com/apache/iceberg/pull/11288#issuecomment-2537466505 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [PR] update PartitionSpec with snapshot'schema [iceberg]

github-actions[bot] commented on PR #11196: URL: https://github.com/apache/iceberg/pull/11196#issuecomment-2537466436 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Core: Store schema and spec in TaskContext to avoid unnecessary deserialization (#11235) [iceberg]

github-actions[bot] commented on PR #11280: URL: https://github.com/apache/iceberg/pull/11280#issuecomment-2537466476 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [I] Add support for Spark 4.0.0 [iceberg]

github-actions[bot] commented on issue #10497: URL: https://github.com/apache/iceberg/issues/10497#issuecomment-2537466356 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Merge into using the exactly dataset copy the entire data [iceberg]

github-actions[bot] closed issue #9736: Merge into using the exactly dataset copy the entire data URL: https://github.com/apache/iceberg/issues/9736 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] update PartitionSpec with snapshot'schema [iceberg]

github-actions[bot] closed pull request #11196: update PartitionSpec with snapshot'schema URL: https://github.com/apache/iceberg/pull/11196 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Remove Hive 2 [iceberg]

manuzhang commented on PR #10996: URL: https://github.com/apache/iceberg/pull/10996#issuecomment-2537550479 We have a [consensus on the dev list](https://lists.apache.org/thread/jfcqfw9vhq4j7h0kwnlf338jgyzcq8s4) to drop hive-runtime and upgrade to Hive 4. I've submitted https://github.com/

Re: [PR] Spark 3.5: Fix comment and assertion mismatch in PartitionedWritesTestBase/TestRewritePositionDeleteFilesAction [iceberg]

wzx140 commented on code in PR #11748: URL: https://github.com/apache/iceberg/pull/11748#discussion_r1881295827 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewritePositionDeleteFilesAction.java: ## @@ -275,7 +275,7 @@ public void testRewriteFilter() th

Re: [PR] refactor: avoid async_trait for FileRead and provide object safe dyn methods [iceberg-rust]

liurenjie1024 commented on PR #761: URL: https://github.com/apache/iceberg-rust/pull/761#issuecomment-2537679283 Hi, @wenym1 I saw you submitted several similar refactoring to current api. While the community appreciate your contribution, could you open an issue to raise discussion about th

[PR] Add plan tasks for TableScan [iceberg-python]

ConeyLiu opened a new pull request, #1427: URL: https://github.com/apache/iceberg-python/pull/1427 Now, we only support plan files. Plan tasks(split large file based on split_offset) would be more useful when we want to read in parallel. -- This is an automated message from the Apache Git

[PR] Spark 3.5: Add query runner in test module [iceberg]

ebyhr opened a new pull request, #11758: URL: https://github.com/apache/iceberg/pull/11758 I propose adding query runners in tests so developers can debug Spark SQL without local jar publish. This is similar to query runners in Trino project. The project provides several query runne

Re: [PR] Use SupportsPrefixOperations for Remove OrphanFile Procedure [iceberg]

Samreay commented on PR #7914: URL: https://github.com/apache/iceberg/pull/7914#issuecomment-2537695273 Has anyone got a nice workaround for how to remove orphan files for an S3-located iceberg table? -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [I] Compatibility Issue with pydantic and annotated-types in pyiceberg 0.8.1 [iceberg-python]

pawansanz commented on issue #1418: URL: https://github.com/apache/iceberg-python/issues/1418#issuecomment-2537769371 can anyone help with original issue that was reported regarding "Apache Iceberg version" -- This is an automated message from the Apache Git Service. To respond to the me

Re: [PR] Spark3.4,3.5,Api,Hive: Fix using NullType in View. [iceberg]

Ppei-Wang commented on code in PR #11728: URL: https://github.com/apache/iceberg/pull/11728#discussion_r1881446544 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -412,6 +413,24 @@ public String toString() { } } + public static class NullType extends P

Re: [PR] Azure: Support vended credentials refresh in ADLSFileIO. [iceberg]

nastra commented on code in PR #11577: URL: https://github.com/apache/iceberg/pull/11577#discussion_r1881531528 ## azure/src/main/java/org/apache/iceberg/azure/adlsv2/AzureSasCredentialRefresher.java: ## @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation (ASF) u

Re: [PR] Azure: Support vended credentials refresh in ADLSFileIO. [iceberg]

nastra commented on code in PR #11577: URL: https://github.com/apache/iceberg/pull/11577#discussion_r1881533944 ## azure/src/main/java/org/apache/iceberg/azure/adlsv2/VendedAdlsCredentialProvider.java: ## @@ -0,0 +1,197 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] Azure: Support vended credentials refresh in ADLSFileIO. [iceberg]

nastra commented on code in PR #11577: URL: https://github.com/apache/iceberg/pull/11577#discussion_r1881531755 ## azure/src/main/java/org/apache/iceberg/azure/adlsv2/AzureSasCredentialRefresher.java: ## @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation (ASF) u

Re: [PR] Azure: Support vended credentials refresh in ADLSFileIO. [iceberg]

nastra commented on code in PR #11577: URL: https://github.com/apache/iceberg/pull/11577#discussion_r1881528468 ## azure/src/main/java/org/apache/iceberg/azure/adlsv2/AzureSasCredentialRefresher.java: ## @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation (ASF) u

Re: [PR] Docs: fix typo in spec [iceberg]

nastra merged PR #11759: URL: https://github.com/apache/iceberg/pull/11759 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Spark3.4,3.5: Fix the BUG of iceberg views when resolved "group by ordinals" [iceberg]

nastra merged PR #11729: URL: https://github.com/apache/iceberg/pull/11729 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Docs: add note for `day` transform [iceberg]

Fokko commented on code in PR #11749: URL: https://github.com/apache/iceberg/pull/11749#discussion_r1881537732 ## format/spec.md: ## @@ -454,7 +454,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ | **`truncate[W]`** | Value truncated to wi

Re: [PR] Azure: Support vended credentials refresh in ADLSFileIO. [iceberg]

nastra commented on PR #11577: URL: https://github.com/apache/iceberg/pull/11577#issuecomment-2538047119 @danielcweeks could you also take a look at this PR please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] Azure: Support vended credentials refresh in ADLSFileIO. [iceberg]

nastra commented on code in PR #11577: URL: https://github.com/apache/iceberg/pull/11577#discussion_r1881537532 ## azure/src/main/java/org/apache/iceberg/azure/adlsv2/VendedAdlsCredentialProvider.java: ## @@ -0,0 +1,197 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] Hadoop: Log where the missing metadata file is located [iceberg]

manuzhang commented on PR #11643: URL: https://github.com/apache/iceberg/pull/11643#issuecomment-2538063691 @nastra any more comments? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Kafka Connect runtime package is missing Nessie catalog jars [iceberg]

nikulaja commented on issue #11733: URL: https://github.com/apache/iceberg/issues/11733#issuecomment-2538002097 @Fokko Thank you for reply! Seems very reasonable. Could this information be added to documentation? I'm very new to iceberg so wouldn't know if this is a regular thing with other

Re: [PR] doc: add note for `day` transform [iceberg]

ZENOTME commented on code in PR #11749: URL: https://github.com/apache/iceberg/pull/11749#discussion_r1881503123 ## format/spec.md: ## @@ -454,7 +454,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ | **`truncate[W]`** | Value truncated to

Re: [PR] doc: add note for `day` transform [iceberg]

xxchan commented on code in PR #11749: URL: https://github.com/apache/iceberg/pull/11749#discussion_r1881514203 ## format/spec.md: ## @@ -454,7 +454,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ | **`truncate[W]`** | Value truncated to w

Re: [PR] Core: Add missing REST endpoint definitions [iceberg]

nastra commented on code in PR #11756: URL: https://github.com/apache/iceberg/pull/11756#discussion_r1881519406 ## core/src/main/java/org/apache/iceberg/rest/Endpoint.java: ## @@ -70,6 +85,7 @@ public class Endpoint { public static final Endpoint V1_DELETE_VIEW = Endpoint.cre

Re: [PR] Core: Add missing REST endpoint definitions [iceberg]

nastra commented on code in PR #11756: URL: https://github.com/apache/iceberg/pull/11756#discussion_r1881520238 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -149,6 +149,7 @@ public class RESTSessionCatalog extends BaseViewSessionCatalog

Re: [PR] Core: Add missing REST endpoint definitions [iceberg]

nastra commented on code in PR #11756: URL: https://github.com/apache/iceberg/pull/11756#discussion_r1881520238 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -149,6 +149,7 @@ public class RESTSessionCatalog extends BaseViewSessionCatalog

Re: [PR] Alter `Transform::Day` to map partition types to `Date` rather than `Int` for consistency with reference implementation [iceberg-rust]

ZENOTME commented on PR #479: URL: https://github.com/apache/iceberg-rust/pull/479#issuecomment-2538033235 Seems this PR affects the result type of Year, Month, Day transform. Should we only change the result type of Transform::Day.🤔 -- This is an automated message from the Apache Git Ser

Re: [PR] Azure: Support vended credentials refresh in ADLSFileIO. [iceberg]

nastra commented on code in PR #11577: URL: https://github.com/apache/iceberg/pull/11577#discussion_r1881527842 ## azure/src/main/java/org/apache/iceberg/azure/AzureProperties.java: ## @@ -90,7 +117,9 @@ public Optional adlsWriteBlockSize() { */ public void applyClientCon

Re: [PR] Azure: Support vended credentials refresh in ADLSFileIO. [iceberg]

nastra commented on code in PR #11577: URL: https://github.com/apache/iceberg/pull/11577#discussion_r1881531305 ## azure/src/main/java/org/apache/iceberg/azure/adlsv2/AzureSasCredentialRefresher.java: ## @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation (ASF) u

Re: [PR] Core, Spark3.5: Fix tests failure due to timeout [iceberg]

manuzhang closed pull request #11654: Core, Spark3.5: Fix tests failure due to timeout URL: https://github.com/apache/iceberg/pull/11654 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Spark 3.5: Fix comment and assertion mismatch in PartitionedWritesTestBase/TestRewritePositionDeleteFilesAction [iceberg]

wzx140 commented on code in PR #11748: URL: https://github.com/apache/iceberg/pull/11748#discussion_r1881295827 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewritePositionDeleteFilesAction.java: ## @@ -275,7 +275,7 @@ public void testRewriteFilter() th

Re: [PR] refactor: avoid async_trait macro for IcebergWriter and provide extra dyn trait for object safety [iceberg-rust]

ZENOTME commented on PR #760: URL: https://github.com/apache/iceberg-rust/pull/760#issuecomment-2537842348 I find that the implementation has some problems now, it will cause recursive calls endlessly and stack overflow finally. Reproduce: ``` #[tokio::test] async fn

Re: [PR] doc: add note for `day` transform [iceberg]

ZENOTME commented on code in PR #11749: URL: https://github.com/apache/iceberg/pull/11749#discussion_r1881503123 ## format/spec.md: ## @@ -454,7 +454,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ | **`truncate[W]`** | Value truncated to

Re: [PR] doc: add note for `day` transform [iceberg]

xxchan commented on code in PR #11749: URL: https://github.com/apache/iceberg/pull/11749#discussion_r1881355866 ## format/spec.md: ## @@ -454,7 +454,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ | **`truncate[W]`** | Value truncated to w

Re: [PR] Hive: Add Hive 4 support and remove Hive 3 [iceberg]

pvary commented on PR #11750: URL: https://github.com/apache/iceberg/pull/11750#issuecomment-2537927187 Will check next week if it is not merged till then -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Reduce code duplication in VectorizedParquetDefinitionLevelReader [iceberg]

wypoon commented on PR #11661: URL: https://github.com/apache/iceberg/pull/11661#issuecomment-2537946114 Hi @nastra, I have addressed your feedback last week. Can you please review this again? -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [PR] refactor: avoid async_trait macro for IcebergWriter and provide extra dyn trait for object safety [iceberg-rust]

wenym1 commented on PR #760: URL: https://github.com/apache/iceberg-rust/pull/760#issuecomment-2537946595 > I find that the implementation has some problems now, it will cause recursive calls endlessly and stack overflow finally. > > Reproduce: > > ``` >#[tokio::test]

<    1   2