Re: [PR] Doc: Do Not Modify the Source Data Table During MergeIntoCommand Exec… [iceberg]

2024-12-15 Thread via GitHub
BsoBird commented on code in PR #11787: URL: https://github.com/apache/iceberg/pull/11787#discussion_r1886334243 ## docs/docs/spark-writes.md: ## @@ -101,6 +101,9 @@ Spark 3.5 added support for `WHEN NOT MATCHED BY SOURCE ... THEN ...` to update WHEN NOT MATCHED BY SOURCE THEN

Re: [PR] Doc: Do Not Modify the Source Data Table During MergeIntoCommand Exec… [iceberg]

2024-12-15 Thread via GitHub
Fokko commented on code in PR #11787: URL: https://github.com/apache/iceberg/pull/11787#discussion_r1886329539 ## docs/docs/spark-writes.md: ## @@ -101,6 +101,9 @@ Spark 3.5 added support for `WHEN NOT MATCHED BY SOURCE ... THEN ...` to update WHEN NOT MATCHED BY SOURCE THEN U

Re: [PR] refactor: Remove spawn and channel inside arrow reader [iceberg-rust]

2024-12-15 Thread via GitHub
Xuanwo commented on PR #806: URL: https://github.com/apache/iceberg-rust/pull/806#issuecomment-2544837741 Also cc @sdd for a look, thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Add license checker [iceberg-cpp]

2024-12-15 Thread via GitHub
Fokko commented on code in PR #10: URL: https://github.com/apache/iceberg-cpp/pull/10#discussion_r1886325420 ## .github/workflows/license_check.yml: ## @@ -0,0 +1,26 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

[PR] refactor: Remove spawn and channel inside arrow reader [iceberg-rust]

2024-12-15 Thread via GitHub
Xuanwo opened a new pull request, #806: URL: https://github.com/apache/iceberg-rust/pull/806 This PR will remove spwan and channel inside arrow reader so users can concurrently read data stream without extra cost. -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Spark: Read DVs when reading from .position_deletes table [iceberg]

2024-12-15 Thread via GitHub
nastra merged PR #11657: URL: https://github.com/apache/iceberg/pull/11657 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Core: Add TableUtil to provide access to a table's format version [iceberg]

2024-12-15 Thread via GitHub
nastra commented on code in PR #11620: URL: https://github.com/apache/iceberg/pull/11620#discussion_r1886323591 ## core/src/main/java/org/apache/iceberg/SerializableTable.java: ## @@ -158,6 +160,21 @@ public Map properties() { return properties; } + public int formatV

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.29.29 to 2.29.34 [iceberg]

2024-12-15 Thread via GitHub
nastra merged PR #11793: URL: https://github.com/apache/iceberg/pull/11793 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Core: Add missing REST endpoint definitions [iceberg]

2024-12-15 Thread via GitHub
nastra merged PR #11756: URL: https://github.com/apache/iceberg/pull/11756 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Spark: Read DVs when reading from .position_deletes table [iceberg]

2024-12-15 Thread via GitHub
nastra commented on code in PR #11657: URL: https://github.com/apache/iceberg/pull/11657#discussion_r1886261776 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/DVIterator.java: ## @@ -0,0 +1,108 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Spark: Read DVs when reading from .position_deletes table [iceberg]

2024-12-15 Thread via GitHub
nastra commented on code in PR #11657: URL: https://github.com/apache/iceberg/pull/11657#discussion_r1886258091 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/DVIterator.java: ## @@ -0,0 +1,108 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] feat: add `DataFileWriter` tests for schema and partition [iceberg-rust]

2024-12-15 Thread via GitHub
Xuanwo merged PR #768: URL: https://github.com/apache/iceberg-rust/pull/768 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [I] Extend the `DataFileWriterBuilder` tests [iceberg-rust]

2024-12-15 Thread via GitHub
Xuanwo closed issue #726: Extend the `DataFileWriterBuilder` tests URL: https://github.com/apache/iceberg-rust/issues/726 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

[I] Validation Exception: Found new conflicting delete files that can apply to records matching true [iceberg]

2024-12-15 Thread via GitHub
Jasonlaradsouza opened a new issue, #11795: URL: https://github.com/apache/iceberg/issues/11795 ### Apache Iceberg version 1.6.1 ### Query engine Spark ### Please describe the bug 🐞 I'm getting Validation Exception : Found new conflicting delete files that

Re: [PR] Parquet: Implement defaults for generic data [iceberg]

2024-12-15 Thread via GitHub
emkornfield commented on code in PR #11785: URL: https://github.com/apache/iceberg/pull/11785#discussion_r1886202200 ## data/src/test/java/org/apache/iceberg/data/parquet/TestGenericData.java: ## @@ -131,14 +136,143 @@ public void testTwoLevelList() throws IOException {

Re: [PR] Parquet: Implement defaults for generic data [iceberg]

2024-12-15 Thread via GitHub
emkornfield commented on code in PR #11785: URL: https://github.com/apache/iceberg/pull/11785#discussion_r1886202200 ## data/src/test/java/org/apache/iceberg/data/parquet/TestGenericData.java: ## @@ -131,14 +136,143 @@ public void testTwoLevelList() throws IOException {

Re: [PR] Doc: Add staus page for different implementations. [iceberg]

2024-12-15 Thread via GitHub
liurenjie1024 commented on code in PR #11772: URL: https://github.com/apache/iceberg/pull/11772#discussion_r1886199331 ## site/docs/status.md: ## @@ -0,0 +1,362 @@ +--- +title: "Implementation Status" +--- + + +# Implementations Status + +Apache iceberg now has implementations o

Re: [PR] Doc: Add staus page for different implementations. [iceberg]

2024-12-15 Thread via GitHub
liurenjie1024 commented on code in PR #11772: URL: https://github.com/apache/iceberg/pull/11772#discussion_r1886197308 ## site/docs/status.md: ## @@ -0,0 +1,362 @@ +--- +title: "Implementation Status" +--- + + +# Implementations Status + +Apache iceberg now has implementations o

Re: [PR] Doc: Add staus page for different implementations. [iceberg]

2024-12-15 Thread via GitHub
liurenjie1024 commented on code in PR #11772: URL: https://github.com/apache/iceberg/pull/11772#discussion_r1886196701 ## site/docs/status.md: ## @@ -0,0 +1,358 @@ +--- +title: "Implementation Status" +--- + + +# Implementations Status + +Apache iceberg's spec is implemented in

Re: [PR] Doc: Add staus page for different implementations. [iceberg]

2024-12-15 Thread via GitHub
liurenjie1024 commented on code in PR #11772: URL: https://github.com/apache/iceberg/pull/11772#discussion_r1886196445 ## site/docs/status.md: ## @@ -0,0 +1,358 @@ +--- +title: "Implementation Status" +--- + + +# Implementations Status + +Apache iceberg's spec is implemented in

Re: [PR] Avro: Support default values for generic data (WIP) [iceberg]

2024-12-15 Thread via GitHub
emkornfield commented on code in PR #11786: URL: https://github.com/apache/iceberg/pull/11786#discussion_r1886184490 ## core/src/main/java/org/apache/iceberg/data/avro/PlannedDataReader.java: ## @@ -0,0 +1,181 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Spec: add variant type [iceberg]

2024-12-15 Thread via GitHub
emkornfield commented on code in PR #10831: URL: https://github.com/apache/iceberg/pull/10831#discussion_r1886156061 ## format/spec.md: ## @@ -182,6 +182,21 @@ A **`list`** is a collection of values with some element type. The element field A **`map`** is a collection of key

Re: [PR] Spec: add variant type [iceberg]

2024-12-15 Thread via GitHub
emkornfield commented on code in PR #10831: URL: https://github.com/apache/iceberg/pull/10831#discussion_r1886155000 ## format/spec.md: ## @@ -178,6 +178,21 @@ A **`list`** is a collection of values with some element type. The element field A **`map`** is a collection of key

Re: [PR] Build: Don't run CI on unrelated changes [iceberg-python]

2024-12-15 Thread via GitHub
manuzhang commented on PR #1395: URL: https://github.com/apache/iceberg-python/pull/1395#issuecomment-2544562617 @kevinjqliu I think it would be better to squash commits and merge for such PR next time. -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] chore(deps): Bump crate-ci/typos from 1.28.2 to 1.28.3 [iceberg-rust]

2024-12-15 Thread via GitHub
liurenjie1024 merged PR #805: URL: https://github.com/apache/iceberg-rust/pull/805 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [I] Ensure Partition Specs can only contain primitive types [iceberg-rust]

2024-12-15 Thread via GitHub
liurenjie1024 commented on issue #718: URL: https://github.com/apache/iceberg-rust/issues/718#issuecomment-2544477957 Instead of ensuring primitive types only, should we add check in `Transform`, e.g. check each transform supports targeting type? -- This is an automated message from the A

Re: [I] Snapshot Testing for Integration Tests [iceberg-rust]

2024-12-15 Thread via GitHub
ZENOTME commented on issue #803: URL: https://github.com/apache/iceberg-rust/issues/803#issuecomment-250914 > @ZENOTME you mentioned: > > > A creative idea is to support Avro format files, allowing us to create snapshots of the entire Iceberg metadata, which can then be used for q

Re: [PR] feat: support `arrow_struct_to_iceberg_struct` [iceberg-rust]

2024-12-15 Thread via GitHub
liurenjie1024 commented on code in PR #731: URL: https://github.com/apache/iceberg-rust/pull/731#discussion_r1886086140 ## crates/iceberg/src/arrow/value.rs: ## @@ -0,0 +1,874 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agree

Re: [PR] feat: support `arrow_struct_to_iceberg_struct` [iceberg-rust]

2024-12-15 Thread via GitHub
liurenjie1024 commented on code in PR #731: URL: https://github.com/apache/iceberg-rust/pull/731#discussion_r1886072909 ## crates/iceberg/src/arrow/value.rs: ## @@ -0,0 +1,1015 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agre

Re: [PR] Feat: support aliyun oss backend. [iceberg-go]

2024-12-15 Thread via GitHub
divinerapier commented on PR #216: URL: https://github.com/apache/iceberg-go/pull/216#issuecomment-2544416465 @zeroshade Automated integration testing is indeed crucial. However, unfortunately, I haven't yet found a reasonable and convenient way to access OSS services in the integra

Re: [I] Implement nan_value_counts && distinct_counts metrics in parquet writer [iceberg-rust]

2024-12-15 Thread via GitHub
liurenjie1024 commented on issue #417: URL: https://github.com/apache/iceberg-rust/issues/417#issuecomment-2544387385 > Yup, let me work out a PR for nan_values first, also just confirming is the method mentioned by me up above correct for nan_values? Yes, exactly. -- This is an au

[I] Why the metric values are all N/A on the spark webui? [iceberg]

2024-12-15 Thread via GitHub
Akeron-Zhu opened a new issue, #11794: URL: https://github.com/apache/iceberg/issues/11794 I had run a sql on spark 3.5 with iceberg 1.6.1, as the picture show, the metric values are all N/A except the "number of rows". Did I not configure spark or iceberg properly? ![spark_3 5_iceberg_1

Re: [I] Schema Deserialization Ignores Field initial-default and write-default Values [iceberg-python]

2024-12-15 Thread via GitHub
paulcichonski commented on issue #1431: URL: https://github.com/apache/iceberg-python/issues/1431#issuecomment-2544246583 Makes sense, I was wondering if it was a v3 thing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] [Discussion] googletest(gtest) or Catch2 [iceberg-cpp]

2024-12-15 Thread via GitHub
lidavidm commented on issue #12: URL: https://github.com/apache/iceberg-cpp/issues/12#issuecomment-2544236270 Catch2 does have rich errors, but I'm rather rattled by how it implements it: ![image](https://github.com/user-attachments/assets/f4090d7c-28bf-4286-a9ae-feebdee28a1d)

Re: [I] The `between` keyword seems to be missing from pyiceberg expression [iceberg-python]

2024-12-15 Thread via GitHub
github-actions[bot] commented on issue #792: URL: https://github.com/apache/iceberg-python/issues/792#issuecomment-2544227939 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apac

Re: [I] The `between` keyword seems to be missing from pyiceberg expression [iceberg-python]

2024-12-15 Thread via GitHub
github-actions[bot] closed issue #792: The `between` keyword seems to be missing from pyiceberg expression URL: https://github.com/apache/iceberg-python/issues/792 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [I] ORC file format support [iceberg-python]

2024-12-15 Thread via GitHub
github-actions[bot] commented on issue #20: URL: https://github.com/apache/iceberg-python/issues/20#issuecomment-2544227976 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] ORC file format support [iceberg-python]

2024-12-15 Thread via GitHub
github-actions[bot] closed issue #20: ORC file format support URL: https://github.com/apache/iceberg-python/issues/20 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

Re: [PR] Flink: Backport #11662 Fix range distribution npe when value is null to Flink 1.18 and 1.19 [iceberg]

2024-12-15 Thread via GitHub
Guosmilesmile commented on PR #11745: URL: https://github.com/apache/iceberg/pull/11745#issuecomment-2544225807 @pvary Hi Peter, Is there anything that needs to be modified in this backport? Could you please take some time to review it? Thank you very much! -- This is an automated mess

Re: [PR] GCP: Add Iceberg Catalog for GCP BigQuery Metastore [iceberg]

2024-12-15 Thread via GitHub
github-actions[bot] commented on PR #11039: URL: https://github.com/apache/iceberg/pull/11039#issuecomment-2544225522 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] Broken links in Spark Writes documentation [iceberg]

2024-12-15 Thread via GitHub
github-actions[bot] closed issue #10409: Broken links in Spark Writes documentation URL: https://github.com/apache/iceberg/issues/10409 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Flink: make `StatisticsOrRecord` to be correctly serialized and deser… [iceberg]

2024-12-15 Thread via GitHub
github-actions[bot] commented on PR #11557: URL: https://github.com/apache/iceberg/pull/11557#issuecomment-2544225556 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] Should we remove the use of versionHintFile from the entire FileSystemCatalog? [iceberg]

2024-12-15 Thread via GitHub
github-actions[bot] commented on issue #10427: URL: https://github.com/apache/iceberg/issues/10427#issuecomment-2544225450 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Should we remove the use of versionHintFile from the entire FileSystemCatalog? [iceberg]

2024-12-15 Thread via GitHub
github-actions[bot] closed issue #10427: Should we remove the use of versionHintFile from the entire FileSystemCatalog? URL: https://github.com/apache/iceberg/issues/10427 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [I] Questions on some requirement in view spec [iceberg]

2024-12-15 Thread via GitHub
github-actions[bot] commented on issue #10410: URL: https://github.com/apache/iceberg/issues/10410#issuecomment-2544225420 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Make ManifestEntry and ManifestReader.liveEntries() as public [iceberg]

2024-12-15 Thread via GitHub
github-actions[bot] commented on issue #10425: URL: https://github.com/apache/iceberg/issues/10425#issuecomment-2544225435 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Make ManifestEntry and ManifestReader.liveEntries() as public [iceberg]

2024-12-15 Thread via GitHub
github-actions[bot] closed issue #10425: Make ManifestEntry and ManifestReader.liveEntries() as public URL: https://github.com/apache/iceberg/issues/10425 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [I] Questions on some requirement in view spec [iceberg]

2024-12-15 Thread via GitHub
github-actions[bot] closed issue #10410: Questions on some requirement in view spec URL: https://github.com/apache/iceberg/issues/10410 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [I] Broken links in Spark Writes documentation [iceberg]

2024-12-15 Thread via GitHub
github-actions[bot] commented on issue #10409: URL: https://github.com/apache/iceberg/issues/10409#issuecomment-2544225403 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [PR] Fix `Table.scan` to enable case sensitive argument [iceberg-python]

2024-12-15 Thread via GitHub
jiakai-li commented on code in PR #1423: URL: https://github.com/apache/iceberg-python/pull/1423#discussion_r1885864835 ## pyiceberg/table/__init__.py: ## @@ -268,12 +268,10 @@ def _apply(self, updates: Tuple[TableUpdate, ...], requirements: Tuple[TableRequ return se

Re: [PR] Fix `Table.scan` to enable case sensitive argument [iceberg-python]

2024-12-15 Thread via GitHub
jiakai-li commented on code in PR #1423: URL: https://github.com/apache/iceberg-python/pull/1423#discussion_r1885864835 ## pyiceberg/table/__init__.py: ## @@ -268,12 +268,10 @@ def _apply(self, updates: Tuple[TableUpdate, ...], requirements: Tuple[TableRequ return se

Re: [PR] REST: Don't reset snapshotLog when replacing table [iceberg]

2024-12-15 Thread via GitHub
ebyhr commented on code in PR #11779: URL: https://github.com/apache/iceberg/pull/11779#discussion_r1885915752 ## core/src/main/java/org/apache/iceberg/MetadataUpdateParser.java: ## @@ -548,7 +552,8 @@ private static MetadataUpdate readSetSnapshotRef(JsonNode node) { priva

Re: [PR] Fix `Table.scan` to enable case sensitive argument [iceberg-python]

2024-12-15 Thread via GitHub
jiakai-li commented on code in PR #1423: URL: https://github.com/apache/iceberg-python/pull/1423#discussion_r1885864835 ## pyiceberg/table/__init__.py: ## @@ -268,12 +268,10 @@ def _apply(self, updates: Tuple[TableUpdate, ...], requirements: Tuple[TableRequ return se

Re: [PR] Fix `Table.scan` to enable case sensitive argument [iceberg-python]

2024-12-15 Thread via GitHub
jiakai-li commented on code in PR #1423: URL: https://github.com/apache/iceberg-python/pull/1423#discussion_r1885864835 ## pyiceberg/table/__init__.py: ## @@ -268,12 +268,10 @@ def _apply(self, updates: Tuple[TableUpdate, ...], requirements: Tuple[TableRequ return se

Re: [PR] Fix `Table.scan` to enable case sensitive argument [iceberg-python]

2024-12-15 Thread via GitHub
jiakai-li commented on code in PR #1423: URL: https://github.com/apache/iceberg-python/pull/1423#discussion_r1885864835 ## pyiceberg/table/__init__.py: ## @@ -268,12 +268,10 @@ def _apply(self, updates: Tuple[TableUpdate, ...], requirements: Tuple[TableRequ return se

Re: [PR] Manifest list encryption [iceberg]

2024-12-15 Thread via GitHub
caushie-akamai commented on PR #7770: URL: https://github.com/apache/iceberg/pull/7770#issuecomment-2544057649 Hi! Are there any plans to have this feature merged in 1.8.0? This would be extremely helpful -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [I] Schema Deserialization Ignores Field initial-default and write-default Values [iceberg-python]

2024-12-15 Thread via GitHub
Fokko commented on issue #1431: URL: https://github.com/apache/iceberg-python/issues/1431#issuecomment-2544049590 Thanks for noticing and picking this up right away @paulcichonski 🙌 There is more context around this I'd like to share. Officially this is supported only in V3 and onward (th

Re: [PR] feat: add `DataFileWriter` tests for schema and partition [iceberg-rust]

2024-12-15 Thread via GitHub
jonathanc-n commented on PR #768: URL: https://github.com/apache/iceberg-rust/pull/768#issuecomment-2544037562 @Fokko Sorry for the wait, was on a break. Merge errors should be fixed! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [I] Schema Deserialization Ignores Field initial-default and write-default Values [iceberg-python]

2024-12-15 Thread via GitHub
paulcichonski commented on issue #1431: URL: https://github.com/apache/iceberg-python/issues/1431#issuecomment-2544029289 @kevinjqliu sure thing, gave it a shot in https://github.com/apache/iceberg-python/pull/1432 . -- This is an automated message from the Apache Git Service. To respon

[PR] Deserialize NestedField initial-default and write-default Attributes [iceberg-python]

2024-12-15 Thread via GitHub
paulcichonski opened a new pull request, #1432: URL: https://github.com/apache/iceberg-python/pull/1432 Ensures that these attributes are correctly applied to the NestedField when reading an Iceberg schema json file. Fixes #1431 -- This is an automated message from the Apache Git

Re: [I] Store min/max stats per column per partition [iceberg]

2024-12-15 Thread via GitHub
osscm commented on issue #11083: URL: https://github.com/apache/iceberg/issues/11083#issuecomment-2543996612 +1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscri

Re: [I] Use Min, Max, and NumOfNulls from Manifest Files for Spark Column Stats [iceberg]

2024-12-15 Thread via GitHub
osscm commented on issue #10791: URL: https://github.com/apache/iceberg/issues/10791#issuecomment-2543996345 >I think we could introduce a property that allows users to choose whether to calculate the statistics on the fly. this looks a good option. Not sure if we have discussed or

Re: [PR] Fix ParallelIterable deadlock [iceberg]

2024-12-15 Thread via GitHub
osscm commented on code in PR #11781: URL: https://github.com/apache/iceberg/pull/11781#discussion_r1885784288 ## core/src/main/java/org/apache/iceberg/util/ParallelIterable.java: ## @@ -257,17 +257,17 @@ private static class Task implements Supplier>>, Closeable { @Overri

Re: [I] [Discussion] googletest(gtest) or Catch2 [iceberg-cpp]

2024-12-15 Thread via GitHub
pitrou commented on issue #12: URL: https://github.com/apache/iceberg-cpp/issues/12#issuecomment-2543934975 As I wrote on your PR, my main concern with GTest is compilation times, but I don't know if Catch2 would improve things on this axis. One question I have about Catch2 is whether

Re: [PR] Integrate Test Framework [iceberg-cpp]

2024-12-15 Thread via GitHub
pitrou commented on PR #13: URL: https://github.com/apache/iceberg-cpp/pull/13#issuecomment-2543934115 Thanks @zhjwpku ! What I would be interested to know is the compile and link times, respectively, for both GTest and Catch2. My experience in Arrow is that GTest's main header `gt

Re: [PR] Parquet: Implement defaults for generic data [iceberg]

2024-12-15 Thread via GitHub
Fokko commented on code in PR #11785: URL: https://github.com/apache/iceberg/pull/11785#discussion_r1885729020 ## data/src/test/java/org/apache/iceberg/data/parquet/TestGenericData.java: ## @@ -131,14 +136,143 @@ public void testTwoLevelList() throws IOException { .

Re: [I] Schema Deserialization Ignores Field initial-default and write-default Values [iceberg-python]

2024-12-15 Thread via GitHub
kevinjqliu commented on issue #1431: URL: https://github.com/apache/iceberg-python/issues/1431#issuecomment-2543921592 Hi @paulcichonski, thanks for reporting this issue! It does indeed seem like a bug, and your proposed fix looks reasonable. Would you be interested in contributing the fix

Re: [I] Implement `namespace_exists` function on the REST Catalog [iceberg-python]

2024-12-15 Thread via GitHub
kevinjqliu commented on issue #1430: URL: https://github.com/apache/iceberg-python/issues/1430#issuecomment-2543920101 @AhmedNader42 assigned to you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] Discussion: make DataFile Serializable && Deserializable [iceberg-rust]

2024-12-15 Thread via GitHub
ZENOTME commented on issue #774: URL: https://github.com/apache/iceberg-rust/issues/774#issuecomment-2543919300 > Hi, thank you @ZENOTME for starting this discussion. I prefer to make `DataFile` itself serializable. Maybe we can do this samething for `DataFile` like `TableMetadata`: `TableM

Re: [PR] feat!: Remove `BoundPartitionSpec` [iceberg-rust]

2024-12-15 Thread via GitHub
Xuanwo merged PR #771: URL: https://github.com/apache/iceberg-rust/pull/771 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [I] Why shouldn't we return an `UnboundPartitionSpec` instead? [iceberg-rust]

2024-12-15 Thread via GitHub
Xuanwo closed issue #694: Why shouldn't we return an `UnboundPartitionSpec` instead? URL: https://github.com/apache/iceberg-rust/issues/694 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] feat!: Remove `BoundPartitionSpec` [iceberg-rust]

2024-12-15 Thread via GitHub
Xuanwo commented on PR #771: URL: https://github.com/apache/iceberg-rust/pull/771#issuecomment-2543915869 Thank you @c-thiel for working on this, and thank @Fokko, @liurenjie1024 and @sdd for the review. We have waited for this for so long, let's move! -- This is an automated message from

Re: [PR] Build: Bump nessie from 0.101.0 to 0.101.2 [iceberg]

2024-12-15 Thread via GitHub
Fokko merged PR #11791: URL: https://github.com/apache/iceberg/pull/11791 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Fix `Table.scan` to enable case sensitive argument [iceberg-python]

2024-12-15 Thread via GitHub
Fokko commented on code in PR #1423: URL: https://github.com/apache/iceberg-python/pull/1423#discussion_r1885721293 ## pyiceberg/table/__init__.py: ## @@ -470,17 +470,23 @@ def overwrite( for data_file in data_files: update_snapshot.append_d

Re: [I] Dectect schema evolution or partition evolution for append DataFile [iceberg-rust]

2024-12-15 Thread via GitHub
Fokko commented on issue #777: URL: https://github.com/apache/iceberg-rust/issues/777#issuecomment-2543892622 I've actually started on adding some more conversions, good beginner task for a Rust novice like me :) -- This is an automated message from the Apache Git Service. To respond to t

[PR] chore(deps): Bump crate-ci/typos from 1.28.2 to 1.28.3 [iceberg-rust]

2024-12-15 Thread via GitHub
dependabot[bot] opened a new pull request, #805: URL: https://github.com/apache/iceberg-rust/pull/805 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.28.2 to 1.28.3. Release notes Sourced from https://github.com/crate-ci/typos/releases";>crate-ci/typos's releases.

Re: [PR] REST: Don't reset snapshotLog when replacing table [iceberg]

2024-12-15 Thread via GitHub
ebyhr commented on code in PR #11779: URL: https://github.com/apache/iceberg/pull/11779#discussion_r1885687470 ## core/src/main/java/org/apache/iceberg/MetadataUpdateParser.java: ## @@ -417,6 +420,7 @@ private static void writeSetSnapshotRef(MetadataUpdate.SetSnapshotRef update

Re: [I] Implement `namespace_exists` function on the REST Catalog [iceberg-python]

2024-12-15 Thread via GitHub
AhmedNader42 commented on issue #1430: URL: https://github.com/apache/iceberg-python/issues/1430#issuecomment-2543844223 Would love to implement this functionality if no one is assigned -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] feat!: Remove `BoundPartitionSpec` [iceberg-rust]

2024-12-15 Thread via GitHub
c-thiel commented on PR #771: URL: https://github.com/apache/iceberg-rust/pull/771#issuecomment-2543820634 @liurenjie1024 conflicts resolved. Had to slightly change a function signature: https://github.com/apache/iceberg-rust/pull/771/files#diff-8389535350ef7cddc266dfd18d589a978643da033

Re: [PR] feat!: Remove `BoundPartitionSpec` [iceberg-rust]

2024-12-15 Thread via GitHub
sdd commented on PR #771: URL: https://github.com/apache/iceberg-rust/pull/771#issuecomment-2543833200 Thanks @c-thiel, those comments are great, looks good to me :-) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[PR] feat: Add RemovePartitionSpecs table update [iceberg-rust]

2024-12-15 Thread via GitHub
c-thiel opened a new pull request, #804: URL: https://github.com/apache/iceberg-rust/pull/804 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

Re: [PR] Materialized View Spec [iceberg]

2024-12-15 Thread via GitHub
JanKaul commented on code in PR #11041: URL: https://github.com/apache/iceberg/pull/11041#discussion_r1885605556 ## format/view-spec.md: ## @@ -82,9 +98,12 @@ Each version in `versions` is a struct with the following fields: | _required_ | `representations` | A list of [re

Re: [PR] Materialized View Spec [iceberg]

2024-12-15 Thread via GitHub
szehon-ho commented on PR #11041: URL: https://github.com/apache/iceberg/pull/11041#issuecomment-2543646586 > > If i understand correctly @wmoustafa comment on the mailing list, then there is some ambiguity here for what to put, if the same table in expressed in the various forms (catalog.d

Re: [PR] Doc: Do Not Modify the Source Data Table During MergeIntoCommand Exec… [iceberg]

2024-12-15 Thread via GitHub
BsoBird commented on PR #11787: URL: https://github.com/apache/iceberg/pull/11787#issuecomment-2543564046 @RussellSpitzer Hello Sir,I have modified the SQL used in our production environment according to your suggestions, and currently, it appears that the issue of data loss has indeed

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.46 to 1.17.47 [iceberg-go]

2024-12-15 Thread via GitHub
dependabot[bot] opened a new pull request, #229: URL: https://github.com/apache/iceberg-go/pull/229 Bumps [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) from 1.17.46 to 1.17.47. Commits https://github.com/aws/aws-sdk-go-v2/commit/776903f3dd720

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.28.5 to 1.28.6 [iceberg-go]

2024-12-15 Thread via GitHub
dependabot[bot] opened a new pull request, #228: URL: https://github.com/apache/iceberg-go/pull/228 Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.28.5 to 1.28.6. Commits https://github.com/aws/aws-sdk-go-v2/commit/776903f3dd7208803912

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/glue from 1.102.0 to 1.104.0 [iceberg-go]

2024-12-15 Thread via GitHub
dependabot[bot] opened a new pull request, #227: URL: https://github.com/apache/iceberg-go/pull/227 Bumps [github.com/aws/aws-sdk-go-v2/service/glue](https://github.com/aws/aws-sdk-go-v2) from 1.102.0 to 1.104.0. Changelog Sourced from https://github.com/aws/aws-sdk-go-v2/blob/ser

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/glue from 1.102.0 to 1.103.0 [iceberg-go]

2024-12-15 Thread via GitHub
dependabot[bot] commented on PR #224: URL: https://github.com/apache/iceberg-go/pull/224#issuecomment-2543546231 Superseded by #227. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/glue from 1.102.0 to 1.103.0 [iceberg-go]

2024-12-15 Thread via GitHub
dependabot[bot] closed pull request #224: build(deps): bump github.com/aws/aws-sdk-go-v2/service/glue from 1.102.0 to 1.103.0 URL: https://github.com/apache/iceberg-go/pull/224 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and