Re: [PR] AddPartitionSpec: A new way to set new partition specs [iceberg]

2024-08-05 Thread via GitHub
shanielh commented on code in PR #10737: URL: https://github.com/apache/iceberg/pull/10737#discussion_r1703630708 ## core/src/main/java/org/apache/iceberg/BaseUpdatePartitionSpec.java: ## @@ -62,15 +62,20 @@ class BaseUpdatePartitionSpec implements UpdatePartitionSpec { priv

Re: [PR] Build: Bump nessie from 0.94.2 to 0.94.4 [iceberg]

2024-08-05 Thread via GitHub
Fokko commented on PR #10869: URL: https://github.com/apache/iceberg/pull/10869#issuecomment-2268322497 Thanks for the review @ajantha-bhat 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Build: Bump org.xerial:sqlite-jdbc from 3.46.0.0 to 3.46.0.1 [iceberg]

2024-08-05 Thread via GitHub
Fokko merged PR #10871: URL: https://github.com/apache/iceberg/pull/10871 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.26.25 to 2.26.29 [iceberg]

2024-08-05 Thread via GitHub
Fokko merged PR #10866: URL: https://github.com/apache/iceberg/pull/10866 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Build: Bump org.apache.commons:commons-compress from 1.26.0 to 1.26.2 [iceberg]

2024-08-05 Thread via GitHub
Fokko merged PR #10868: URL: https://github.com/apache/iceberg/pull/10868 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Build: Bump mkdocs-material from 9.5.30 to 9.5.31 [iceberg]

2024-08-05 Thread via GitHub
Fokko merged PR #10863: URL: https://github.com/apache/iceberg/pull/10863 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Build: Fix Scala compilation [iceberg]

2024-08-05 Thread via GitHub
nastra merged PR #10860: URL: https://github.com/apache/iceberg/pull/10860 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Bump to Parquet 1.14.1 [iceberg]

2024-08-05 Thread via GitHub
Fokko commented on PR #10209: URL: https://github.com/apache/iceberg/pull/10209#issuecomment-2268335571 @findepi Looks like some false positive from the shadow plugin that's unable to coop with the Java-version specific files. Have to dig deeper into this. -- This is an automated message

Re: [PR] Spec: Add expiry time config to REST table load [iceberg]

2024-08-05 Thread via GitHub
nastra commented on code in PR #10873: URL: https://github.com/apache/iceberg/pull/10873#discussion_r1703643211 ## open-api/rest-catalog-open-api.py: ## @@ -1112,6 +1112,7 @@ class LoadTableResult(BaseModel): - `s3.secret-access-key`: secret for credentials that provide ac

Re: [I] Spark: Add read/write support for UUIDs from bytes [iceberg]

2024-08-05 Thread via GitHub
nastra commented on issue #10635: URL: https://github.com/apache/iceberg/issues/10635#issuecomment-2268346007 @anuragmantri I believe this is the correct place to do the casting. Spark itself doesn't support UUID as a type and so you can only represent it as a string when you write a UUID.

Re: [I] Review new FormatStringAnnotation error-prone check [iceberg]

2024-08-05 Thread via GitHub
nastra closed issue #10854: Review new FormatStringAnnotation error-prone check URL: https://github.com/apache/iceberg/issues/10854 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Build: Enable FormatStringAnnotation error-prone check [iceberg]

2024-08-05 Thread via GitHub
nastra merged PR #10856: URL: https://github.com/apache/iceberg/pull/10856 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Update metadata-log for non-rest catalogs [iceberg-python]

2024-08-05 Thread via GitHub
HonahX commented on code in PR #977: URL: https://github.com/apache/iceberg-python/pull/977#discussion_r1703643942 ## pyiceberg/table/__init__.py: ## @@ -224,6 +225,9 @@ class TableProperties: MANIFEST_MERGE_ENABLED = "commit.manifest-merge.enabled" MANIFEST_MERGE_ENAB

Re: [PR] Update metadata-log for non-rest catalogs [iceberg-python]

2024-08-05 Thread via GitHub
soumya-ghosh commented on code in PR #977: URL: https://github.com/apache/iceberg-python/pull/977#discussion_r1703667518 ## pyiceberg/table/__init__.py: ## @@ -224,6 +225,9 @@ class TableProperties: MANIFEST_MERGE_ENABLED = "commit.manifest-merge.enabled" MANIFEST_MERG

Re: [PR] Pyarrow IO property for configuring large v small types on read [iceberg-python]

2024-08-05 Thread via GitHub
HonahX commented on PR #986: URL: https://github.com/apache/iceberg-python/pull/986#issuecomment-2268417308 @sungwy Thanks for working on this! It seems we also need to update `schema_to_pyarrow`/`_cast_if_needed` to honor the new property. Otherwise https://github.com/apache/iceb

Re: [PR] Build: Enable FormatStringAnnotation error-prone check [iceberg]

2024-08-05 Thread via GitHub
RussellSpitzer commented on PR #10856: URL: https://github.com/apache/iceberg/pull/10856#issuecomment-2268432684 Yeah I thought I had cleaned these up a while ago, but glad to have it turned on. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] SQL Catalog [iceberg-rust]

2024-08-05 Thread via GitHub
Xuanwo commented on PR #503: URL: https://github.com/apache/iceberg-rust/pull/503#issuecomment-2268443288 Hi @callum-ryan, I believe the entire SQL Catalog is quite a large project. Although @JanKaul and you have put in a lot of effort, there are still many areas that need improvement befor

Re: [PR] Spec: Add expiry time config to REST table load [iceberg]

2024-08-05 Thread via GitHub
munendrasn commented on code in PR #10873: URL: https://github.com/apache/iceberg/pull/10873#discussion_r1703815874 ## open-api/rest-catalog-open-api.py: ## @@ -1112,6 +1112,7 @@ class LoadTableResult(BaseModel): - `s3.secret-access-key`: secret for credentials that provid

[I] Improvement: reuse the tcp connection when plan files [iceberg-rust]

2024-08-05 Thread via GitHub
chenzl25 opened a new issue, #522: URL: https://github.com/apache/iceberg-rust/issues/522 Considering I am trying to read an iceberg table from S3. Currently, `plan_files()` seems unable to reuse the TCP connection for HTTP requests. It will lead to a relatively high latency. I am not sure

Re: [PR] Spec: Add expiry time config to REST table load [iceberg]

2024-08-05 Thread via GitHub
nastra commented on code in PR #10873: URL: https://github.com/apache/iceberg/pull/10873#discussion_r1703864919 ## open-api/rest-catalog-open-api.py: ## @@ -1112,6 +1112,7 @@ class LoadTableResult(BaseModel): - `s3.secret-access-key`: secret for credentials that provide ac

Re: [PR] Spec: Add expiry time config to REST table load [iceberg]

2024-08-05 Thread via GitHub
munendrasn commented on code in PR #10873: URL: https://github.com/apache/iceberg/pull/10873#discussion_r1703897063 ## open-api/rest-catalog-open-api.py: ## @@ -1112,6 +1112,7 @@ class LoadTableResult(BaseModel): - `s3.secret-access-key`: secret for credentials that provid

Re: [PR] Spec: Add expiry time config to REST table load [iceberg]

2024-08-05 Thread via GitHub
nastra commented on PR #10873: URL: https://github.com/apache/iceberg/pull/10873#issuecomment-2268756788 FYI there is also https://github.com/apache/iceberg/pull/10722 to standardize the different credentials -- This is an automated message from the Apache Git Service. To respond to the m

Re: [PR] Core: Use %2E as namespace separator instead of %1F [iceberg]

2024-08-05 Thread via GitHub
nastra closed pull request #10843: Core: Use %2E as namespace separator instead of %1F URL: https://github.com/apache/iceberg/pull/10843 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] `DataFrameWriterV2` fails to write to Iceberg table with schema having nested array[struct] fields [iceberg]

2024-08-05 Thread via GitHub
pasha-i commented on issue #10522: URL: https://github.com/apache/iceberg/issues/10522#issuecomment-2268792654 Hey @RussellSpitzer, yes, we do -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Kafka Connect: Stop commits on terminated coordinator [iceberg]

2024-08-05 Thread via GitHub
nk1506 commented on code in PR #10814: URL: https://github.com/apache/iceberg/pull/10814#discussion_r1703995263 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/Coordinator.java: ## @@ -218,6 +220,10 @@ private void commitToTable( .fil

Re: [PR] DO-NOT-MERGE: Jackson access issue [iceberg]

2024-08-05 Thread via GitHub
pan3793 commented on PR #10460: URL: https://github.com/apache/iceberg/pull/10460#issuecomment-226543 @pvary Unfortunately, I think this is not an issue specify to Hive 2.3.10, the whole Hive 2.3 series have issues on loading Jackson 2.x classes. > Do we have a clear understanding

Re: [PR] Fix: accept empty arrays in struct field lookup [iceberg-python]

2024-08-05 Thread via GitHub
ndrluis commented on PR #997: URL: https://github.com/apache/iceberg-python/pull/997#issuecomment-2268945753 @grobgl Thank you for the fix. Could you add a integration test to guarantee the expected behavior? -- This is an automated message from the Apache Git Service. To respond to the m

Re: [PR] Core: Use encoding/decoding methods for namespaces and deprecate Splitter/Joiner [iceberg]

2024-08-05 Thread via GitHub
nastra commented on PR #10858: URL: https://github.com/apache/iceberg/pull/10858#issuecomment-2268969067 thanks for the review @Fokko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Core: Use encoding/decoding methods for namespaces and deprecate Splitter/Joiner [iceberg]

2024-08-05 Thread via GitHub
nastra merged PR #10858: URL: https://github.com/apache/iceberg/pull/10858 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Spec: Add expiry time config to REST table load [iceberg]

2024-08-05 Thread via GitHub
munendrasn commented on PR #10873: URL: https://github.com/apache/iceberg/pull/10873#issuecomment-2268991600 Thank you @nastra, for sharing the PR. The inspiration for the current PR is both https://github.com/apache/iceberg/pull/10722 and https://github.com/apache/iceberg/pull/10576 (asked

Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

2024-08-05 Thread via GitHub
liurenjie1024 commented on code in PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#discussion_r1704049202 ## crates/iceberg/src/spec/partition.rs: ## @@ -117,22 +144,312 @@ pub struct UnboundPartitionField { } /// Unbound partition spec can be built without a s

Re: [I] Improvement: reuse the tcp connection when plan files [iceberg-rust]

2024-08-05 Thread via GitHub
liurenjie1024 commented on issue #522: URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2269007589 > Considering I am trying to read an iceberg table from S3. Currently, `plan_files()` seems unable to reuse the TCP connection for HTTP requests. It will lead to a relativel

Re: [I] Improvement: reuse the tcp connection when plan files [iceberg-rust]

2024-08-05 Thread via GitHub
Xuanwo commented on issue #522: URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2269056564 Hi, OpenDAL handles those connection-related tasks (by reqwest). Currently, FileIO builds new operators every time: https://github.com/apache/iceberg-rust/blob/4083f8

Re: [PR] feat: Establish subproject pyiceberg_core [iceberg-rust]

2024-08-05 Thread via GitHub
liurenjie1024 commented on code in PR #518: URL: https://github.com/apache/iceberg-rust/pull/518#discussion_r1704119358 ## Cargo.toml: ## @@ -18,11 +18,14 @@ [workspace] resolver = "2" members = [ - "crates/catalog/*", - "crates/examples", - "crates/iceberg", - "crates/in

Re: [PR] feat: support for gcs storage [iceberg-rust]

2024-08-05 Thread via GitHub
liurenjie1024 commented on code in PR #520: URL: https://github.com/apache/iceberg-rust/pull/520#discussion_r1704136142 ## crates/iceberg/src/io/storage.rs: ## @@ -117,7 +125,24 @@ impl Storage { )) } } -#[cfg(all(no

Re: [PR] feat: support for gcs storage [iceberg-rust]

2024-08-05 Thread via GitHub
jdockerty commented on code in PR #520: URL: https://github.com/apache/iceberg-rust/pull/520#discussion_r1704151685 ## crates/iceberg/Cargo.toml: ## @@ -29,12 +29,13 @@ license = { workspace = true } keywords = ["iceberg"] [features] -default = ["storage-memory", "storage-fs

Re: [PR] feat: support for gcs storage [iceberg-rust]

2024-08-05 Thread via GitHub
jdockerty commented on code in PR #520: URL: https://github.com/apache/iceberg-rust/pull/520#discussion_r1704151685 ## crates/iceberg/Cargo.toml: ## @@ -29,12 +29,13 @@ license = { workspace = true } keywords = ["iceberg"] [features] -default = ["storage-memory", "storage-fs

Re: [PR] feat: support for gcs storage [iceberg-rust]

2024-08-05 Thread via GitHub
jdockerty commented on code in PR #520: URL: https://github.com/apache/iceberg-rust/pull/520#discussion_r1704182016 ## crates/iceberg/src/io/storage.rs: ## @@ -117,7 +125,24 @@ impl Storage { )) } } -#[cfg(all(not(fe

Re: [PR] feat: support for gcs storage [iceberg-rust]

2024-08-05 Thread via GitHub
jdockerty commented on code in PR #520: URL: https://github.com/apache/iceberg-rust/pull/520#discussion_r1704188779 ## crates/iceberg/tests/file_io_gcs_test.rs: ## @@ -0,0 +1,103 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license ag

Re: [PR] Update metadata-log for non-rest catalogs [iceberg-python]

2024-08-05 Thread via GitHub
soumya-ghosh commented on code in PR #977: URL: https://github.com/apache/iceberg-python/pull/977#discussion_r1704190963 ## pyiceberg/table/__init__.py: ## @@ -224,6 +225,9 @@ class TableProperties: MANIFEST_MERGE_ENABLED = "commit.manifest-merge.enabled" MANIFEST_MERG

[I] Athena Iceberg does not delete orphan files [iceberg]

2024-08-05 Thread via GitHub
Smotrov opened a new issue, #10878: URL: https://github.com/apache/iceberg/issues/10878 ### Apache Iceberg version 1.6.0 (latest release) ### Query engine Athena ### Please describe the bug 🐞 I have Athena Iceberg table. The table has 2 partitions.

Re: [PR] feat: support for gcs storage [iceberg-rust]

2024-08-05 Thread via GitHub
Xuanwo commented on code in PR #520: URL: https://github.com/apache/iceberg-rust/pull/520#discussion_r1704167334 ## crates/iceberg/src/io/storage_gcs.rs: ## @@ -0,0 +1,74 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements

Re: [PR] feat: support for gcs storage [iceberg-rust]

2024-08-05 Thread via GitHub
Xuanwo commented on code in PR #520: URL: https://github.com/apache/iceberg-rust/pull/520#discussion_r1704197839 ## crates/iceberg/tests/file_io_gcs_test.rs: ## @@ -0,0 +1,103 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agree

Re: [PR] feat: Establish subproject pyiceberg_core [iceberg-rust]

2024-08-05 Thread via GitHub
Xuanwo commented on code in PR #518: URL: https://github.com/apache/iceberg-rust/pull/518#discussion_r1704205451 ## Cargo.toml: ## @@ -18,11 +18,14 @@ [workspace] resolver = "2" members = [ - "crates/catalog/*", - "crates/examples", - "crates/iceberg", - "crates/integrati

Re: [PR] feat: Establish subproject pyiceberg_core [iceberg-rust]

2024-08-05 Thread via GitHub
Xuanwo commented on code in PR #518: URL: https://github.com/apache/iceberg-rust/pull/518#discussion_r1704205451 ## Cargo.toml: ## @@ -18,11 +18,14 @@ [workspace] resolver = "2" members = [ - "crates/catalog/*", - "crates/examples", - "crates/iceberg", - "crates/integrati

Re: [PR] Deprecate Redundant Identifier Support in TableIdentifier, and row_filter [iceberg-python]

2024-08-05 Thread via GitHub
sungwy commented on PR #994: URL: https://github.com/apache/iceberg-python/pull/994#issuecomment-2269225830 > @sungwy Awesome work! > > Do you know if these changes could impact the multi-level namespace case? I believe it would be nice for us to have some integration tests with Polar

Re: [I] Athena Iceberg does not delete orphan files [iceberg]

2024-08-05 Thread via GitHub
Smotrov commented on issue #10878: URL: https://github.com/apache/iceberg/issues/10878#issuecomment-2269235051 When I manually delete from the `data` directory anything that is not listed in the `files` query result I have following error on any random select. ``` CEBERG_CANNOT_OP

Re: [PR] Simplify PrimitiveLiteral [iceberg-rust]

2024-08-05 Thread via GitHub
ZENOTME commented on code in PR #502: URL: https://github.com/apache/iceberg-rust/pull/502#discussion_r1704224086 ## crates/iceberg/src/spec/values.rs: ## @@ -65,24 +65,14 @@ pub enum PrimitiveLiteral { Float(OrderedFloat), /// Stored as 8-byte little-endian Doubl

Re: [PR] feat: support for gcs storage [iceberg-rust]

2024-08-05 Thread via GitHub
jdockerty commented on code in PR #520: URL: https://github.com/apache/iceberg-rust/pull/520#discussion_r1704228696 ## crates/iceberg/src/io/storage_gcs.rs: ## @@ -0,0 +1,74 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreeme

Re: [PR] feat: support for gcs storage [iceberg-rust]

2024-08-05 Thread via GitHub
jdockerty commented on code in PR #520: URL: https://github.com/apache/iceberg-rust/pull/520#discussion_r1704236572 ## crates/iceberg/src/io/storage_gcs.rs: ## @@ -0,0 +1,74 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreeme

Re: [PR] core: support support move a column with same name after rename column [iceberg]

2024-08-05 Thread via GitHub
FANNG1 commented on code in PR #10862: URL: https://github.com/apache/iceberg/pull/10862#discussion_r1704240921 ## core/src/main/java/org/apache/iceberg/SchemaUpdate.java: ## @@ -201,6 +204,8 @@ public UpdateSchema renameColumn(String name, String newName) { Types.NestedFi

Re: [PR] feat: support for gcs storage [iceberg-rust]

2024-08-05 Thread via GitHub
jdockerty commented on code in PR #520: URL: https://github.com/apache/iceberg-rust/pull/520#discussion_r1704242728 ## crates/iceberg/tests/file_io_gcs_test.rs: ## @@ -0,0 +1,103 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license ag

Re: [PR] Deprecate Redundant Identifier Support in TableIdentifier, and row_filter [iceberg-python]

2024-08-05 Thread via GitHub
sungwy commented on code in PR #994: URL: https://github.com/apache/iceberg-python/pull/994#discussion_r1704242416 ## pyiceberg/catalog/__init__.py: ## @@ -613,6 +613,11 @@ def update_namespace_properties( ValueError: If removals and updates have overlapping keys.

Re: [PR] core: support support move a column with same name after rename column [iceberg]

2024-08-05 Thread via GitHub
FANNG1 commented on code in PR #10862: URL: https://github.com/apache/iceberg/pull/10862#discussion_r1704245450 ## core/src/main/java/org/apache/iceberg/SchemaUpdate.java: ## @@ -185,6 +186,8 @@ private void internalAddColumn( @Override public UpdateSchema deleteColumn(S

Re: [I] Check if dependencies in libs.versions.toml are the latest supported ones for JDK11 [iceberg]

2024-08-05 Thread via GitHub
nastra commented on issue #10852: URL: https://github.com/apache/iceberg/issues/10852#issuecomment-2269317210 @imneerajsharma yes sure, I'll assign it to you. Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

2024-08-05 Thread via GitHub
c-thiel commented on code in PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#discussion_r1704298527 ## crates/iceberg/src/spec/partition.rs: ## @@ -117,22 +144,312 @@ pub struct UnboundPartitionField { } /// Unbound partition spec can be built without a schema

Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

2024-08-05 Thread via GitHub
c-thiel commented on code in PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#discussion_r1704310954 ## crates/iceberg/src/spec/partition.rs: ## @@ -117,22 +144,312 @@ pub struct UnboundPartitionField { } /// Unbound partition spec can be built without a schema

Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

2024-08-05 Thread via GitHub
c-thiel commented on code in PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#discussion_r1704314380 ## crates/iceberg/src/spec/partition.rs: ## @@ -117,22 +144,312 @@ pub struct UnboundPartitionField { } /// Unbound partition spec can be built without a schema

Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

2024-08-05 Thread via GitHub
c-thiel commented on code in PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#discussion_r1704319118 ## crates/iceberg/src/spec/partition.rs: ## @@ -117,22 +144,312 @@ pub struct UnboundPartitionField { } /// Unbound partition spec can be built without a schema

Re: [PR] Pyarrow IO property for configuring large v small types on read [iceberg-python]

2024-08-05 Thread via GitHub
sungwy commented on PR #986: URL: https://github.com/apache/iceberg-python/pull/986#issuecomment-2269392619 > @sungwy Thanks for working on this! > > It seems we also need to update `schema_to_pyarrow`/`_cast_if_needed` to honor the new property. Thanks @HonahX ! I've updated t

Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

2024-08-05 Thread via GitHub
c-thiel commented on code in PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#discussion_r1704331295 ## crates/iceberg/src/spec/partition.rs: ## @@ -117,22 +144,312 @@ pub struct UnboundPartitionField { } /// Unbound partition spec can be built without a schema

Re: [PR] Spec: Deprecate the file system table scheme. [iceberg]

2024-08-05 Thread via GitHub
ajantha-bhat commented on code in PR #10833: URL: https://github.com/apache/iceberg/pull/10833#discussion_r1704346921 ## format/spec.md: ## @@ -1393,4 +1395,4 @@ This section covers topics not required by the specification but recommendations Iceberg supports two types of hist

Re: [PR] Spec: Deprecate the file system table scheme. [iceberg]

2024-08-05 Thread via GitHub
ajantha-bhat commented on code in PR #10833: URL: https://github.com/apache/iceberg/pull/10833#discussion_r1704346921 ## format/spec.md: ## @@ -1393,4 +1395,4 @@ This section covers topics not required by the specification but recommendations Iceberg supports two types of hist

Re: [PR] Spec: Deprecate the file system table scheme. [iceberg]

2024-08-05 Thread via GitHub
amogh-jahagirdar commented on code in PR #10833: URL: https://github.com/apache/iceberg/pull/10833#discussion_r1704357131 ## format/spec.md: ## @@ -1393,4 +1395,4 @@ This section covers topics not required by the specification but recommendations Iceberg supports two types of

[I] Prevent `add_files` from adding a file that's already referenced by the Iceberg Table [iceberg-python]

2024-08-05 Thread via GitHub
sungwy opened a new issue, #998: URL: https://github.com/apache/iceberg-python/issues/998 ### Feature Request / Improvement Currently `add_files` doesn't have a check to prevent adding an object that's already referenced by the Iceberg Table. We should include these two checks

Re: [PR] feat: support for gcs storage [iceberg-rust]

2024-08-05 Thread via GitHub
Xuanwo commented on code in PR #520: URL: https://github.com/apache/iceberg-rust/pull/520#discussion_r1704397856 ## crates/iceberg/src/io/storage_gcs.rs: ## @@ -0,0 +1,74 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements

[PR] Flink: add 1.20 support and remove 1.17 [iceberg]

2024-08-05 Thread via GitHub
stevenzwu opened a new pull request, #10881: URL: https://github.com/apache/iceberg/pull/10881 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [I] table.scan queries failing sometimes when result is empty [iceberg-python]

2024-08-05 Thread via GitHub
jurossiar commented on issue #992: URL: https://github.com/apache/iceberg-python/issues/992#issuecomment-2269566492 Sorry! I've just saw the messages. I appended [example.zip](https://github.com/user-attachments/files/16474537/example.zip) where you can create the example from scratch. I

Re: [I] table.scan queries failing sometimes when result is empty [iceberg-python]

2024-08-05 Thread via GitHub
jurossiar commented on issue #992: URL: https://github.com/apache/iceberg-python/issues/992#issuecomment-2269572366 > Here's what I ran based on the video, I cannot reproduce the issue using 0.7.0. > > In Jupyter, > > ``` > !pip install pyiceberg==0.7.0 --force -q > ```

Re: [PR] Flink: add 1.20 support and remove 1.17 [iceberg]

2024-08-05 Thread via GitHub
stevenzwu commented on code in PR #10881: URL: https://github.com/apache/iceberg/pull/10881#discussion_r1704443576 ## flink/v1.19/flink/src/jmh/java/org/apache/iceberg/flink/sink/shuffle/MapRangePartitionerBenchmark.java: ## @@ -66,6 +69,8 @@ public class MapRangePartitionerBenc

Re: [I] table.scan queries failing sometimes when result is empty [iceberg-python]

2024-08-05 Thread via GitHub
jurossiar commented on issue #992: URL: https://github.com/apache/iceberg-python/issues/992#issuecomment-2269581909 Seems that @grobgl was able to reproduce it -> https://github.com/grobgl/iceberg-python/commit/9f28f5ad94fd2f6a480b4cb788e730e57cc5c94c -- This is an automated message from

Re: [I] The ColumnarToRow Spark optimization is not applied when using nested fields from an Iceberg table [iceberg]

2024-08-05 Thread via GitHub
cccs-jc commented on issue #10828: URL: https://github.com/apache/iceberg/issues/10828#issuecomment-2269583914 ha okay, so there is probably no show stopper in replicating what has been done in the Spark Vectorized read since then. Is there plans to update the Iceberg SparkBatch soon?

Re: [PR] SQL Catalog [iceberg-rust]

2024-08-05 Thread via GitHub
callum-ryan closed pull request #503: SQL Catalog URL: https://github.com/apache/iceberg-rust/pull/503 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: i

[PR] feat: initialise SQL Catalog [iceberg-rust]

2024-08-05 Thread via GitHub
callum-ryan opened a new pull request, #524: URL: https://github.com/apache/iceberg-rust/pull/524 As discussed in [PR503](https://github.com/apache/iceberg-rust/pull/503) - splitting out SQL catalog PRs to make development + review easier. Addressed @Xuanwo's comments about formatting

Re: [I] Prevent `add_files` from adding a file that's already referenced by the Iceberg Table [iceberg-python]

2024-08-05 Thread via GitHub
amitgilad3 commented on issue #998: URL: https://github.com/apache/iceberg-python/issues/998#issuecomment-2269615389 Hey, im new to the pyiceberg but would love to take take a crack at this -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Aliyun: Replace assert usage with assertThat [iceberg]

2024-08-05 Thread via GitHub
findepi commented on code in PR #10880: URL: https://github.com/apache/iceberg/pull/10880#discussion_r1704491193 ## aliyun/src/test/java/org/apache/iceberg/aliyun/oss/mock/AliyunOSSMockLocalStore.java: ## @@ -170,17 +174,21 @@ ObjectMetadata putObject( void deleteObject(Str

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

2024-08-05 Thread via GitHub
nastra commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1704495785 ## core/src/test/java/org/apache/iceberg/TestUpdateRequirements.java: ## @@ -199,7 +199,21 @@ public void upgradeFormatVersion() { } @Test - public void upgra

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

2024-08-05 Thread via GitHub
nastra commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1704497762 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -1452,52 +1452,148 @@ public void testCreateV2MetadataThroughTableProperty() { } @Test

Re: [PR] Core: Make namespace separator configurable [iceberg]

2024-08-05 Thread via GitHub
nastra closed pull request #10877: Core: Make namespace separator configurable URL: https://github.com/apache/iceberg/pull/10877 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Aliyun: Replace assert usage with assertThat [iceberg]

2024-08-05 Thread via GitHub
nastra commented on code in PR #10880: URL: https://github.com/apache/iceberg/pull/10880#discussion_r1704500449 ## aliyun/src/test/java/org/apache/iceberg/aliyun/oss/mock/AliyunOSSMockLocalStore.java: ## @@ -170,17 +174,21 @@ ObjectMetadata putObject( void deleteObject(Stri

Re: [PR] Aliyun: Replace assert usage with assertThat [iceberg]

2024-08-05 Thread via GitHub
nastra commented on PR #10880: URL: https://github.com/apache/iceberg/pull/10880#issuecomment-2269678978 thanks for the review @findepi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Aliyun: Replace assert usage with assertThat [iceberg]

2024-08-05 Thread via GitHub
nastra merged PR #10880: URL: https://github.com/apache/iceberg/pull/10880 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Improve test_version_format() error message for version mismatches [iceberg-python]

2024-08-05 Thread via GitHub
laksh-krishna-sharma commented on PR #991: URL: https://github.com/apache/iceberg-python/pull/991#issuecomment-2269685994 I will investigate the issue and address the failing tests as soon as possible. I'll update once the problem is resolved. -- This is an automated message from the Apac

[I] MERGE INTO TABLE is not supported temporarily. [iceberg]

2024-08-05 Thread via GitHub
CharanHS30 opened a new issue, #10882: URL: https://github.com/apache/iceberg/issues/10882 ### Query engine import org.apache.spark.SparkConf import org.apache.spark.internal.Logging import org.apache.spark.sql.{DataFrame, SparkSession} import org.scalatest.{BeforeAndAfter, Be

Re: [PR] Aliyun: Replace assert usage with assertThat [iceberg]

2024-08-05 Thread via GitHub
findepi commented on code in PR #10880: URL: https://github.com/apache/iceberg/pull/10880#discussion_r1704567580 ## aliyun/src/test/java/org/apache/iceberg/aliyun/oss/mock/AliyunOSSMockLocalStore.java: ## @@ -170,17 +174,21 @@ ObjectMetadata putObject( void deleteObject(Str

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

2024-08-05 Thread via GitHub
amogh-jahagirdar commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1704575946 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1015,6 +1015,11 @@ public Builder upgradeFormatVersion(int newFormatVersion) {

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

2024-08-05 Thread via GitHub
amogh-jahagirdar commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1704575683 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1015,6 +1015,11 @@ public Builder upgradeFormatVersion(int newFormatVersion) {

Re: [PR] feat: support for gcs storage [iceberg-rust]

2024-08-05 Thread via GitHub
jdockerty commented on code in PR #520: URL: https://github.com/apache/iceberg-rust/pull/520#discussion_r1703160577 ## crates/iceberg/tests/file_io_gcs_test.rs: ## @@ -0,0 +1,103 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license ag

Re: [PR] Core: Extract filePath comparator into it's own class [iceberg]

2024-08-05 Thread via GitHub
flyrain merged PR #10664: URL: https://github.com/apache/iceberg/pull/10664 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Core: Extract filePath comparator into it's own class [iceberg]

2024-08-05 Thread via GitHub
flyrain commented on PR #10664: URL: https://github.com/apache/iceberg/pull/10664#issuecomment-2269885205 Thanks @deniskuzZ for the PR. Thanks @dramaticlly @anuragmantri @RussellSpitzer for the review. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Docs: Fix SQL in branching docs [iceberg]

2024-08-05 Thread via GitHub
amogh-jahagirdar merged PR #10876: URL: https://github.com/apache/iceberg/pull/10876 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

2024-08-05 Thread via GitHub
leangjonathan commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1704622985 ## core/src/test/java/org/apache/iceberg/TestFormatVersions.java: ## @@ -28,51 +28,79 @@ public class TestFormatVersions extends TestBase { @Parameters(name

Re: [I] Prevent `add_files` from adding a file that's already referenced by the Iceberg Table [iceberg-python]

2024-08-05 Thread via GitHub
sungwy commented on issue #998: URL: https://github.com/apache/iceberg-python/issues/998#issuecomment-2269917041 Hi @amitgilad3 sounds great! I'll get this assigned to you. Please let me know if you'd like some pointers :) -- This is an automated message from the Apache Git Service. To re

Re: [PR] Kafka Connect: Stop commits on terminated coordinator [iceberg]

2024-08-05 Thread via GitHub
bryanck commented on code in PR #10814: URL: https://github.com/apache/iceberg/pull/10814#discussion_r1704648164 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/Coordinator.java: ## @@ -218,6 +220,10 @@ private void commitToTable( .fi

Re: [PR] API: Add SupportsRecoveryOperations mixin for FileIO [iceberg]

2024-08-05 Thread via GitHub
amogh-jahagirdar commented on PR #10711: URL: https://github.com/apache/iceberg/pull/10711#issuecomment-2269960658 Since there's utility in this mixin beyond the repair manifests implementation (someone can make use of this mixin in their own code/FileIO implementation for recovering indivi

Re: [PR] API: Add SupportsRecoveryOperations mixin for FileIO [iceberg]

2024-08-05 Thread via GitHub
amogh-jahagirdar merged PR #10711: URL: https://github.com/apache/iceberg/pull/10711 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

[PR] Bump pypa/cibuildwheel from 2.19.2 to 2.20.0 [iceberg-python]

2024-08-05 Thread via GitHub
dependabot[bot] opened a new pull request, #999: URL: https://github.com/apache/iceberg-python/pull/999 Bumps [pypa/cibuildwheel](https://github.com/pypa/cibuildwheel) from 2.19.2 to 2.20.0. Release notes Sourced from https://github.com/pypa/cibuildwheel/releases";>pypa/cibuildwhee

[PR] Bump moto from 5.0.11 to 5.0.12 [iceberg-python]

2024-08-05 Thread via GitHub
dependabot[bot] opened a new pull request, #1000: URL: https://github.com/apache/iceberg-python/pull/1000 Bumps [moto](https://github.com/getmoto/moto) from 5.0.11 to 5.0.12. Changelog Sourced from https://github.com/getmoto/moto/blob/master/CHANGELOG.md";>moto's changelog.

[PR] Bump mkdocs-material from 9.5.30 to 9.5.31 [iceberg-python]

2024-08-05 Thread via GitHub
dependabot[bot] opened a new pull request, #1001: URL: https://github.com/apache/iceberg-python/pull/1001 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.30 to 9.5.31. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>

[PR] Bump cython from 3.0.10 to 3.0.11 [iceberg-python]

2024-08-05 Thread via GitHub
dependabot[bot] opened a new pull request, #1002: URL: https://github.com/apache/iceberg-python/pull/1002 Bumps [cython](https://github.com/cython/cython) from 3.0.10 to 3.0.11. Changelog Sourced from https://github.com/cython/cython/blob/master/CHANGES.rst";>cython's changelog.

  1   2   >