Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

2024-08-05 Thread via GitHub
c-thiel commented on code in PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#discussion_r1704969609 ## crates/iceberg/src/spec/partition.rs: ## @@ -117,22 +144,312 @@ pub struct UnboundPartitionField { } /// Unbound partition spec can be built without a schema

Re: [I] Improvement: reuse the tcp connection when plan files [iceberg-rust]

2024-08-05 Thread via GitHub
Xuanwo commented on issue #522: URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2270424332 Thanks for the sharing. I believe this can be addressed by resuing the same http client. -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

2024-08-05 Thread via GitHub
leangjonathan commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1704906987 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1015,6 +1015,11 @@ public Builder upgradeFormatVersion(int newFormatVersion) { "

Re: [PR] feat: initialise SQL Catalog [iceberg-rust]

2024-08-05 Thread via GitHub
Xuanwo commented on code in PR #524: URL: https://github.com/apache/iceberg-rust/pull/524#discussion_r1704882565 ## crates/catalog/sql/Cargo.toml: ## @@ -0,0 +1,43 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See th

Re: [PR] Flink: add 1.20 support and remove 1.17 [iceberg]

2024-08-05 Thread via GitHub
stevenzwu commented on PR #10881: URL: https://github.com/apache/iceberg/pull/10881#issuecomment-2270358383 > Hi @stevenzwu Is it possible to keep the original commits and authors in the newly introduced flink1.20 , so that people could easily to track how was the current version iterated ?

Re: [I] Improvement: reuse the tcp connection when plan files [iceberg-rust]

2024-08-05 Thread via GitHub
chenzl25 commented on issue #522: URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2270355507 However, risingwave the database use iceberg-rust seems can't reuse any TCP connection. You can see that every HTTP request is in its own TCP connection. https://github.com

Re: [I] Improvement: reuse the tcp connection when plan files [iceberg-rust]

2024-08-05 Thread via GitHub
chenzl25 commented on issue #522: URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2270352785 I have tested duckdb and clickhouse to scan an iceberg table in MinIO as well. They can reuse a TCP connection to send HTTP requests. BTW, clickhouse can reuse connections aggres

Re: [I] Improvement: reuse the tcp connection when plan files [iceberg-rust]

2024-08-05 Thread via GitHub
chenzl25 commented on issue #522: URL: https://github.com/apache/iceberg-rust/issues/522#issuecomment-2270330523 Great, Looking forward to it! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Flink: add 1.20 support and remove 1.17 [iceberg]

2024-08-05 Thread via GitHub
openinx commented on PR #10881: URL: https://github.com/apache/iceberg/pull/10881#issuecomment-2270324527 Hi @stevenzwu Is it possible to keep the original commits and authors in the newly introduced flink1.20 , so that people could easily to track how was the current version iterated ?

Re: [PR] Kafka Connect: Add kerberos authentication option [iceberg]

2024-08-05 Thread via GitHub
Dawnpool commented on PR #10173: URL: https://github.com/apache/iceberg/pull/10173#issuecomment-2270321532 Hi @bryanck I did some research and I couldn't really find a way to set the user principal via an environment variable. Could you please provide more detailed information about tha

Re: [PR] Spec: Deprecate the file system table scheme. [iceberg]

2024-08-05 Thread via GitHub
manuzhang commented on code in PR #10833: URL: https://github.com/apache/iceberg/pull/10833#discussion_r1704850668 ## format/spec.md: ## @@ -1393,4 +1395,4 @@ This section covers topics not required by the specification but recommendations Iceberg supports two types of histori

Re: [I] table.scan queries failing sometimes when result is empty [iceberg-python]

2024-08-05 Thread via GitHub
kevinjqliu commented on issue #992: URL: https://github.com/apache/iceberg-python/issues/992#issuecomment-2270291709 Thank you. I was able to reproduce the error with ``` requested_columns=['*'] scan = table.scan( selected_fields = (requested_columns), row_filt

[I] support pyarrow recordbatch as a valid data source for writing Iceberg table [iceberg-python]

2024-08-05 Thread via GitHub
djouallah opened a new issue, #1004: URL: https://github.com/apache/iceberg-python/issues/1004 ### Feature Request / Improvement currently using this in a an environment with limited RAM ``` df=final.arrow() catalog , table_location = connect_catalog(storage) cata

Re: [PR] Improve test_version_format() error message for version mismatches [iceberg-python]

2024-08-05 Thread via GitHub
kevinjqliu commented on PR #991: URL: https://github.com/apache/iceberg-python/pull/991#issuecomment-2270272118 Thanks @laksh-krishna-sharma Looks like this is a linter issue ``` fix end of files.Failed - hook id: end-of-fi

Re: [I] Tracking issues of aligning storage support with iceberg-java [iceberg-rust]

2024-08-05 Thread via GitHub
Xuanwo commented on issue #408: URL: https://github.com/apache/iceberg-rust/issues/408#issuecomment-2270260824 > cc @Xuanwo Do you still plan to finish this before in 0.3.0? Or we can postpone it to next release? There are some more work to do at opendal side. I believe we can let 0.3

Re: [PR] Simplify PrimitiveLiteral [iceberg-rust]

2024-08-05 Thread via GitHub
liurenjie1024 commented on code in PR #502: URL: https://github.com/apache/iceberg-rust/pull/502#discussion_r1704830446 ## crates/iceberg/src/spec/values.rs: ## @@ -65,24 +65,14 @@ pub enum PrimitiveLiteral { Float(OrderedFloat), /// Stored as 8-byte little-endian

Re: [I] Tracking issues of aligning storage support with iceberg-java [iceberg-rust]

2024-08-05 Thread via GitHub
liurenjie1024 commented on issue #408: URL: https://github.com/apache/iceberg-rust/issues/408#issuecomment-2270251078 cc @Xuanwo Do you still plan to finish this before in 0.3.0? Or we can postpone it to next release? -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

2024-08-05 Thread via GitHub
leangjonathan commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1704792858 ## core/src/test/java/org/apache/iceberg/TestFormatVersions.java: ## @@ -28,51 +28,79 @@ public class TestFormatVersions extends TestBase { @Parameters(name

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

2024-08-05 Thread via GitHub
leangjonathan commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1704792740 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -1452,52 +1452,148 @@ public void testCreateV2MetadataThroughTableProperty() { }

Re: [PR] Spec: Clarify identity partition edge cases. [iceberg]

2024-08-05 Thread via GitHub
amogh-jahagirdar merged PR #10835: URL: https://github.com/apache/iceberg/pull/10835 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

2024-08-05 Thread via GitHub
leangjonathan commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1704770301 ## core/src/test/java/org/apache/iceberg/TestUpdateRequirements.java: ## @@ -199,7 +199,21 @@ public void upgradeFormatVersion() { } @Test - public voi

Re: [I] Quick start docker-compose demo doesn't work [iceberg]

2024-08-05 Thread via GitHub
awongCM commented on issue #6708: URL: https://github.com/apache/iceberg/issues/6708#issuecomment-2270165294 > Can you run a `docker-compose pull` to make sure that you're running the latest version? We recently did some updates to the stack. Hi, I'm also running into this issue when

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

2024-08-05 Thread via GitHub
leangjonathan commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1704767712 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -1452,52 +1452,148 @@ public void testCreateV2MetadataThroughTableProperty() { }

Re: [PR] Spec: Clarify identity partition edge cases. [iceberg]

2024-08-05 Thread via GitHub
emkornfield commented on PR #10835: URL: https://github.com/apache/iceberg/pull/10835#issuecomment-2270162248 Could a committer merge? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Spec: Clarify identity partition edge cases. [iceberg]

2024-08-05 Thread via GitHub
emkornfield commented on PR #10835: URL: https://github.com/apache/iceberg/pull/10835#issuecomment-2270162084 [Vote passed](https://lists.apache.org/thread/44s6vkd9ky6x4ty2fyn9j7r7ylry6lgh) -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [I] Use griffe to find breaking changes [iceberg-python]

2024-08-05 Thread via GitHub
github-actions[bot] commented on issue #334: URL: https://github.com/apache/iceberg-python/issues/334#issuecomment-2270133714 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity oc

Re: [PR] AWS: Make the error message of GlueCatalog's isValidIdentifier more detailed [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] commented on PR #3668: URL: https://github.com/apache/iceberg/pull/3668#issuecomment-2270132271 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] SPARK[docs]:Add Warning information under the MERGE INTO syntax [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] commented on PR #3644: URL: https://github.com/apache/iceberg/pull/3644#issuecomment-2270132212 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] Regenerate the metadata of an Iceberg table [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] commented on issue #3639: URL: https://github.com/apache/iceberg/issues/3639#issuecomment-2270132199 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] can not change nullable via Spark SQL sometimes [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] commented on issue #3617: URL: https://github.com/apache/iceberg/issues/3617#issuecomment-2270132154 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] BaseDataReader is not properly closed during query by spark sql [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] commented on issue #3636: URL: https://github.com/apache/iceberg/issues/3636#issuecomment-2270132171 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Implement a call procedure for DeleteReachableFiles action [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] commented on issue #3588: URL: https://github.com/apache/iceberg/issues/3588#issuecomment-2270132136 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Doc: add a page to explain row-level deletes [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] commented on PR #3432: URL: https://github.com/apache/iceberg/pull/3432#issuecomment-2270132019 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Doc: add a page to explain row-level deletes [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] closed pull request #3432: Doc: add a page to explain row-level deletes URL: https://github.com/apache/iceberg/pull/3432 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Add schema validation during parquet file Import [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] commented on PR #3435: URL: https://github.com/apache/iceberg/pull/3435#issuecomment-2270132034 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [I] Table to retractStream error [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] commented on issue #3121: URL: https://github.com/apache/iceberg/issues/3121#issuecomment-2270131822 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [PR] Core, Spark: Fallback when snapshot does not have schema id [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] commented on PR #3852: URL: https://github.com/apache/iceberg/pull/3852#issuecomment-2270132319 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] Docs: Add `Streaming upsert write` in flink.md [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] commented on PR #3827: URL: https://github.com/apache/iceberg/pull/3827#issuecomment-2270132311 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [PR] HiveTableOperations: update HMS transient_lastDdlTime param on snapshot writes [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] commented on PR #3670: URL: https://github.com/apache/iceberg/pull/3670#issuecomment-2270132290 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] Adding MergeInto into the Spark Scala API [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] commented on issue #3665: URL: https://github.com/apache/iceberg/issues/3665#issuecomment-2270132245 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] retain flinkCheckpointInfo when rewrite datafiles to ensure restore flink job from checkpoint success [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] commented on PR #3647: URL: https://github.com/apache/iceberg/pull/3647#issuecomment-2270132224 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull

Re: [I] Regenerate the metadata of an Iceberg table [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] closed issue #3639: Regenerate the metadata of an Iceberg table URL: https://github.com/apache/iceberg/issues/3639 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [I] iceberg0.12 support removing duplicate primary key all the partition or one partition? [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] commented on issue #3566: URL: https://github.com/apache/iceberg/issues/3566#issuecomment-2270132122 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Add schema validation during parquet file Import [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] closed pull request #3435: Add schema validation during parquet file Import URL: https://github.com/apache/iceberg/pull/3435 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] API: Add function for removing Specs from Metadata.json which are no … [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] closed pull request #3462: API: Add function for removing Specs from Metadata.json which are no … URL: https://github.com/apache/iceberg/pull/3462 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] API: Add function for removing Specs from Metadata.json which are no … [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] commented on PR #3462: URL: https://github.com/apache/iceberg/pull/3462#issuecomment-2270132053 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Data: Use proper temp file in TestMetricsRowGroupFilterTypes [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] commented on PR #3416: URL: https://github.com/apache/iceberg/pull/3416#issuecomment-2270131992 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Data: Use proper temp file in TestMetricsRowGroupFilterTypes [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] closed pull request #3416: Data: Use proper temp file in TestMetricsRowGroupFilterTypes URL: https://github.com/apache/iceberg/pull/3416 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] Table to retractStream error [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] closed issue #3121: Table to retractStream error URL: https://github.com/apache/iceberg/issues/3121 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [I] When writing data to S3 using Glue Catalog, current snapshot ID is -1 and not updated in the metadata file generated [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] closed issue #3124: When writing data to S3 using Glue Catalog, current snapshot ID is -1 and not updated in the metadata file generated URL: https://github.com/apache/iceberg/issues/3124 -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [I] When writing data to S3 using Glue Catalog, current snapshot ID is -1 and not updated in the metadata file generated [iceberg]

2024-08-05 Thread via GitHub
github-actions[bot] commented on issue #3124: URL: https://github.com/apache/iceberg/issues/3124#issuecomment-2270131838 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

2024-08-05 Thread via GitHub
leangjonathan commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1704731376 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1015,6 +1015,11 @@ public Builder upgradeFormatVersion(int newFormatVersion) { "

Re: [PR] Core: Make namespace separator configurable [iceberg]

2024-08-05 Thread via GitHub
amogh-jahagirdar commented on code in PR #10877: URL: https://github.com/apache/iceberg/pull/10877#discussion_r1704687240 ## core/src/main/java/org/apache/iceberg/rest/RESTUtil.java: ## @@ -215,8 +232,26 @@ public static String encodeNamespace(Namespace ns) { * @return a nam

[PR] Bump sqlalchemy from 2.0.31 to 2.0.32 [iceberg-python]

2024-08-05 Thread via GitHub
dependabot[bot] opened a new pull request, #1003: URL: https://github.com/apache/iceberg-python/pull/1003 Bumps [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) from 2.0.31 to 2.0.32. Release notes Sourced from https://github.com/sqlalchemy/sqlalchemy/releases";>sqlalchemy's

[PR] Bump cython from 3.0.10 to 3.0.11 [iceberg-python]

2024-08-05 Thread via GitHub
dependabot[bot] opened a new pull request, #1002: URL: https://github.com/apache/iceberg-python/pull/1002 Bumps [cython](https://github.com/cython/cython) from 3.0.10 to 3.0.11. Changelog Sourced from https://github.com/cython/cython/blob/master/CHANGES.rst";>cython's changelog.

[PR] Bump mkdocs-material from 9.5.30 to 9.5.31 [iceberg-python]

2024-08-05 Thread via GitHub
dependabot[bot] opened a new pull request, #1001: URL: https://github.com/apache/iceberg-python/pull/1001 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.30 to 9.5.31. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>

[PR] Bump moto from 5.0.11 to 5.0.12 [iceberg-python]

2024-08-05 Thread via GitHub
dependabot[bot] opened a new pull request, #1000: URL: https://github.com/apache/iceberg-python/pull/1000 Bumps [moto](https://github.com/getmoto/moto) from 5.0.11 to 5.0.12. Changelog Sourced from https://github.com/getmoto/moto/blob/master/CHANGELOG.md";>moto's changelog.

[PR] Bump pypa/cibuildwheel from 2.19.2 to 2.20.0 [iceberg-python]

2024-08-05 Thread via GitHub
dependabot[bot] opened a new pull request, #999: URL: https://github.com/apache/iceberg-python/pull/999 Bumps [pypa/cibuildwheel](https://github.com/pypa/cibuildwheel) from 2.19.2 to 2.20.0. Release notes Sourced from https://github.com/pypa/cibuildwheel/releases";>pypa/cibuildwhee

Re: [PR] API: Add SupportsRecoveryOperations mixin for FileIO [iceberg]

2024-08-05 Thread via GitHub
amogh-jahagirdar merged PR #10711: URL: https://github.com/apache/iceberg/pull/10711 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] API: Add SupportsRecoveryOperations mixin for FileIO [iceberg]

2024-08-05 Thread via GitHub
amogh-jahagirdar commented on PR #10711: URL: https://github.com/apache/iceberg/pull/10711#issuecomment-2269960658 Since there's utility in this mixin beyond the repair manifests implementation (someone can make use of this mixin in their own code/FileIO implementation for recovering indivi

Re: [PR] Kafka Connect: Stop commits on terminated coordinator [iceberg]

2024-08-05 Thread via GitHub
bryanck commented on code in PR #10814: URL: https://github.com/apache/iceberg/pull/10814#discussion_r1704648164 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/Coordinator.java: ## @@ -218,6 +220,10 @@ private void commitToTable( .fi

Re: [I] Prevent `add_files` from adding a file that's already referenced by the Iceberg Table [iceberg-python]

2024-08-05 Thread via GitHub
sungwy commented on issue #998: URL: https://github.com/apache/iceberg-python/issues/998#issuecomment-2269917041 Hi @amitgilad3 sounds great! I'll get this assigned to you. Please let me know if you'd like some pointers :) -- This is an automated message from the Apache Git Service. To re

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

2024-08-05 Thread via GitHub
leangjonathan commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1704622985 ## core/src/test/java/org/apache/iceberg/TestFormatVersions.java: ## @@ -28,51 +28,79 @@ public class TestFormatVersions extends TestBase { @Parameters(name

Re: [PR] Docs: Fix SQL in branching docs [iceberg]

2024-08-05 Thread via GitHub
amogh-jahagirdar merged PR #10876: URL: https://github.com/apache/iceberg/pull/10876 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Core: Extract filePath comparator into it's own class [iceberg]

2024-08-05 Thread via GitHub
flyrain commented on PR #10664: URL: https://github.com/apache/iceberg/pull/10664#issuecomment-2269885205 Thanks @deniskuzZ for the PR. Thanks @dramaticlly @anuragmantri @RussellSpitzer for the review. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Core: Extract filePath comparator into it's own class [iceberg]

2024-08-05 Thread via GitHub
flyrain merged PR #10664: URL: https://github.com/apache/iceberg/pull/10664 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] feat: support for gcs storage [iceberg-rust]

2024-08-05 Thread via GitHub
jdockerty commented on code in PR #520: URL: https://github.com/apache/iceberg-rust/pull/520#discussion_r1703160577 ## crates/iceberg/tests/file_io_gcs_test.rs: ## @@ -0,0 +1,103 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license ag

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

2024-08-05 Thread via GitHub
amogh-jahagirdar commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1704575683 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1015,6 +1015,11 @@ public Builder upgradeFormatVersion(int newFormatVersion) {

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

2024-08-05 Thread via GitHub
amogh-jahagirdar commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1704575946 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1015,6 +1015,11 @@ public Builder upgradeFormatVersion(int newFormatVersion) {

Re: [PR] Aliyun: Replace assert usage with assertThat [iceberg]

2024-08-05 Thread via GitHub
findepi commented on code in PR #10880: URL: https://github.com/apache/iceberg/pull/10880#discussion_r1704567580 ## aliyun/src/test/java/org/apache/iceberg/aliyun/oss/mock/AliyunOSSMockLocalStore.java: ## @@ -170,17 +174,21 @@ ObjectMetadata putObject( void deleteObject(Str

[I] MERGE INTO TABLE is not supported temporarily. [iceberg]

2024-08-05 Thread via GitHub
CharanHS30 opened a new issue, #10882: URL: https://github.com/apache/iceberg/issues/10882 ### Query engine import org.apache.spark.SparkConf import org.apache.spark.internal.Logging import org.apache.spark.sql.{DataFrame, SparkSession} import org.scalatest.{BeforeAndAfter, Be

Re: [PR] Improve test_version_format() error message for version mismatches [iceberg-python]

2024-08-05 Thread via GitHub
laksh-krishna-sharma commented on PR #991: URL: https://github.com/apache/iceberg-python/pull/991#issuecomment-2269685994 I will investigate the issue and address the failing tests as soon as possible. I'll update once the problem is resolved. -- This is an automated message from the Apac

Re: [PR] Aliyun: Replace assert usage with assertThat [iceberg]

2024-08-05 Thread via GitHub
nastra merged PR #10880: URL: https://github.com/apache/iceberg/pull/10880 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Aliyun: Replace assert usage with assertThat [iceberg]

2024-08-05 Thread via GitHub
nastra commented on PR #10880: URL: https://github.com/apache/iceberg/pull/10880#issuecomment-2269678978 thanks for the review @findepi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Aliyun: Replace assert usage with assertThat [iceberg]

2024-08-05 Thread via GitHub
nastra commented on code in PR #10880: URL: https://github.com/apache/iceberg/pull/10880#discussion_r1704500449 ## aliyun/src/test/java/org/apache/iceberg/aliyun/oss/mock/AliyunOSSMockLocalStore.java: ## @@ -170,17 +174,21 @@ ObjectMetadata putObject( void deleteObject(Stri

Re: [PR] Core: Make namespace separator configurable [iceberg]

2024-08-05 Thread via GitHub
nastra closed pull request #10877: Core: Make namespace separator configurable URL: https://github.com/apache/iceberg/pull/10877 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

2024-08-05 Thread via GitHub
nastra commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1704497762 ## core/src/test/java/org/apache/iceberg/TestTableMetadata.java: ## @@ -1452,52 +1452,148 @@ public void testCreateV2MetadataThroughTableProperty() { } @Test

Re: [PR] Core: V3 Metadata Upgrade Validation and Testing [iceberg]

2024-08-05 Thread via GitHub
nastra commented on code in PR #10861: URL: https://github.com/apache/iceberg/pull/10861#discussion_r1704495785 ## core/src/test/java/org/apache/iceberg/TestUpdateRequirements.java: ## @@ -199,7 +199,21 @@ public void upgradeFormatVersion() { } @Test - public void upgra

Re: [PR] Aliyun: Replace assert usage with assertThat [iceberg]

2024-08-05 Thread via GitHub
findepi commented on code in PR #10880: URL: https://github.com/apache/iceberg/pull/10880#discussion_r1704491193 ## aliyun/src/test/java/org/apache/iceberg/aliyun/oss/mock/AliyunOSSMockLocalStore.java: ## @@ -170,17 +174,21 @@ ObjectMetadata putObject( void deleteObject(Str

Re: [I] Prevent `add_files` from adding a file that's already referenced by the Iceberg Table [iceberg-python]

2024-08-05 Thread via GitHub
amitgilad3 commented on issue #998: URL: https://github.com/apache/iceberg-python/issues/998#issuecomment-2269615389 Hey, im new to the pyiceberg but would love to take take a crack at this -- This is an automated message from the Apache Git Service. To respond to the message, please log

[PR] feat: initialise SQL Catalog [iceberg-rust]

2024-08-05 Thread via GitHub
callum-ryan opened a new pull request, #524: URL: https://github.com/apache/iceberg-rust/pull/524 As discussed in [PR503](https://github.com/apache/iceberg-rust/pull/503) - splitting out SQL catalog PRs to make development + review easier. Addressed @Xuanwo's comments about formatting

Re: [PR] SQL Catalog [iceberg-rust]

2024-08-05 Thread via GitHub
callum-ryan closed pull request #503: SQL Catalog URL: https://github.com/apache/iceberg-rust/pull/503 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: i

Re: [I] The ColumnarToRow Spark optimization is not applied when using nested fields from an Iceberg table [iceberg]

2024-08-05 Thread via GitHub
cccs-jc commented on issue #10828: URL: https://github.com/apache/iceberg/issues/10828#issuecomment-2269583914 ha okay, so there is probably no show stopper in replicating what has been done in the Spark Vectorized read since then. Is there plans to update the Iceberg SparkBatch soon?

Re: [I] table.scan queries failing sometimes when result is empty [iceberg-python]

2024-08-05 Thread via GitHub
jurossiar commented on issue #992: URL: https://github.com/apache/iceberg-python/issues/992#issuecomment-2269581909 Seems that @grobgl was able to reproduce it -> https://github.com/grobgl/iceberg-python/commit/9f28f5ad94fd2f6a480b4cb788e730e57cc5c94c -- This is an automated message from

Re: [PR] Flink: add 1.20 support and remove 1.17 [iceberg]

2024-08-05 Thread via GitHub
stevenzwu commented on code in PR #10881: URL: https://github.com/apache/iceberg/pull/10881#discussion_r1704443576 ## flink/v1.19/flink/src/jmh/java/org/apache/iceberg/flink/sink/shuffle/MapRangePartitionerBenchmark.java: ## @@ -66,6 +69,8 @@ public class MapRangePartitionerBenc

Re: [I] table.scan queries failing sometimes when result is empty [iceberg-python]

2024-08-05 Thread via GitHub
jurossiar commented on issue #992: URL: https://github.com/apache/iceberg-python/issues/992#issuecomment-2269572366 > Here's what I ran based on the video, I cannot reproduce the issue using 0.7.0. > > In Jupyter, > > ``` > !pip install pyiceberg==0.7.0 --force -q > ```

Re: [I] table.scan queries failing sometimes when result is empty [iceberg-python]

2024-08-05 Thread via GitHub
jurossiar commented on issue #992: URL: https://github.com/apache/iceberg-python/issues/992#issuecomment-2269566492 Sorry! I've just saw the messages. I appended [example.zip](https://github.com/user-attachments/files/16474537/example.zip) where you can create the example from scratch. I

[PR] Flink: add 1.20 support and remove 1.17 [iceberg]

2024-08-05 Thread via GitHub
stevenzwu opened a new pull request, #10881: URL: https://github.com/apache/iceberg/pull/10881 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] feat: support for gcs storage [iceberg-rust]

2024-08-05 Thread via GitHub
Xuanwo commented on code in PR #520: URL: https://github.com/apache/iceberg-rust/pull/520#discussion_r1704397856 ## crates/iceberg/src/io/storage_gcs.rs: ## @@ -0,0 +1,74 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements

[I] Prevent `add_files` from adding a file that's already referenced by the Iceberg Table [iceberg-python]

2024-08-05 Thread via GitHub
sungwy opened a new issue, #998: URL: https://github.com/apache/iceberg-python/issues/998 ### Feature Request / Improvement Currently `add_files` doesn't have a check to prevent adding an object that's already referenced by the Iceberg Table. We should include these two checks

Re: [PR] Spec: Deprecate the file system table scheme. [iceberg]

2024-08-05 Thread via GitHub
amogh-jahagirdar commented on code in PR #10833: URL: https://github.com/apache/iceberg/pull/10833#discussion_r1704357131 ## format/spec.md: ## @@ -1393,4 +1395,4 @@ This section covers topics not required by the specification but recommendations Iceberg supports two types of

Re: [PR] Spec: Deprecate the file system table scheme. [iceberg]

2024-08-05 Thread via GitHub
ajantha-bhat commented on code in PR #10833: URL: https://github.com/apache/iceberg/pull/10833#discussion_r1704346921 ## format/spec.md: ## @@ -1393,4 +1395,4 @@ This section covers topics not required by the specification but recommendations Iceberg supports two types of hist

Re: [PR] Spec: Deprecate the file system table scheme. [iceberg]

2024-08-05 Thread via GitHub
ajantha-bhat commented on code in PR #10833: URL: https://github.com/apache/iceberg/pull/10833#discussion_r1704346921 ## format/spec.md: ## @@ -1393,4 +1395,4 @@ This section covers topics not required by the specification but recommendations Iceberg supports two types of hist

Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

2024-08-05 Thread via GitHub
c-thiel commented on code in PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#discussion_r1704331295 ## crates/iceberg/src/spec/partition.rs: ## @@ -117,22 +144,312 @@ pub struct UnboundPartitionField { } /// Unbound partition spec can be built without a schema

Re: [PR] Pyarrow IO property for configuring large v small types on read [iceberg-python]

2024-08-05 Thread via GitHub
sungwy commented on PR #986: URL: https://github.com/apache/iceberg-python/pull/986#issuecomment-2269392619 > @sungwy Thanks for working on this! > > It seems we also need to update `schema_to_pyarrow`/`_cast_if_needed` to honor the new property. Thanks @HonahX ! I've updated t

Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

2024-08-05 Thread via GitHub
c-thiel commented on code in PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#discussion_r1704319118 ## crates/iceberg/src/spec/partition.rs: ## @@ -117,22 +144,312 @@ pub struct UnboundPartitionField { } /// Unbound partition spec can be built without a schema

Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

2024-08-05 Thread via GitHub
c-thiel commented on code in PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#discussion_r1704314380 ## crates/iceberg/src/spec/partition.rs: ## @@ -117,22 +144,312 @@ pub struct UnboundPartitionField { } /// Unbound partition spec can be built without a schema

Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

2024-08-05 Thread via GitHub
c-thiel commented on code in PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#discussion_r1704310954 ## crates/iceberg/src/spec/partition.rs: ## @@ -117,22 +144,312 @@ pub struct UnboundPartitionField { } /// Unbound partition spec can be built without a schema

Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

2024-08-05 Thread via GitHub
c-thiel commented on code in PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#discussion_r1704298527 ## crates/iceberg/src/spec/partition.rs: ## @@ -117,22 +144,312 @@ pub struct UnboundPartitionField { } /// Unbound partition spec can be built without a schema

Re: [I] Check if dependencies in libs.versions.toml are the latest supported ones for JDK11 [iceberg]

2024-08-05 Thread via GitHub
nastra commented on issue #10852: URL: https://github.com/apache/iceberg/issues/10852#issuecomment-2269317210 @imneerajsharma yes sure, I'll assign it to you. Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

  1   2   >