Re: [PR] Add basic CMake support for the iceberg library [iceberg-cpp]

2024-12-01 Thread via GitHub
gaborkaszab commented on code in PR #3: URL: https://github.com/apache/iceberg-cpp/pull/3#discussion_r1865359742 ## src/demo.cc: ## @@ -0,0 +1,26 @@ +/* Review Comment: I think I'd structure the code a bit differently. How I imagined the structure of the c++ lib is somethin

Re: [PR] Docs: Default value of table level distribution-mode should be not set [iceberg]

2024-12-01 Thread via GitHub
nastra commented on code in PR #11663: URL: https://github.com/apache/iceberg/pull/11663#discussion_r1865347866 ## docs/docs/configuration.md: ## @@ -67,10 +67,10 @@ Iceberg tables support table properties to configure table behavior, like the de | write.metadata.metrics.colum

Re: [I] Iceberg rest catalog with DynamodbCatalog implementation is throwing an exception with *dynamodb local* build. [iceberg]

2024-12-01 Thread via GitHub
nastra commented on issue #11674: URL: https://github.com/apache/iceberg/issues/11674#issuecomment-2510768230 @krisroops I don't know exactly what Dynamo needs in terms of secrets, but my guess is that the REST image needs to talk to Minio and to DynamoDB, so you need to make sure that you

Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

2024-12-01 Thread via GitHub
huaxingao commented on code in PR #11551: URL: https://github.com/apache/iceberg/pull/11551#discussion_r1865324477 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java: ## @@ -245,5 +247,16 @@ void applyEqDelete(ColumnarBatch column

Re: [PR] AWS: Enable RetryMode for AWS KMS client [iceberg]

2024-12-01 Thread via GitHub
hsiang-c commented on code in PR #11420: URL: https://github.com/apache/iceberg/pull/11420#discussion_r1865124927 ## aws/src/main/java/org/apache/iceberg/aws/kms/KmsClientProperties.java: ## @@ -0,0 +1,45 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

2024-12-01 Thread via GitHub
flyrain commented on code in PR #11551: URL: https://github.com/apache/iceberg/pull/11551#discussion_r1865274246 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java: ## @@ -245,5 +247,16 @@ void applyEqDelete(ColumnarBatch columnar

Re: [PR] chore(deps): Bump crate-ci/typos from 1.27.3 to 1.28.1 [iceberg-rust]

2024-12-01 Thread via GitHub
dependabot[bot] commented on PR #747: URL: https://github.com/apache/iceberg-rust/pull/747#issuecomment-2510624243 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version,

Re: [PR] chore(deps): Bump crate-ci/typos from 1.27.3 to 1.28.1 [iceberg-rust]

2024-12-01 Thread via GitHub
liurenjie1024 closed pull request #747: chore(deps): Bump crate-ci/typos from 1.27.3 to 1.28.1 URL: https://github.com/apache/iceberg-rust/pull/747 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

2024-12-01 Thread via GitHub
huaxingao commented on code in PR #11551: URL: https://github.com/apache/iceberg/pull/11551#discussion_r1865249456 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java: ## @@ -622,6 +623,50 @@ public void testPosDeletesOnParquetFileWithM

Re: [PR] Build: Bump org.xerial:sqlite-jdbc from 3.47.0.0 to 3.47.1.0 [iceberg]

2024-12-01 Thread via GitHub
Fokko merged PR #11682: URL: https://github.com/apache/iceberg/pull/11682 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Build: Bump jackson-bom from 2.18.1 to 2.18.2 [iceberg]

2024-12-01 Thread via GitHub
Fokko merged PR #11681: URL: https://github.com/apache/iceberg/pull/11681 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Bug Fix: Expose tableExists endpoint on RESTCatalogAdapter [iceberg]

2024-12-01 Thread via GitHub
Fokko commented on PR #11678: URL: https://github.com/apache/iceberg/pull/11678#issuecomment-2510610819 Thanks @ebyhr for pointing that out, and sorry for leaving that PR lingering. Thanks to @sungwy and @ebyhr for fixing this 🙌 I will close this one now since it is a duplicate. -- This

Re: [PR] Bug Fix: Expose tableExists endpoint on RESTCatalogAdapter [iceberg]

2024-12-01 Thread via GitHub
Fokko closed pull request #11678: Bug Fix: Expose tableExists endpoint on RESTCatalogAdapter URL: https://github.com/apache/iceberg/pull/11678 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.29.20 to 2.29.23 [iceberg]

2024-12-01 Thread via GitHub
Fokko merged PR #11683: URL: https://github.com/apache/iceberg/pull/11683 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] REST: Use HEAD request to check table existence [iceberg]

2024-12-01 Thread via GitHub
Fokko commented on PR #10999: URL: https://github.com/apache/iceberg/pull/10999#issuecomment-2510609023 @ebyhr Thanks, let's move this forward. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] REST: Use HEAD request to check table existence [iceberg]

2024-12-01 Thread via GitHub
Fokko merged PR #10999: URL: https://github.com/apache/iceberg/pull/10999 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Spark: Remove extra columns for ColumnBatch [iceberg]

2024-12-01 Thread via GitHub
flyrain commented on code in PR #11551: URL: https://github.com/apache/iceberg/pull/11551#discussion_r1865218949 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java: ## @@ -622,6 +623,50 @@ public void testPosDeletesOnParquetFileWithMul

Re: [I] Use `HEAD` operation for `tableExists` in REST-Catalog [iceberg]

2024-12-01 Thread via GitHub
Fokko closed issue #10993: Use `HEAD` operation for `tableExists` in REST-Catalog URL: https://github.com/apache/iceberg/issues/10993 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] Build: Bump mkdocs-material from 9.5.45 to 9.5.46 [iceberg]

2024-12-01 Thread via GitHub
Fokko merged PR #11680: URL: https://github.com/apache/iceberg/pull/11680 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Spark-3.5: make `where` sql case sensitive setting alterable in rewrite data files procedure [iceberg]

2024-12-01 Thread via GitHub
huaxingao commented on code in PR #11439: URL: https://github.com/apache/iceberg/pull/11439#discussion_r1865206027 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: ## @@ -153,6 +156,11 @@ public RewriteDataFilesSparkAction filt

Re: [I] Is iceberg support "Predicate Pushdown" when spark read data from it? [iceberg]

2024-12-01 Thread via GitHub
Akeron-Zhu commented on issue #11617: URL: https://github.com/apache/iceberg/issues/11617#issuecomment-2510526449 > Depends on the query, there are some finicky details there but sometimes (especially in early versions of Iceberg and Spark) predicates don't translate correctly. > > I

Re: [I] Sort Order Replacement API [iceberg-rust]

2024-12-01 Thread via GitHub
flaneur2020 commented on issue #734: URL: https://github.com/apache/iceberg-rust/issues/734#issuecomment-2510523752 take :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[PR] Added force virtual addressing configuration for S3 [iceberg-python]

2024-12-01 Thread via GitHub
helmiazizm opened a new pull request, #1392: URL: https://github.com/apache/iceberg-python/pull/1392 Added the option to enforce virtual hosted style request for S3 PyArrow IO as it already has the [support](https://github.com/apache/arrow/issues/39779). Also added `oss://` and `r2://` prot

Re: [PR] Feature: Write to branches [iceberg-python]

2024-12-01 Thread via GitHub
vinjai commented on code in PR #941: URL: https://github.com/apache/iceberg-python/pull/941#discussion_r1865177262 ## tests/table/test_init.py: ## @@ -982,28 +982,43 @@ def test_assert_table_uuid(table_v2: Table) -> None: def test_assert_ref_snapshot_id(table_v2: Table) -> No

Re: [PR] Feature: Write to branches [iceberg-python]

2024-12-01 Thread via GitHub
vinjai commented on code in PR #941: URL: https://github.com/apache/iceberg-python/pull/941#discussion_r1865176166 ## tests/table/test_init.py: ## @@ -982,28 +982,43 @@ def test_assert_table_uuid(table_v2: Table) -> None: def test_assert_ref_snapshot_id(table_v2: Table) -> No

Re: [PR] chore(deps): Bump actions/setup-python from 4 to 5 [iceberg-rust]

2024-12-01 Thread via GitHub
liurenjie1024 merged PR #746: URL: https://github.com/apache/iceberg-rust/pull/746 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [I] Why shouldn't we return an `UnboundPartitionSpec` instead? [iceberg-rust]

2024-12-01 Thread via GitHub
liurenjie1024 commented on issue #694: URL: https://github.com/apache/iceberg-rust/issues/694#issuecomment-2510475431 > @liurenjie1024, @Xuanwo, @Fokko may I ask for another round of Feedback for this? I believe if we decide to not introduce a `SchemalessPartitionSpec`, we should do so befo

[PR] Docs: Fix Spark catalog `table-override` description [iceberg]

2024-12-01 Thread via GitHub
manuzhang opened a new pull request, #11684: URL: https://github.com/apache/iceberg/pull/11684 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [I] Implement the equality delete writer [iceberg-rust]

2024-12-01 Thread via GitHub
liurenjie1024 closed issue #341: Implement the equality delete writer URL: https://github.com/apache/iceberg-rust/issues/341 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-12-01 Thread via GitHub
liurenjie1024 commented on PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#issuecomment-2510452060 > @ZENOTME thanks for getting back to me here. I don't want to be the one holding this up, so I would suggest that we get this in. There are still some gaps I see because I think

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-12-01 Thread via GitHub
liurenjie1024 commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1865150243 ## crates/iceberg/src/transaction.rs: ## @@ -122,6 +172,365 @@ impl<'a> Transaction<'a> { } } +/// FastAppendAction is a transaction action for fast ap

Re: [PR] AWS: Enable RetryMode for AWS KMS client [iceberg]

2024-12-01 Thread via GitHub
hsiang-c commented on code in PR #11420: URL: https://github.com/apache/iceberg/pull/11420#discussion_r1865124927 ## aws/src/main/java/org/apache/iceberg/aws/kms/KmsClientProperties.java: ## @@ -0,0 +1,45 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] AWS: Enable RetryMode for AWS KMS client [iceberg]

2024-12-01 Thread via GitHub
hsiang-c commented on code in PR #11420: URL: https://github.com/apache/iceberg/pull/11420#discussion_r1865124927 ## aws/src/main/java/org/apache/iceberg/aws/kms/KmsClientProperties.java: ## @@ -0,0 +1,45 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] IO Implementation using Go CDK [iceberg-go]

2024-12-01 Thread via GitHub
dwilson1988 commented on PR #176: URL: https://github.com/apache/iceberg-go/pull/176#issuecomment-2510407143 > Hi @dwilson1988, do you think you'll have time to take a look at this? I opened a PR on your branch earlier today -- This is an automated message from the Apache Git Servi

Re: [I] Should we remove the use of versionHintFile from the entire FileSystemCatalog? [iceberg]

2024-12-01 Thread via GitHub
github-actions[bot] commented on issue #10427: URL: https://github.com/apache/iceberg/issues/10427#issuecomment-2510325705 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] check-ordering enablement for flink config [iceberg]

2024-12-01 Thread via GitHub
github-actions[bot] closed issue #10360: check-ordering enablement for flink config URL: https://github.com/apache/iceberg/issues/10360 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] IO Implementation using Go CDK [iceberg-go]

2024-12-01 Thread via GitHub
loicalleyne commented on PR #176: URL: https://github.com/apache/iceberg-go/pull/176#issuecomment-2510372691 Hi @dwilson1988, do you think you'll have time to take a look at this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [I] Make ManifestEntry and ManifestReader.liveEntries() as public [iceberg]

2024-12-01 Thread via GitHub
github-actions[bot] commented on issue #10425: URL: https://github.com/apache/iceberg/issues/10425#issuecomment-2510325692 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] check-ordering enablement for flink config [iceberg]

2024-12-01 Thread via GitHub
github-actions[bot] commented on issue #10360: URL: https://github.com/apache/iceberg/issues/10360#issuecomment-2510325635 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Broken links in Spark Writes documentation [iceberg]

2024-12-01 Thread via GitHub
github-actions[bot] commented on issue #10409: URL: https://github.com/apache/iceberg/issues/10409#issuecomment-2510325671 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Questions on some requirement in view spec [iceberg]

2024-12-01 Thread via GitHub
github-actions[bot] commented on issue #10410: URL: https://github.com/apache/iceberg/issues/10410#issuecomment-2510325684 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] IO Implementation using Go CDK [iceberg-go]

2024-12-01 Thread via GitHub
dwilson1988 commented on code in PR #176: URL: https://github.com/apache/iceberg-go/pull/176#discussion_r1865055118 ## io/blob.go: ## @@ -0,0 +1,311 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file

[PR] WIP: Use localhost instead of container hostname [iceberg-rust]

2024-12-01 Thread via GitHub
Fokko opened a new pull request, #748: URL: https://github.com/apache/iceberg-rust/pull/748 To see if I can run this locally as well -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[PR] automate building Release Candidate in github action [iceberg-python]

2024-12-01 Thread via GitHub
kevinjqliu opened a new pull request, #1391: URL: https://github.com/apache/iceberg-python/pull/1391 This PR changes `.github/workflows/python-release.yml` to build the artifacts for both svn and pypi. * Add required input fields for `version` and `rc` * Validate input format and curre

Re: [PR] Spark-3.5: make `where` sql case sensitive setting alterable in rewrite data files procedure [iceberg]

2024-12-01 Thread via GitHub
huaxingao commented on PR #11439: URL: https://github.com/apache/iceberg/pull/11439#issuecomment-2510198274 @ludlows Thanks for the quick reply. I know my example will pass with the PR's fix. However, the problem will arise without the fix. We need a simple test that fails without the fix a

Re: [PR] Core: Add list/map block sizes [iceberg]

2024-12-01 Thread via GitHub
rustyconover commented on PR #10973: URL: https://github.com/apache/iceberg/pull/10973#issuecomment-2509886579 This still seems to be pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Docs: Default value of table level distribution-mode should be not set [iceberg]

2024-12-01 Thread via GitHub
manuzhang commented on code in PR #11663: URL: https://github.com/apache/iceberg/pull/11663#discussion_r1864934319 ## docs/docs/configuration.md: ## @@ -67,10 +67,10 @@ Iceberg tables support table properties to configure table behavior, like the de | write.metadata.metrics.co

[I] Can't seem to read a table (unexpected error, crashes JVM processes) [iceberg-python]

2024-12-01 Thread via GitHub
shanielh opened a new issue, #1390: URL: https://github.com/apache/iceberg-python/issues/1390 ### Apache Iceberg version None ### Please describe the bug 🐞 I tried both using DuckDB 1.1.3 (Latest as of this post) and 1.2.0-SNAPSHOT (using DBeaver). Reproduction:

Re: [PR] Spark-3.5: make `where` sql case sensitive setting alterable in rewrite data files procedure [iceberg]

2024-12-01 Thread via GitHub
ludlows commented on PR #11439: URL: https://github.com/apache/iceberg/pull/11439#issuecomment-2509767926 @huaxingao thanks for the comment. but i don't think the problem will be raised since the bug has been fixed by this PR. please check the test code belove: ```java @TestTem

Re: [I] Iceberg-rust Write support [iceberg-rust]

2024-12-01 Thread via GitHub
c-thiel commented on issue #700: URL: https://github.com/apache/iceberg-rust/issues/700#issuecomment-2509749712 > sorry came up late after a business trip, if there's still open task available, please assign to me 🙏 Sure :) How about https://github.com/apache/iceberg-rust/issues/7

[PR] chore(deps): Bump actions/setup-python from 4 to 5 [iceberg-rust]

2024-12-01 Thread via GitHub
dependabot[bot] opened a new pull request, #746: URL: https://github.com/apache/iceberg-rust/pull/746 Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4 to 5. Release notes Sourced from https://github.com/actions/setup-python/releases";>actions/setup-pytho

[PR] chore(deps): Bump crate-ci/typos from 1.27.3 to 1.28.1 [iceberg-rust]

2024-12-01 Thread via GitHub
dependabot[bot] opened a new pull request, #747: URL: https://github.com/apache/iceberg-rust/pull/747 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.27.3 to 1.28.1. Release notes Sourced from https://github.com/crate-ci/typos/releases";>crate-ci/typos's releases.

Re: [I] Implement nan_value_counts && distinct_counts metrics in parquet writer [iceberg-rust]

2024-12-01 Thread via GitHub
feniljain commented on issue #417: URL: https://github.com/apache/iceberg-rust/issues/417#issuecomment-2509721763 Hey @Fokko ! 👋🏻 As the original author has not replied, I am interested in taking it up :) Few points regardless who this gets assigned to: - I couldn't see

Re: [I] [DISCUSSION] Project Goal [iceberg-cpp]

2024-12-01 Thread via GitHub
Fokko commented on issue #2: URL: https://github.com/apache/iceberg-cpp/issues/2#issuecomment-2509645269 > What about creating yet another `iceberg::Field` to wrap `arrow::DataType` with better support of iceberg concepts? I think it should then wrap a `arrow: Field`, otherwise we wou

Re: [PR] Core/RewriteFiles: Duplicate Data Bug - Fixed dropping delete files that are still required [iceberg]

2024-12-01 Thread via GitHub
jasonf20 commented on code in PR #10962: URL: https://github.com/apache/iceberg/pull/10962#discussion_r1864787713 ## core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java: ## @@ -833,7 +833,17 @@ public List apply(TableMetadata base, Snapshot snapshot) { f

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.67.1 to 1.68.0 [iceberg-go]

2024-12-01 Thread via GitHub
dependabot[bot] commented on PR #217: URL: https://github.com/apache/iceberg-go/pull/217#issuecomment-2509629908 Superseded by #219. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.67.1 to 1.68.0 [iceberg-go]

2024-12-01 Thread via GitHub
dependabot[bot] closed pull request #217: build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.67.1 to 1.68.0 URL: https://github.com/apache/iceberg-go/pull/217 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Spark-3.5: make `where` sql case sensitive setting alterable in rewrite data files procedure [iceberg]

2024-12-01 Thread via GitHub
huaxingao commented on PR #11439: URL: https://github.com/apache/iceberg/pull/11439#issuecomment-2509628202 @ludlows I think you can simply reproduce the problem by something like ``` createTable(); insertData(10); sql("SET %s=false", SQLConf.CASE_SENSITIVE().key());

[PR] build(deps): bump github.com/pterm/pterm from 0.12.79 to 0.12.80 [iceberg-go]

2024-12-01 Thread via GitHub
dependabot[bot] opened a new pull request, #220: URL: https://github.com/apache/iceberg-go/pull/220 Bumps [github.com/pterm/pterm](https://github.com/pterm/pterm) from 0.12.79 to 0.12.80. Release notes Sourced from https://github.com/pterm/pterm/releases";>github.com/pterm/pterm's

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.67.1 to 1.69.0 [iceberg-go]

2024-12-01 Thread via GitHub
dependabot[bot] opened a new pull request, #219: URL: https://github.com/apache/iceberg-go/pull/219 Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.67.1 to 1.69.0. Commits https://github.com/aws/aws-sdk-go-v2/commit/03a0ffc0e9a9fcdf