Re: [I] ERROR when executing UPDATE/DELETE queries in Iceberg 1.6.0: "Cannot add fieldId 1 as an identifier field" [iceberg]

2024-11-24 Thread via GitHub
y2k6879 commented on issue #11341: URL: https://github.com/apache/iceberg/issues/11341#issuecomment-2497144763 I got same issue I run the query below with spark sql, ``` CREATE OR REPLACE TABLE my_db.my_table ( id_iceberg STRING, ... edit_datetime TIMESTAMP,

Re: [I] `catalog.table_exists()` returns 'False' when table exists in Polaris catalog [iceberg-python]

2024-11-24 Thread via GitHub
JasperHG90 commented on issue #1363: URL: https://github.com/apache/iceberg-python/issues/1363#issuecomment-2497134805 FYI PR that fixes status code for this and other endpoints: https://github.com/apache/polaris/pull/472 -- This is an automated message from the Apache Git Service. To re

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.29.15 to 2.29.20 [iceberg]

2024-11-24 Thread via GitHub
Fokko merged PR #11639: URL: https://github.com/apache/iceberg/pull/11639 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Core: Add support for `view-default` property in catalog [iceberg]

2024-11-24 Thread via GitHub
nastra commented on code in PR #11064: URL: https://github.com/apache/iceberg/pull/11064#discussion_r1855997389 ## core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java: ## @@ -637,6 +637,30 @@ public void testCompleteCreateTable() { .isEqualTo(UUID.fromString

Re: [PR] Spark 3.3: Correct the two-stage parsing strategy of antlr parser [iceberg]

2024-11-24 Thread via GitHub
Fokko commented on PR #11630: URL: https://github.com/apache/iceberg/pull/11630#issuecomment-2497128393 Thanks @pan3793 for fixing this, and thanks @nastra for the review 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Fix when reading struct-type data without an id in iceberg-parquet [iceberg]

2024-11-24 Thread via GitHub
nastra commented on PR #11378: URL: https://github.com/apache/iceberg/pull/11378#issuecomment-2497128317 @joyCurry30 please ping me once you added the test and I'll take a look -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [PR] Spark 3.3: Correct the two-stage parsing strategy of antlr parser [iceberg]

2024-11-24 Thread via GitHub
Fokko merged PR #11630: URL: https://github.com/apache/iceberg/pull/11630 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Hadoop: Log where the missing metadata file is located [iceberg]

2024-11-24 Thread via GitHub
nastra commented on code in PR #11643: URL: https://github.com/apache/iceberg/pull/11643#discussion_r1856001804 ## core/src/test/java/org/apache/iceberg/hadoop/TestHadoopCatalog.java: ## @@ -548,6 +549,32 @@ public void testVersionHintFileMissingMetadata() throws Exception {

Re: [PR] Build: Bump mkdocs-material from 9.5.44 to 9.5.45 [iceberg]

2024-11-24 Thread via GitHub
Fokko merged PR #11641: URL: https://github.com/apache/iceberg/pull/11641 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Build: Bump testcontainers from 1.20.3 to 1.20.4 [iceberg]

2024-11-24 Thread via GitHub
Fokko merged PR #11640: URL: https://github.com/apache/iceberg/pull/11640 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Materialized View Spec [iceberg]

2024-11-24 Thread via GitHub
JanKaul commented on code in PR #11041: URL: https://github.com/apache/iceberg/pull/11041#discussion_r1855985153 ## format/view-spec.md: ## @@ -42,12 +42,27 @@ An atomic swap of one view metadata file for another provides the basis for maki Writers create view metadata files

[I] Flink Use distribution-mode: RANGE ,partition bucket become [iceberg]

2024-11-24 Thread via GitHub
Guosmilesmile opened a new issue, #11644: URL: https://github.com/apache/iceberg/issues/11644 ### Apache Iceberg version 1.7.0 (latest release) ### Query engine Flink ### Please describe the bug 🐞 ![1](https://github.com/user-attachments/assets/0fb37d17-5cb

Re: [PR] Support changelog scan for table with delete files [iceberg]

2024-11-24 Thread via GitHub
wypoon commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2496928372 @aokolnychyi I have rebased the PR on main. How can we move forward with this? If we agree that the behavior is correct, how about we at least have something that works and optimize i

Re: [PR] fix `KeyError` raised by `add_files` when parquet file doe not have column stats [iceberg-python]

2024-11-24 Thread via GitHub
binayakd commented on PR #1354: URL: https://github.com/apache/iceberg-python/pull/1354#issuecomment-2496921401 @kevinjqliu pushed the test readability fix. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] Add Python Release Action to publish `pyiceberg_core` dist to Pypi [iceberg-rust]

2024-11-24 Thread via GitHub
Xuanwo merged PR #705: URL: https://github.com/apache/iceberg-rust/pull/705 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Create publish-docker.yml [iceberg]

2024-11-24 Thread via GitHub
sungwy commented on PR #11632: URL: https://github.com/apache/iceberg/pull/11632#issuecomment-2496635598 > in addition to pushing the latest, it would be great to publish images tagged with specific releases (1.7/1.8/etc) also, I found the GitHub action that generates the hive docker image

Re: [PR] Build: Bump testcontainers from 1.20.3 to 1.20.4 [iceberg]

2024-11-24 Thread via GitHub
sullis commented on PR #11640: URL: https://github.com/apache/iceberg/pull/11640#issuecomment-2496601314 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [PR] Bugfix for incorrect Deletion of Snapshot Metadata Due to OutOfMemoryError [iceberg]

2024-11-24 Thread via GitHub
ZhendongBai commented on PR #11576: URL: https://github.com/apache/iceberg/pull/11576#issuecomment-2496591462 > @ZhendongBai: Thanks for the fix, and sorry for missing your reply. Also thanks @RussellSpitzer for merging! 😄 @pvary ok,and thanks again. -- This is an automated message

Re: [PR] feat: Add equality delete writer [iceberg-rust]

2024-11-24 Thread via GitHub
ZENOTME commented on code in PR #703: URL: https://github.com/apache/iceberg-rust/pull/703#discussion_r1855711268 ## crates/iceberg/src/writer/base_writer/equality_delete_writer.rs: ## @@ -0,0 +1,544 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more c

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-11-24 Thread via GitHub
ZENOTME commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1855708285 ## crates/iceberg/src/transaction.rs: ## @@ -96,6 +109,60 @@ impl<'a> Transaction<'a> { Ok(self) } +fn generate_unique_snapshot_id(&self) -> i64

Re: [PR] Document procedure for stats collection [iceberg]

2024-11-24 Thread via GitHub
manuzhang commented on code in PR #11606: URL: https://github.com/apache/iceberg/pull/11606#discussion_r1855696945 ## docs/docs/spark-procedures.md: ## @@ -936,3 +936,40 @@ as an `UPDATE_AFTER` image, resulting in the following pre/post update images: |-||-

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-11-24 Thread via GitHub
ZENOTME commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1855695281 ## crates/iceberg/src/transaction.rs: ## @@ -122,6 +189,383 @@ impl<'a> Transaction<'a> { } } +/// FastAppendAction is a transaction action for fast append d

Re: [PR] feat: Add equality delete writer [iceberg-rust]

2024-11-24 Thread via GitHub
liurenjie1024 commented on code in PR #703: URL: https://github.com/apache/iceberg-rust/pull/703#discussion_r1855688069 ## crates/iceberg/src/writer/base_writer/equality_delete_writer.rs: ## @@ -0,0 +1,538 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or

Re: [PR] Document procedure for stats collection [iceberg]

2024-11-24 Thread via GitHub
manuzhang commented on code in PR #11606: URL: https://github.com/apache/iceberg/pull/11606#discussion_r1855691038 ## docs/docs/spark-procedures.md: ## @@ -936,3 +936,40 @@ as an `UPDATE_AFTER` image, resulting in the following pre/post update images: |-||-

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-11-24 Thread via GitHub
ZENOTME commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1855684639 ## crates/iceberg/src/transaction.rs: ## @@ -122,6 +189,383 @@ impl<'a> Transaction<'a> { } } +/// FastAppendAction is a transaction action for fast append d

Re: [PR] API: Support removeUnusedSpecs in ExpireSnapshots [iceberg]

2024-11-24 Thread via GitHub
advancedxy commented on PR #10755: URL: https://github.com/apache/iceberg/pull/10755#issuecomment-2496541354 @nastra @amogh-jahagirdar PTAL again, I think all your comments are addressed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] feat: support append data file and add e2e test [iceberg-rust]

2024-11-24 Thread via GitHub
ZENOTME commented on code in PR #349: URL: https://github.com/apache/iceberg-rust/pull/349#discussion_r1855682480 ## crates/iceberg/src/transaction.rs: ## @@ -122,6 +189,383 @@ impl<'a> Transaction<'a> { } } +/// FastAppendAction is a transaction action for fast append d

Re: [I] Consider Using object_store as IO Abstraction [iceberg-rust]

2024-11-24 Thread via GitHub
liurenjie1024 commented on issue #172: URL: https://github.com/apache/iceberg-rust/issues/172#issuecomment-2496527112 Hi, @BlakeOrth Thanks for trying this, and yes it's quite close to what's in my mind. So the question is that we can't keep the return type of `reader`/`writer` function? I

Re: [I] Consider Using object_store as IO Abstraction [iceberg-rust]

2024-11-24 Thread via GitHub
liurenjie1024 commented on issue #172: URL: https://github.com/apache/iceberg-rust/issues/172#issuecomment-2496515292 > I _think_ that should work, the DataFusion wrapper can just hook the iceberg metadata operations into via that StorageProvider trait, and then use the DataFusion machinery

[PR] Docs: Add RisiingWave [iceberg]

2024-11-24 Thread via GitHub
hengm3467 opened a new pull request, #11642: URL: https://github.com/apache/iceberg/pull/11642 Adds an overview page about RisingWave's support for Iceberg to help developers get started with the RisingWave Iceberg integration. -- This is an automated message from the Apache Git Service.

Re: [I] Review new DangerousJavaDeserialization error-prone check [iceberg]

2024-11-24 Thread via GitHub
RussellSpitzer commented on issue #10853: URL: https://github.com/apache/iceberg/issues/10853#issuecomment-2496477160 We generally do not assign issues since we get a pretty low follow through. That said please let us know when you have a PR ready -- This is an automated message from the

Re: [I] Add view support for Hadoop catalog [iceberg]

2024-11-24 Thread via GitHub
github-actions[bot] commented on issue #10387: URL: https://github.com/apache/iceberg/issues/10387#issuecomment-2496442524 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] Feature/otf 1500 column comparisons 1521 [iceberg]

2024-11-24 Thread via GitHub
github-actions[bot] commented on PR #11164: URL: https://github.com/apache/iceberg/pull/11164#issuecomment-2496442593 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Feature/otf 1500 column comparisons 1521 [iceberg]

2024-11-24 Thread via GitHub
github-actions[bot] closed pull request #11164: Feature/otf 1500 column comparisons 1521 URL: https://github.com/apache/iceberg/pull/11164 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [I] Copy iceberg table from hdfs to GCS and register table to BLMS [iceberg]

2024-11-24 Thread via GitHub
github-actions[bot] commented on issue #10389: URL: https://github.com/apache/iceberg/issues/10389#issuecomment-2496442540 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Iceberg Roadmap is 404 [iceberg]

2024-11-24 Thread via GitHub
github-actions[bot] commented on issue #10390: URL: https://github.com/apache/iceberg/issues/10390#issuecomment-2496442558 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Delete orphan files [iceberg-python]

2024-11-24 Thread via GitHub
ndrluis commented on issue #1200: URL: https://github.com/apache/iceberg-python/issues/1200#issuecomment-2496257742 @omkenge I believe you will need to wait for the merge of #1285. In the meantime, I will work on the partition statistics over the next few weeks. Before that, I believe we w

Re: [PR] Add new standards for deprecating APIs [iceberg-python]

2024-11-24 Thread via GitHub
ndrluis commented on PR #1365: URL: https://github.com/apache/iceberg-python/pull/1365#issuecomment-2496258494 I copied the tests from Conda, but I think they aren't clear enough, so I plan to rewrite all of them. -- This is an automated message from the Apache Git Service. To respond to

Re: [I] Do not deprecate Botocore Session in upcoming release (0.8) [iceberg-python]

2024-11-24 Thread via GitHub
cshenrik commented on issue #1104: URL: https://github.com/apache/iceberg-python/issues/1104#issuecomment-2496242084 > @cshenrik I have the same issue of trying to utilize AWS profile that hits a web service to drive automatically refreshable credentials. Did you find any solution to this?

[PR] feat(puffin): Add reader and writer [iceberg-rust]

2024-11-24 Thread via GitHub
fqaiser94 opened a new pull request, #714: URL: https://github.com/apache/iceberg-rust/pull/714 # Not Ready For Reviews I don't recommend reviewing this formally just yet as I plan to break this up into multiple PRs to make it easier to review and I also have some minor TODOs to complete

Re: [I] Review new DangerousJavaDeserialization error-prone check [iceberg]

2024-11-24 Thread via GitHub
aadvik93 commented on issue #10853: URL: https://github.com/apache/iceberg/issues/10853#issuecomment-2496217605 Hi, I'd like to work on this issue. Could you please assign it to me? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] Spec: Support geo type [iceberg]

2024-11-24 Thread via GitHub
jiayuasu commented on PR #10981: URL: https://github.com/apache/iceberg/pull/10981#issuecomment-2496207469 @dmitrykoval The behavior of `spherical` mode is very inconsistent among different engines. It is not an easy job to come up with a edge interpolation standard that makes everybody hap

Re: [PR] Spec: Support geo type [iceberg]

2024-11-24 Thread via GitHub
dmitrykoval commented on PR #10981: URL: https://github.com/apache/iceberg/pull/10981#issuecomment-2496204899 @jiayuasu @szehon-ho Just to clarify, are there any concerns with adding back the `edges` property aside from improving the definition of bounding box stats semantics? -- This is

Re: [PR] Document procedure for stats collection [iceberg]

2024-11-24 Thread via GitHub
szehon-ho commented on code in PR #11606: URL: https://github.com/apache/iceberg/pull/11606#discussion_r1855513995 ## docs/docs/spark-procedures.md: ## @@ -936,3 +936,40 @@ as an `UPDATE_AFTER` image, resulting in the following pre/post update images: |-||-

Re: [PR] Bump pydantic from 2.10.0 to 2.10.1 [iceberg-python]

2024-11-24 Thread via GitHub
Fokko merged PR #1364: URL: https://github.com/apache/iceberg-python/pull/1364 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Bump getdaft from 0.3.13 to 0.3.14 [iceberg-python]

2024-11-24 Thread via GitHub
Fokko merged PR #1361: URL: https://github.com/apache/iceberg-python/pull/1361 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [I] [DISCUSSION] Project Goal [iceberg-cpp]

2024-11-24 Thread via GitHub
zeroshade commented on issue #2: URL: https://github.com/apache/iceberg-cpp/issues/2#issuecomment-2496089511 For Arrow decimal types you'll need to specify which decimal type to use, I recommend using 128-bit because that's the max supported by iceberg. For the geometry type, you can

Re: [I] [DISCUSSION] Project Goal [iceberg-cpp]

2024-11-24 Thread via GitHub
wgtmac commented on issue #2: URL: https://github.com/apache/iceberg-cpp/issues/2#issuecomment-2496086676 Thanks @zeroshade for the detail! The table below is the type mapping between iceberg and arrow. I think we can provide a wrapper around arrow data types to use only a subset of

Re: [PR] Document procedure for stats collection [iceberg]

2024-11-24 Thread via GitHub
RussellSpitzer commented on PR #11606: URL: https://github.com/apache/iceberg/pull/11606#issuecomment-2496041028 That would mean documenting it in the method itself not the description of the return value On Sun, Nov 24, 2024 at 2:33 AM Manu Zhang ***@***.***> wrote: > ***@**

Re: [PR] Core: Add support for `view-default` property in catalog [iceberg]

2024-11-24 Thread via GitHub
ebyhr commented on code in PR #11064: URL: https://github.com/apache/iceberg/pull/11064#discussion_r1855449851 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -1200,6 +1200,8 @@ private RESTViewBuilder(SessionContext context, TableIdentifier identif

Re: [PR] Document procedure for stats collection [iceberg]

2024-11-24 Thread via GitHub
manuzhang commented on code in PR #11606: URL: https://github.com/apache/iceberg/pull/11606#discussion_r1855394563 ## docs/docs/spark-procedures.md: ## @@ -936,3 +936,40 @@ as an `UPDATE_AFTER` image, resulting in the following pre/post update images: |-||-

[PR] build(deps): bump github.com/stretchr/testify from 1.9.0 to 1.10.0 [iceberg-go]

2024-11-24 Thread via GitHub
dependabot[bot] opened a new pull request, #218: URL: https://github.com/apache/iceberg-go/pull/218 Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.9.0 to 1.10.0. Release notes Sourced from https://github.com/stretchr/testify/releases";>github.com/st

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.67.1 to 1.68.0 [iceberg-go]

2024-11-24 Thread via GitHub
dependabot[bot] opened a new pull request, #217: URL: https://github.com/apache/iceberg-go/pull/217 Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.67.1 to 1.68.0. Commits https://github.com/aws/aws-sdk-go-v2/commit/18fea5d7df1f6a92