Re: [I] Support `initial-default` when reading Avro [iceberg-rust]

Fokko commented on issue #119: URL: https://github.com/apache/iceberg-rust/issues/119#issuecomment-1855355460 > If we read a V1 Avro file as a V2 object, when we write back, we also need to set [format-version in manifest](https://iceberg.apache.org/spec/#manifests:~:text=required-,format%2

Re: [PR] Add Hive integration tests [iceberg-python]

HonahX commented on code in PR #207: URL: https://github.com/apache/iceberg-python/pull/207#discussion_r1426357825 ## tests/integration/test_hive.py: ## @@ -0,0 +1,409 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. Se

Re: [PR] Add Hive integration tests [iceberg-python]

HonahX commented on code in PR #207: URL: https://github.com/apache/iceberg-python/pull/207#discussion_r1426280842 ## tests/integration/__init__.py: ## @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

Re: [PR] test: Remove binary manifest list avro file [iceberg-rust]

Fokko commented on code in PR #118: URL: https://github.com/apache/iceberg-rust/pull/118#discussion_r1426366787 ## crates/iceberg/src/spec/manifest.rs: ## @@ -1311,511 +1311,452 @@ mod tests { use crate::spec::Type; use std::sync::Arc; -#[test] -fn test_parse

Re: [PR] Add Hive integration tests [iceberg-python]

HonahX commented on code in PR #207: URL: https://github.com/apache/iceberg-python/pull/207#discussion_r1426357825 ## tests/integration/test_hive.py: ## @@ -0,0 +1,409 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. Se

Re: [PR] test: Remove binary manifest list avro file [iceberg-rust]

Fokko commented on code in PR #118: URL: https://github.com/apache/iceberg-rust/pull/118#discussion_r1426370018 ## crates/iceberg/src/spec/manifest_list.rs: ## @@ -931,55 +931,104 @@ mod test { use super::_serde::ManifestListV2; -#[test] -fn test_parse_manifest_

Re: [I] Support `initial-default` when reading Avro [iceberg-rust]

ZENOTME commented on issue #119: URL: https://github.com/apache/iceberg-rust/issues/119#issuecomment-1855364719 > > If we read a V1 Avro file as a V2 object, when we write back, we also need to set [format-version in manifest](https://iceberg.apache.org/spec/#manifests:~:text=required-,form

Re: [PR] Restore RuntimeIOException for use [iceberg]

Fokko merged PR #5640: URL: https://github.com/apache/iceberg/pull/5640 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [I] Remove `RuntimeIOException` usage [iceberg]

ajantha-bhat closed issue #9208: Remove `RuntimeIOException` usage URL: https://github.com/apache/iceberg/issues/9208 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

Re: [I] Remove `RuntimeIOException` usage [iceberg]

ajantha-bhat commented on issue #9208: URL: https://github.com/apache/iceberg/issues/9208#issuecomment-1855382609 Fixed by https://github.com/apache/iceberg/pull/5640 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Replace deprecated RuntimeIOException with UncheckedIOException [iceberg]

ajantha-bhat commented on PR #4776: URL: https://github.com/apache/iceberg/pull/4776#issuecomment-1855384162 Closing as we concluded to undo deprecation. Fixed by https://github.com/apache/iceberg/pull/5640 -- This is an automated message from the Apache Git Service. To respond to t

Re: [PR] Replace deprecated RuntimeIOException with UncheckedIOException [iceberg]

ajantha-bhat closed pull request #4776: Replace deprecated RuntimeIOException with UncheckedIOException URL: https://github.com/apache/iceberg/pull/4776 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] API: Replace special case of deprecated RuntimeIOException [iceberg]

ajantha-bhat closed pull request #6887: API: Replace special case of deprecated RuntimeIOException URL: https://github.com/apache/iceberg/pull/6887 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] API: Replace special case of deprecated RuntimeIOException [iceberg]

ajantha-bhat commented on PR #6887: URL: https://github.com/apache/iceberg/pull/6887#issuecomment-1855384419 Closing as we concluded to undo deprecation. Fixed by https://github.com/apache/iceberg/pull/5640 -- This is an automated message from the Apache Git Service. To respond to t

Re: [PR] test: Remove binary manifest list avro file [iceberg-rust]

Fokko commented on code in PR #118: URL: https://github.com/apache/iceberg-rust/pull/118#discussion_r1426386121 ## Makefile: ## @@ -34,6 +34,9 @@ cargo-sort: check: check-fmt check-clippy cargo-sort +unit-test: Review Comment: Are these ran in the CI? -- This is an

Re: [PR] WIP: Glue catalog commit [iceberg-python]

nicor88 commented on code in PR #140: URL: https://github.com/apache/iceberg-python/pull/140#discussion_r1426412419 ## pyiceberg/catalog/glue.py: ## @@ -247,8 +278,50 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons Raises:

Re: [PR] test: Remove binary manifest list avro file [iceberg-rust]

liurenjie1024 commented on code in PR #118: URL: https://github.com/apache/iceberg-rust/pull/118#discussion_r1426414222 ## Makefile: ## @@ -34,6 +34,9 @@ cargo-sort: check: check-fmt check-clippy cargo-sort +unit-test: Review Comment: The ci will run all tests, using th

Re: [PR] WIP: Glue catalog commit [iceberg-python]

nicor88 commented on code in PR #140: URL: https://github.com/apache/iceberg-python/pull/140#discussion_r1426413605 ## pyiceberg/catalog/glue.py: ## @@ -177,6 +191,23 @@ def _create_glue_table(self, database_name: str, table_name: str, table_input: T except self.glue.e

Re: [I] An exception occurred while writing iceberg data through Spark: org. apache. iceberg. exceptions. CommitFailedException: metadata location has changed [iceberg]

AllenWee1106 commented on issue #9178: URL: https://github.com/apache/iceberg/issues/9178#issuecomment-1855434363 @Zhangg7723 thx, how to change the isolation level to snapshot -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] test: Remove binary manifest list avro file [iceberg-rust]

liurenjie1024 commented on code in PR #118: URL: https://github.com/apache/iceberg-rust/pull/118#discussion_r1426434871 ## crates/iceberg/src/spec/manifest.rs: ## @@ -1311,511 +1311,452 @@ mod tests { use crate::spec::Type; use std::sync::Arc; -#[test] -fn te

Re: [PR] test: Remove binary manifest list avro file [iceberg-rust]

liurenjie1024 commented on code in PR #118: URL: https://github.com/apache/iceberg-rust/pull/118#discussion_r1426434871 ## crates/iceberg/src/spec/manifest.rs: ## @@ -1311,511 +1311,452 @@ mod tests { use crate::spec::Type; use std::sync::Arc; -#[test] -fn te

[PR] [WIP] Core: Remove deprecated classes related to rewrite data files [iceberg]

ajantha-bhat opened a new pull request, #9296: URL: https://github.com/apache/iceberg/pull/9296 Waiting for https://github.com/apache/iceberg/pull/9295 to be merged to continue work on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [I] Support 'schema.name-mapping.default' Column Projection property [iceberg-python]

Fokko commented on issue #202: URL: https://github.com/apache/iceberg-python/issues/202#issuecomment-1855458645 As discussed on Slack, I've taken the liberty to get in [all the name-mapping plumbing](https://github.com/apache/iceberg-python/pull/212) ((de)serializers, visitors etc). The thi

Re: [I] show table extended not supported for v2 table. [iceberg]

Peeyush-Now commented on issue #5782: URL: https://github.com/apache/iceberg/issues/5782#issuecomment-1855458779 +1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] Add name-mapping [iceberg-python]

Fokko commented on code in PR #212: URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1426443434 ## pyiceberg/table/name_mapping.py: ## @@ -0,0 +1,204 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. Se

Re: [PR] test: Remove binary manifest list avro file [iceberg-rust]

Fokko merged PR #118: URL: https://github.com/apache/iceberg-rust/pull/118 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [I] test: Replace binary avro file by generating it on the fly. [iceberg-rust]

Fokko closed issue #70: test: Replace binary avro file by generating it on the fly. URL: https://github.com/apache/iceberg-rust/issues/70 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] test: Remove binary manifest list avro file [iceberg-rust]

Fokko commented on PR #118: URL: https://github.com/apache/iceberg-rust/pull/118#issuecomment-1855466917 Thanks again @liurenjie1024 for working on this. @Xuanwo and @ZENOTME thanks for the review 👍 -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [PR] Test: Fix flaky tests which concurrently modify HashSet [iceberg]

ajantha-bhat commented on code in PR #9294: URL: https://github.com/apache/iceberg/pull/9294#discussion_r1426446921 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRemoveOrphanFilesAction.java: ## @@ -270,7 +270,7 @@ public void orphanedFileRemovedWithPara

Re: [PR] Test: Fix flaky tests which concurrently modify HashSet [iceberg]

nastra commented on code in PR #9294: URL: https://github.com/apache/iceberg/pull/9294#discussion_r1426452612 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRemoveOrphanFilesAction.java: ## @@ -270,7 +270,7 @@ public void orphanedFileRemovedWithParallelTa

Re: [PR] Test: Fix flaky tests which concurrently modify HashSet [iceberg]

manuzhang commented on code in PR #9294: URL: https://github.com/apache/iceberg/pull/9294#discussion_r1426461042 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRemoveOrphanFilesAction.java: ## @@ -270,7 +270,7 @@ public void orphanedFileRemovedWithParalle

Re: [PR] Adding Snowflake's public documentation [iceberg-docs]

Fokko commented on code in PR #297: URL: https://github.com/apache/iceberg-docs/pull/297#discussion_r1426485040 ## docs/config.toml: ## @@ -78,11 +78,12 @@ home = [ "HTML", "RSS", "SearchIndex" ] { name = "StarRocks", identifier = "_starrocks", weight = 701, url = "https:/

Re: [PR] Spark: Remove support for Spark 3.2 [iceberg]

nastra commented on PR #9295: URL: https://github.com/apache/iceberg/pull/9295#issuecomment-1855532540 thanks @ajantha-bhat for getting this done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] Spark: Remove support for Spark 3.2 [iceberg]

nastra merged PR #9295: URL: https://github.com/apache/iceberg/pull/9295 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [I] Remove Spark-3.2 support for Iceberg 1.5.0 and onwards. [iceberg]

nastra closed issue #9290: Remove Spark-3.2 support for Iceberg 1.5.0 and onwards. URL: https://github.com/apache/iceberg/issues/9290 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [I] Unable to merge CDC data into snapshot data. java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Long [iceberg]

sabyasachinandy commented on issue #8333: URL: https://github.com/apache/iceberg/issues/8333#issuecomment-183371 Any update on this? Given this is a bread and butter usecase and still no one cares for looking into this in priority. -- This is an automated message from the Apache Git

Re: [PR] Core: Look up targeted position deletes by path [iceberg]

aokolnychyi commented on PR #9251: URL: https://github.com/apache/iceberg/pull/9251#issuecomment-1855575637 @szehon-ho, could you take another look? I did some updates and added a benchmark. The performance is OK for now but we can definitely optimize it further in following PRs. -- This

Re: [PR] Core: Look up targeted position deletes by path [iceberg]

aokolnychyi commented on code in PR #9251: URL: https://github.com/apache/iceberg/pull/9251#discussion_r1426529884 ## core/src/main/java/org/apache/iceberg/DeleteFileIndex.java: ## @@ -474,68 +420,64 @@ private Collection loadDeleteFiles() { DeleteFileIndex build() {

Re: [PR] Core: Look up targeted position deletes by path [iceberg]

aokolnychyi commented on code in PR #9251: URL: https://github.com/apache/iceberg/pull/9251#discussion_r1426529884 ## core/src/main/java/org/apache/iceberg/DeleteFileIndex.java: ## @@ -474,68 +420,64 @@ private Collection loadDeleteFiles() { DeleteFileIndex build() {

Re: [PR] Core: Look up targeted position deletes by path [iceberg]

aokolnychyi commented on code in PR #9251: URL: https://github.com/apache/iceberg/pull/9251#discussion_r1425370701 ## core/src/main/java/org/apache/iceberg/util/ArrayUtil.java: ## @@ -320,4 +320,29 @@ public static boolean isStrictlyAscending(long[] array) { return true;

Re: [I] Flaky test: TestRemoveOrphanFilesAction3 > orphanedFileRemovedWithParallelTasks [iceberg]

nastra closed issue #8824: Flaky test: TestRemoveOrphanFilesAction3 > orphanedFileRemovedWithParallelTasks URL: https://github.com/apache/iceberg/issues/8824 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] Test: Fix flaky tests which concurrently modify HashSet [iceberg]

nastra merged PR #9294: URL: https://github.com/apache/iceberg/pull/9294 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Remove redundant error propagation check. [iceberg]

pvary commented on code in PR #9143: URL: https://github.com/apache/iceberg/pull/9143#discussion_r1426539287 ## common/src/main/java/org/apache/iceberg/common/DynConstructors.java: ## @@ -58,7 +58,6 @@ public C newInstanceChecked(Object... args) throws Exception { throw

Re: [PR] Remove redundant error propagation check. [iceberg]

pvary commented on code in PR #9143: URL: https://github.com/apache/iceberg/pull/9143#discussion_r1426541260 ## common/src/main/java/org/apache/iceberg/common/DynConstructors.java: ## @@ -67,7 +66,6 @@ public C newInstance(Object... args) { try { return newInstan

Re: [PR] Remove redundant error propagation check. [iceberg]

pvary commented on code in PR #9143: URL: https://github.com/apache/iceberg/pull/9143#discussion_r1426541867 ## common/src/main/java/org/apache/iceberg/common/DynMethods.java: ## @@ -62,7 +62,6 @@ public R invokeChecked(Object target, Object... args) throws Exception {

Re: [PR] Remove redundant error propagation check. [iceberg]

pvary commented on code in PR #9143: URL: https://github.com/apache/iceberg/pull/9143#discussion_r1426542220 ## common/src/main/java/org/apache/iceberg/common/DynMethods.java: ## @@ -71,7 +70,6 @@ public R invoke(Object target, Object... args) { try { return thi

Re: [PR] Spark3.1, Spark3.2, Spark3.3: Support setting current snapshot with ref [iceberg]

manuzhang closed pull request #8392: Spark3.1, Spark3.2, Spark3.3: Support setting current snapshot with ref URL: https://github.com/apache/iceberg/pull/8392 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

pvary commented on PR #8918: URL: https://github.com/apache/iceberg/pull/8918#issuecomment-1855610235 Thanks for all the work you put into this @nk1506 and @nastra! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [I] Apache Flink not committing new snapshots to Iceberg Table [iceberg]

FranMorilloAWS commented on issue #9089: URL: https://github.com/apache/iceberg/issues/9089#issuecomment-1855633844 I have the following issue which i believe is a bug. I have a Flink job that is consuming from one kinesis data stream and writes into multiple different Iceberg Tables. Whene

Re: [PR] Flink 1.17: Support specifying equality columns with write options [iceberg]

manuzhang commented on PR #8195: URL: https://github.com/apache/iceberg/pull/8195#issuecomment-1855640410 @stevenzwu @pvary please help review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Spark 3.2: Output the net changes across snapshots for carryover rows in CDC [iceberg]

ajantha-bhat commented on PR #7952: URL: https://github.com/apache/iceberg/pull/7952#issuecomment-1855665615 Closing this as Spark 3.2 is no longer supported in the latest version. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [PR] Spark 3.2: Output the net changes across snapshots for carryover rows in CDC [iceberg]

ajantha-bhat closed pull request #7952: Spark 3.2: Output the net changes across snapshots for carryover rows in CDC URL: https://github.com/apache/iceberg/pull/7952 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Spark 3.2: Optimized add_files procedure's listPartitions [iceberg]

ajantha-bhat closed pull request #7363: Spark 3.2: Optimized add_files procedure's listPartitions URL: https://github.com/apache/iceberg/pull/7363 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Spark 3.2: Optimized add_files procedure's listPartitions [iceberg]

ajantha-bhat commented on PR #7363: URL: https://github.com/apache/iceberg/pull/7363#issuecomment-1855668224 Closing this as Spark 3.2 is no longer supported in the latest version. If issue exist on other spark version, please handle there. -- This is an automated message from the

Re: [PR] Spark 3.2: Commit consumed offsets to the checkpoint location [iceberg]

ajantha-bhat commented on PR #4473: URL: https://github.com/apache/iceberg/pull/4473#issuecomment-1855668600 Closing this as Spark 3.2 is no longer supported in the latest version. If issue exist on other spark version, please handle there. -- This is an automated message from the A

Re: [PR] Spark 3.2: Commit consumed offsets to the checkpoint location [iceberg]

ajantha-bhat closed pull request #4473: Spark 3.2: Commit consumed offsets to the checkpoint location URL: https://github.com/apache/iceberg/pull/4473 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Apache Flink not committing new snapshots to Iceberg Table [iceberg]

pvary commented on issue #9089: URL: https://github.com/apache/iceberg/issues/9089#issuecomment-1855679569 @FranMorilloAWS: Do you have data committed to the tables where the checkpointId is not increased? Normally, if Flink sees a commit without additional files, it just skips creating

Re: [I] Unable to merge CDC data into snapshot data. java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Long [iceberg]

nastra commented on issue #8333: URL: https://github.com/apache/iceberg/issues/8333#issuecomment-1855684168 Sorry that it feels that nobody is actively looking into this. People might be busy with other things or might not be aware about this issue. To raise awareness and get some additi

Re: [PR] Flink 1.17: Support specifying equality columns with write options [iceberg]

pvary commented on PR #8195: URL: https://github.com/apache/iceberg/pull/8195#issuecomment-1855685224 > Currently, equality columns are parsed from primary key specified in Flink create table statement. However, an Iceberg table might have been created without primary key, but we still want

Re: [I] Unable to merge CDC data into snapshot data. java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Long [iceberg]

sabyasachinandy commented on issue #8333: URL: https://github.com/apache/iceberg/issues/8333#issuecomment-1855701368 I will do @nastra. I apologize for the rude tone. Surely I will mention it in the mailing list or maybe use slack -- This is an automated message from the Apache Git Servic

[I] hive4 MapReduce query execution engine support [iceberg]

wfy-belief opened a new issue, #9297: URL: https://github.com/apache/iceberg/issues/9297 ### Feature Request / Improvement now, when i use hive:4.0.0-beta-2-SNAPSHOT i can write data in a iceberg table，but can't query data，so i read the doc of https://iceberg.apache.org/docs/latest/h

Re: [PR] [WIP] Core: Remove deprecated classes related to rewrite data files [iceberg]

ajantha-bhat commented on code in PR #9296: URL: https://github.com/apache/iceberg/pull/9296#discussion_r1426650686 ## spark/v3.3/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java: ## @@ -35,7 +35,7 @@ import org.apache.iceberg.SortDirection

Re: [PR] Flink 1.17: Support specifying equality columns with write options [iceberg]

manuzhang commented on PR #8195: URL: https://github.com/apache/iceberg/pull/8195#issuecomment-1855777652 @pvary I've updated my description. Our use case is that the table is created in Spark while data are sinked from Flink -- This is an automated message from the Apache Git Service. To

[PR] Core: Remove deprecated method from BaseMetadataTable [iceberg]

ajantha-bhat opened a new pull request, #9298: URL: https://github.com/apache/iceberg/pull/9298 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Flink: fix flaky test that might fail due to classloader check [iceberg]

ajantha-bhat commented on PR #9216: URL: https://github.com/apache/iceberg/pull/9216#issuecomment-1855809256 I think this is not fully fixed. With the latest code, it has failed again https://github.com/apache/iceberg/actions/runs/7208855885/job/19638664888?pr=9296 cc: @nas

Re: [PR] Flink: fix flaky test that might fail due to classloader check [iceberg]

ajantha-bhat commented on PR #9216: URL: https://github.com/apache/iceberg/pull/9216#issuecomment-1855811751 Aah, never mind. We need to backport to other flink version. I will do it now. The failure was in 1.16 -- This is an automated message from the Apache Git Service. To respo

Re: [PR] Flink: fix flaky test that might fail due to classloader check [iceberg]

ajantha-bhat commented on PR #9216: URL: https://github.com/apache/iceberg/pull/9216#issuecomment-1855820890 @stevenzwu: Looks like the code is already there in 1.16, but still it is failed. Please take a look. -- This is an automated message from the Apache Git Service. To respond to t

Re: [PR] Core: Remove deprecated classes related to rewrite data files [iceberg]

ajantha-bhat closed pull request #9296: Core: Remove deprecated classes related to rewrite data files URL: https://github.com/apache/iceberg/pull/9296 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Core: Remove deprecated classes related to rewrite data files [iceberg]

ajantha-bhat commented on PR #9296: URL: https://github.com/apache/iceberg/pull/9296#issuecomment-1855823294 Retriggering build due to flaky test from Flink https://github.com/apache/iceberg/actions/runs/7208855885/job/19638664888?pr=9296 -- This is an automated message from the Apache

Re: [I] hive4 MapReduce query execution engine support [iceberg]

pvary commented on issue #9297: URL: https://github.com/apache/iceberg/issues/9297#issuecomment-1855869513 @wfy-belief: AFAIK the MR engine is deprecated in Hive, but I would suggest to ask this question on the Hive dev/user mailing list, as the Hive team works on the Hive 4.0.0 integration

Re: [PR] Flink 1.17: Support specifying equality columns with write options [iceberg]

pvary commented on PR #8195: URL: https://github.com/apache/iceberg/pull/8195#issuecomment-1855873133 @manuzhang: The main point of Iceberg is that we have an engine independent table format. The columns are either primary keys, or not. If they are primary keys, they should be primary keys

Re: [PR] Core: Remove deprecated method from BaseMetadataTable [iceberg]

ajantha-bhat commented on code in PR #9298: URL: https://github.com/apache/iceberg/pull/9298#discussion_r1426742343 ## core/src/main/java/org/apache/iceberg/SerializableTable.java: ## @@ -105,6 +105,8 @@ private String metadataFileLocation(Table table) { if (table instanceo

Re: [PR] Modernize the readme's Status paragraph [iceberg]

szehon-ho commented on code in PR #9272: URL: https://github.com/apache/iceberg/pull/9272#discussion_r1426751402 ## README.md: ## @@ -31,15 +31,17 @@ Background and documentation is available at Iceberg is under active development at the Apache S

Re: [PR] Core: Look up targeted position deletes by path [iceberg]

szehon-ho commented on code in PR #9251: URL: https://github.com/apache/iceberg/pull/9251#discussion_r1426773779 ## core/src/main/java/org/apache/iceberg/DeleteFileIndex.java: ## @@ -582,93 +513,187 @@ private Iterable>> deleteManifestRea } } - // a group of indexed

[I] position delete in BaseEqualityDeltaWriter write function will lead to unstable result when equalityFieldColumns is not null and upsert is false [iceberg]

sunnyzhuzhu opened a new issue, #9299: URL: https://github.com/apache/iceberg/issues/9299 position delete in BaseEqualityDeltaWriter write function will lead to unstable result when equalityFieldColumns is not null and upsert is false. when equalityFieldColumns is not null and upsert is

Re: [PR] Core: Remove deprecated method from BaseMetadataTable [iceberg]

ajantha-bhat commented on PR #9298: URL: https://github.com/apache/iceberg/pull/9298#issuecomment-1855949807 Looks like some tests are directly casting metadata tables with `HasTableOperations`. So, some more work is needed for this PR. Let me work on it and ping when it is ready. --

[PR] fix when equalityFieldColumns is not null and upsert is false, position delete in write function will lead to unstable result if flink checkpoint interval is not same [iceberg]

sunnyzhuzhu opened a new pull request, #9300: URL: https://github.com/apache/iceberg/pull/9300 fix when equalityFieldColumns is not null and upsert is false, position delete in write function will lead to unstable result if flink checkpoint interval is not same -- This is an automated me

Re: [PR] fix when equalityFieldColumns is not null and upsert is false, position delete in write function will lead to unstable result if flink checkpoint interval is not same [iceberg]

sunnyzhuzhu commented on PR #9300: URL: https://github.com/apache/iceberg/pull/9300#issuecomment-1855962287 @openinx Can you help to review the code, thanks very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] Adding Snowflake's public documentation [iceberg-docs]

scottteal commented on code in PR #297: URL: https://github.com/apache/iceberg-docs/pull/297#discussion_r1426837375 ## docs/config.toml: ## @@ -78,11 +78,12 @@ home = [ "HTML", "RSS", "SearchIndex" ] { name = "StarRocks", identifier = "_starrocks", weight = 701, url = "htt

Re: [PR] fix when equalityFieldColumns is not null and upsert is false, position delete in write function will lead to unstable result if flink checkpoint interval is not same [iceberg]

pvary commented on code in PR #9300: URL: https://github.com/apache/iceberg/pull/9300#discussion_r1426840799 ## core/src/main/java/org/apache/iceberg/io/BaseTaskWriter.java: ## @@ -137,11 +137,7 @@ public void write(T row) throws IOException { StructLike copiedKey = Stru

Re: [I] Unable to merge CDC data into snapshot data. java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Long [iceberg]

harshith-bolar-rapido commented on issue #8333: URL: https://github.com/apache/iceberg/issues/8333#issuecomment-1856030578 Quick heads up, the merge into query works fine in version `1.0.0`, starts breaking from `1.1.0`. -- This is an automated message from the Apache Git Service. To resp

Re: [PR] BugFix: ORC reader is not closed when SortedMerge iterator is used for positional deletes [iceberg]

pvary commented on code in PR #9301: URL: https://github.com/apache/iceberg/pull/9301#discussion_r1426870221 ## orc/src/main/java/org/apache/iceberg/orc/OrcIterable.java: ## @@ -134,7 +134,13 @@ private static VectorizedRowBatchIterator newOrcIterator( try { return

Re: [PR] Core, API: Move SQLViewRepresentation to API [iceberg]

nastra commented on code in PR #9278: URL: https://github.com/apache/iceberg/pull/9278#discussion_r1426921944 ## .palantir/revapi.yml: ## @@ -877,6 +877,10 @@ acceptedBreaks: - code: "java.field.serialVersionUIDChanged" new: "field org.apache.iceberg.util.Serializabl

Re: [PR] Modernize the readme's Status paragraph [iceberg]

Fokko merged PR #9272: URL: https://github.com/apache/iceberg/pull/9272 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [I] The "Status" paragraph in the readme seems very outdated [iceberg]

Fokko closed issue #9127: The "Status" paragraph in the readme seems very outdated URL: https://github.com/apache/iceberg/issues/9127 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] Spark Streaming: Fix clobbering of files across streaming epochs [iceberg]

veeraswamy87 commented on code in PR #9255: URL: https://github.com/apache/iceberg/pull/9255#discussion_r1426924289 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java: ## @@ -673,11 +673,11 @@ public DataWriter createWriter(int partitionId, long t

Re: [PR] API, Core: Move SQLViewRepresentation to API [iceberg]

nastra commented on code in PR #9302: URL: https://github.com/apache/iceberg/pull/9302#discussion_r1426926992 ## core/src/main/java/org/apache/iceberg/view/BaseSQLViewRepresentation.java: ## @@ -0,0 +1,30 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] API, Core: Move SQLViewRepresentation to API [iceberg]

nastra commented on code in PR #9302: URL: https://github.com/apache/iceberg/pull/9302#discussion_r1426926992 ## core/src/main/java/org/apache/iceberg/view/BaseSQLViewRepresentation.java: ## @@ -0,0 +1,30 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Modernize the readme's Status paragraph [iceberg]

Fokko commented on code in PR #9272: URL: https://github.com/apache/iceberg/pull/9272#discussion_r1426931189 ## README.md: ## @@ -31,15 +31,17 @@ Background and documentation is available at Iceberg is under active development at the Apache Softw

Re: [PR] Core: Remove deprecated classes related to rewrite data files [iceberg]

nastra commented on PR #9296: URL: https://github.com/apache/iceberg/pull/9296#issuecomment-1856132673 thanks @ajantha-bhat -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Modernize the readme's Status paragraph [iceberg]

Fokko commented on PR #9272: URL: https://github.com/apache/iceberg/pull/9272#issuecomment-1856129863 Thanks for taking care of this @mt-ronkorving and thanks for the reviews @bitsondatadev, @nastra and @szehon-ho 🙌 -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Core: Remove deprecated classes related to rewrite data files [iceberg]

nastra merged PR #9296: URL: https://github.com/apache/iceberg/pull/9296 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] API, Core: Move SQLViewRepresentation to API [iceberg]

amogh-jahagirdar commented on code in PR #9302: URL: https://github.com/apache/iceberg/pull/9302#discussion_r1426944012 ## core/src/main/java/org/apache/iceberg/view/BaseSQLViewRepresentation.java: ## @@ -0,0 +1,30 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

nastra merged PR #8918: URL: https://github.com/apache/iceberg/pull/8918 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] BugFix: ORC reader is not closed when SortedMerge iterator is used for positional deletes [iceberg]

deniskuzZ commented on code in PR #9301: URL: https://github.com/apache/iceberg/pull/9301#discussion_r1426947971 ## orc/src/main/java/org/apache/iceberg/orc/OrcIterable.java: ## @@ -134,7 +134,13 @@ private static VectorizedRowBatchIterator newOrcIterator( try { re

Re: [PR] BugFix: ORC reader is not closed when SortedMerge iterator is used for positional deletes [iceberg]

deniskuzZ commented on code in PR #9301: URL: https://github.com/apache/iceberg/pull/9301#discussion_r1426947971 ## orc/src/main/java/org/apache/iceberg/orc/OrcIterable.java: ## @@ -134,7 +134,13 @@ private static VectorizedRowBatchIterator newOrcIterator( try { re

Re: [PR] BugFix: ORC reader is not closed when SortedMerge iterator is used for positional deletes [iceberg]

deniskuzZ commented on code in PR #9301: URL: https://github.com/apache/iceberg/pull/9301#discussion_r1426947971 ## orc/src/main/java/org/apache/iceberg/orc/OrcIterable.java: ## @@ -134,7 +134,13 @@ private static VectorizedRowBatchIterator newOrcIterator( try { re

Re: [PR] BugFix: ORC reader is not closed when SortedMerge iterator is used for positional deletes [iceberg]

deniskuzZ commented on code in PR #9301: URL: https://github.com/apache/iceberg/pull/9301#discussion_r1426947971 ## orc/src/main/java/org/apache/iceberg/orc/OrcIterable.java: ## @@ -134,7 +134,13 @@ private static VectorizedRowBatchIterator newOrcIterator( try { re

Re: [PR] API, Core: Move SQLViewRepresentation to API [iceberg]

nastra commented on PR #9302: URL: https://github.com/apache/iceberg/pull/9302#issuecomment-1856177487 thanks for the review @amogh-jahagirdar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] API, Core: Move SQLViewRepresentation to API [iceberg]

nastra merged PR #9302: URL: https://github.com/apache/iceberg/pull/9302 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

  1   2   >