Re: [I] Automatically create the tables for the `SqlCatalog` [iceberg-python]

Fokko commented on issue #184: URL: https://github.com/apache/iceberg-python/issues/184#issuecomment-1842280596 Hey @cosmastech Thanks for chiming in here! The confusion here is between the Iceberg tables in general and the tables used by the SqlCatalog (which uses Alembic). We don't

Re: [I] Delete files not eventually removed if RewriteDataFile run right after delete (when using 'use-starting-sequence-number' default) [iceberg]

chenwyi2 commented on issue #4127: URL: https://github.com/apache/iceberg/issues/4127#issuecomment-1842175872 @szehon-ho Is there a plan to deal with equality delete? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] RemoveDanglingDeleteFiles [iceberg]

chenwyi2 commented on issue #6126: URL: https://github.com/apache/iceberg/issues/6126#issuecomment-1842153186 so how about equality delete file? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [I] Flink Rewrite Files Action OOM [iceberg]

bhupixb commented on issue #9193: URL: https://github.com/apache/iceberg/issues/9193#issuecomment-1842146242 Thank you, this helped us give some direction. We disabled the equality field column and upsert property. After that it is working correctly. Though our job does not upsert more th

Re: [I] How many concurrent operations can be supported at most when multiple Spark tasks write to iceberg same table? [iceberg]

AllenWee1106 commented on issue #9218: URL: https://github.com/apache/iceberg/issues/9218#issuecomment-1842133235 @Fokko However, the following anomalies are occurring frequently now： `Caused by: org.apache.iceberg.exceptions.CommitFailedException: Cannot commit datalake.slp_qual

Re: [I] Ignore downcasting of column types when "mergeSchema" is set. [iceberg]

yyy1000 commented on issue #4849: URL: https://github.com/apache/iceberg/issues/4849#issuecomment-1842133002 Since no one is working on this, I'd like a try. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] Style: Replace Arrays.asList with Collections.singletonList [iceberg]

pvary commented on PR #9213: URL: https://github.com/apache/iceberg/pull/9213#issuecomment-1842125593 The Flink flaky test is fixed by #9216. Thanks @stevenzwu! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] Flink: Adds support for 1.18 version [iceberg]

pvary commented on code in PR #9211: URL: https://github.com/apache/iceberg/pull/9211#discussion_r1416686437 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/TestHelpers.java: ## @@ -416,7 +416,7 @@ private static void assertAvroEquals( Collection expectedArr

Re: [I] Remove `RuntimeIOException` usage [iceberg]

ajantha-bhat commented on issue #9208: URL: https://github.com/apache/iceberg/issues/9208#issuecomment-1842119111 @danielcweeks, @Fokko, @nastra: I have revived the PRs and it is ready for review. https://github.com/apache/iceberg/pull/6887 https://github.com/apache/iceberg/pul

Re: [I] How many concurrent operations can be supported at most when multiple Spark tasks write to iceberg same table? [iceberg]

AllenWee1106 commented on issue #9218: URL: https://github.com/apache/iceberg/issues/9218#issuecomment-1842108506 @Fokko I am using the JDCB catalog . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[I] Timestamp data losing precision when Flink writing to Iceberg [iceberg]

yeunghl-shoalter opened a new issue, #9229: URL: https://github.com/apache/iceberg/issues/9229 ### Apache Iceberg version 1.4.2 (latest release) ### Query engine Flink ### Please describe the bug 🐞 When writing data (timestamp with timezone precision 3) into

Re: [I] Parquet file overwritten by spark streaming job in subsequent execution with same spark streaming checkpoint location [iceberg]

Abhishek-Dutta commented on issue #9172: URL: https://github.com/apache/iceberg/issues/9172#issuecomment-1842071333 @amogh-jahagirdar Yes, these two are related. I am a colleague @amitmittal5, and after raising this issue later we found similar issue was opened end of October. The differenc

Re: [PR] Use Pydantic's `model_copy` for model modification when updating table metadata [iceberg-python]

HonahX commented on code in PR #182: URL: https://github.com/apache/iceberg-python/pull/182#discussion_r1416622061 ## pyiceberg/table/__init__.py: ## @@ -533,6 +535,8 @@ def update_table_metadata(base_metadata: TableMetadata, updates: Tuple[TableUpda for update in updates:

[I] Flink SQL encountered an exception while reading the iceberg table [iceberg]

AllenWee1106 opened a new issue, #9228: URL: https://github.com/apache/iceberg/issues/9228 ### Query engine flink 1.16.2 iceberg 1.3.1 ### Question `flink 1.16.2 iceberg 1.3.1` `tabenv.executeSql("create catalog jdbc with " + "('ty

Re: [PR] Style: Replace Arrays.asList with Collections.singletonList [iceberg]

yyy1000 commented on PR #9213: URL: https://github.com/apache/iceberg/pull/9213#issuecomment-1841972935 Thank you for your review, @aokolnychyi! I like the project and I can definitely contribute more. :) -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] Flink: Create JUnit5 version of TestFlinkScan [iceberg]

cgpoh commented on code in PR #9185: URL: https://github.com/apache/iceberg/pull/9185#discussion_r1416542177 ## data/src/test/java/org/apache/iceberg/data/GenAppenderHelper.java: ## @@ -0,0 +1,144 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Flink: Create JUnit5 version of TestFlinkScan [iceberg]

cgpoh commented on code in PR #9185: URL: https://github.com/apache/iceberg/pull/9185#discussion_r1416542497 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestFlinkScan.java: ## @@ -49,37 +51,28 @@ import org.apache.iceberg.types.Types; import org.apache.i

Re: [PR] Flink: Create JUnit5 version of TestFlinkScan [iceberg]

cgpoh commented on code in PR #9185: URL: https://github.com/apache/iceberg/pull/9185#discussion_r1416540648 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/TestFlinkScan.java: ## @@ -49,37 +51,28 @@ import org.apache.iceberg.types.Types; import org.apache.i

Re: [PR] Flink: Create JUnit5 version of TestFlinkScan [iceberg]

cgpoh commented on code in PR #9185: URL: https://github.com/apache/iceberg/pull/9185#discussion_r1416540362 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/HadoopCatResource.java: ## @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Core: Add comment property to ViewProperties [iceberg]

amogh-jahagirdar commented on PR #9181: URL: https://github.com/apache/iceberg/pull/9181#issuecomment-1841927189 Thanks for the review @nastra , merging -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Core: Add comment property to ViewProperties [iceberg]

amogh-jahagirdar merged PR #9181: URL: https://github.com/apache/iceberg/pull/9181 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Fixed certain operations failing to add new data files during retries [iceberg]

rdblue commented on code in PR #9222: URL: https://github.com/apache/iceberg/pull/9222#discussion_r1416470538 ## core/src/main/java/org/apache/iceberg/FastAppend.java: ## @@ -188,7 +188,7 @@ protected void cleanUncommitted(Set committed) { } } - this.newM

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1416462272 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,202 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1416461866 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +168,193 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [I] iceberg table query bug [iceberg]

github-actions[bot] commented on issue #7640: URL: https://github.com/apache/iceberg/issues/7640#issuecomment-1841859529 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] iceberg table query bug [iceberg]

github-actions[bot] closed issue #7640: iceberg table query bug URL: https://github.com/apache/iceberg/issues/7640 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1416456616 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +168,193 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1416455358 ## pyiceberg/table/snapshots.py: ## @@ -116,3 +144,199 @@ class MetadataLogEntry(IcebergBaseModel): class SnapshotLogEntry(IcebergBaseModel): snapshot_id: int =

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1416449698 ## pyiceberg/table/snapshots.py: ## @@ -65,6 +90,25 @@ def __init__(self, operation: Operation, **data: Any) -> None: super().__init__(operation=operation,

Re: [PR] Core: Iceberg streaming streaming-skip-overwrite-snapshots SparkMicroBatchStream only skips over one file per trigger [iceberg]

singhpk234 commented on PR #8980: URL: https://github.com/apache/iceberg/pull/8980#issuecomment-1841776563 @cccs-jc i mean let's have changes for 3.5 with it's test only in 3.5 and we can backport the change with it's test in lower spark version like 3.4 and 3.3, 3.4 test failures are expec

Re: [PR] Core: Add PartitionMap [iceberg]

aokolnychyi commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1416381792 ## core/src/main/java/org/apache/iceberg/util/PartitionMap.java: ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Core: Add PartitionMap [iceberg]

RussellSpitzer commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1416379643 ## core/src/test/java/org/apache/iceberg/util/TestPartitionMap.java: ## @@ -0,0 +1,222 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Core: Add PartitionMap [iceberg]

RussellSpitzer commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1416376393 ## core/src/test/java/org/apache/iceberg/util/TestPartitionMap.java: ## @@ -0,0 +1,222 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Core: Add PartitionMap [iceberg]

RussellSpitzer commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1416374500 ## core/src/test/java/org/apache/iceberg/util/TestPartitionMap.java: ## @@ -0,0 +1,222 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Core: Add PartitionMap [iceberg]

RussellSpitzer commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1416374206 ## core/src/test/java/org/apache/iceberg/util/TestPartitionMap.java: ## @@ -0,0 +1,222 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Core: Add PartitionMap [iceberg]

RussellSpitzer commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1416356324 ## core/src/main/java/org/apache/iceberg/util/PartitionMap.java: ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [I] Automatically create the tables for the `SqlCatalog` [iceberg-python]

cosmastech commented on issue #184: URL: https://github.com/apache/iceberg-python/issues/184#issuecomment-1841735588 Hi @Fokko I was interested in contributing to this, but I'm a little bit confused. I see that `Catalog` class has this included in the docstring for `load_table

Re: [PR] Core: Add PartitionMap [iceberg]

aokolnychyi commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1416348617 ## core/src/main/java/org/apache/iceberg/util/PartitionMap.java: ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Core: Add PartitionMap [iceberg]

RussellSpitzer commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1416347669 ## core/src/main/java/org/apache/iceberg/util/PartitionMap.java: ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] Core: Add PartitionMap [iceberg]

aokolnychyi commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1416344322 ## core/src/main/java/org/apache/iceberg/util/PartitionMap.java: ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Core: Add PartitionMap [iceberg]

aokolnychyi commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1416341234 ## core/src/main/java/org/apache/iceberg/util/PartitionMap.java: ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Core: Add PartitionMap [iceberg]

aokolnychyi commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1416341234 ## core/src/main/java/org/apache/iceberg/util/PartitionMap.java: ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Core: Add PartitionMap [iceberg]

aokolnychyi commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1416341234 ## core/src/main/java/org/apache/iceberg/util/PartitionMap.java: ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Core: Add PartitionMap [iceberg]

aokolnychyi commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1416338150 ## core/src/main/java/org/apache/iceberg/util/PartitionMap.java: ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Core: Add PartitionMap [iceberg]

RussellSpitzer commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1416328166 ## core/src/main/java/org/apache/iceberg/util/PartitionMap.java: ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] Style: Replace Arrays.asList with Collections.singletonList [iceberg]

aokolnychyi commented on PR #9213: URL: https://github.com/apache/iceberg/pull/9213#issuecomment-1841688683 Thank you, @yyy1000! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Core: Add PartitionMap [iceberg]

RussellSpitzer commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1416327633 ## core/src/main/java/org/apache/iceberg/util/PartitionMap.java: ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1416325238 ## pyiceberg/table/snapshots.py: ## @@ -19,15 +19,40 @@ Any, Dict, List, +Mapping, Optional, ) from pydantic import Field, PrivateAttr, mo

Re: [PR] Core: Add PartitionMap [iceberg]

aokolnychyi commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1416323437 ## core/src/main/java/org/apache/iceberg/util/PartitionMap.java: ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Core: Add PartitionMap [iceberg]

aokolnychyi commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1416323437 ## core/src/main/java/org/apache/iceberg/util/PartitionMap.java: ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1416322935 ## pyiceberg/table/snapshots.py: ## @@ -19,15 +19,40 @@ Any, Dict, List, +Mapping, Optional, ) from pydantic import Field, PrivateAttr, mo

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1416322319 ## pyiceberg/table/snapshots.py: ## @@ -19,15 +19,40 @@ Any, Dict, List, +Mapping, Optional, ) from pydantic import Field, PrivateAttr, mo

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1416320627 ## pyiceberg/table/snapshots.py: ## @@ -19,15 +19,40 @@ Any, Dict, List, +Mapping, Optional, ) from pydantic import Field, PrivateAttr, mo

Re: [PR] Add Snapshot logic and Summary generation [iceberg-python]

rdblue commented on code in PR #61: URL: https://github.com/apache/iceberg-python/pull/61#discussion_r1416321121 ## pyiceberg/table/snapshots.py: ## @@ -19,15 +19,40 @@ Any, Dict, List, +Mapping, Optional, ) from pydantic import Field, PrivateAttr, mo

Re: [PR] Core: Add PartitionMap [iceberg]

aokolnychyi commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1416320230 ## core/src/main/java/org/apache/iceberg/util/PartitionMap.java: ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [PR] Core: Add PartitionMap [iceberg]

RussellSpitzer commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1416316893 ## core/src/main/java/org/apache/iceberg/util/PartitionMap.java: ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [I] Replace `Arrays.asList` with `Collections.singletonList` [iceberg]

aokolnychyi closed issue #9207: Replace `Arrays.asList` with `Collections.singletonList` URL: https://github.com/apache/iceberg/issues/9207 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Style: Replace Arrays.asList with Collections.singletonList [iceberg]

aokolnychyi merged PR #9213: URL: https://github.com/apache/iceberg/pull/9213 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] API, Core, Spark 3.5: Parallelize reading of deletes and cache them on executors [iceberg]

aokolnychyi commented on PR #8755: URL: https://github.com/apache/iceberg/pull/8755#issuecomment-1841668881 The Flink test failure does not seem related. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Style: Replace Arrays.asList with Collections.singletonList [iceberg]

aokolnychyi commented on PR #9213: URL: https://github.com/apache/iceberg/pull/9213#issuecomment-1841667707 The Flink test failure does not seem related. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] iceberg-parquet: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

GianlucaPrincipini commented on PR #9161: URL: https://github.com/apache/iceberg/pull/9161#issuecomment-1841655753 @nastra, sounds ok to me. I will work on migrating some other class in the next few days -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [PR] Core: Add PartitionMap [iceberg]

RussellSpitzer commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1416300233 ## core/src/main/java/org/apache/iceberg/util/PartitionMap.java: ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] Flink: Adds support for 1.18 version [iceberg]

rodmeneses commented on code in PR #9211: URL: https://github.com/apache/iceberg/pull/9211#discussion_r1416299412 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/TestHelpers.java: ## @@ -416,7 +416,7 @@ private static void assertAvroEquals( Collection expect

Re: [PR] Flink: Adds support for 1.18 version [iceberg]

stevenzwu commented on code in PR #9211: URL: https://github.com/apache/iceberg/pull/9211#discussion_r1416291551 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/TestHelpers.java: ## @@ -416,7 +416,7 @@ private static void assertAvroEquals( Collection expecte

Re: [PR] Core: Fix logic in BaseTransaction for determining set of committed files when there are no new snapshots [iceberg]

rdblue merged PR #9221: URL: https://github.com/apache/iceberg/pull/9221 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [I] Uploading Data to Iceberg [iceberg-python]

Fokko commented on issue #181: URL: https://github.com/apache/iceberg-python/issues/181#issuecomment-1841522572 Thanks for reaching out @Atiqul-Islam, this functionality is in-flight in https://github.com/apache/iceberg-python/pull/41 -- This is an automated message from the Apache Git Se

Re: [PR] Core: Expired Snapshot files in a transaction should be deleted (cherry-pick to 1.4.x) [iceberg]

bartash commented on PR #9223: URL: https://github.com/apache/iceberg/pull/9223#issuecomment-1841360572 Thanks @amogh-jahagirdar sounds good -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] Core: Expired Snapshot files in a transaction should be deleted (cherry-pick to 1.4.x) [iceberg]

amogh-jahagirdar commented on PR #9223: URL: https://github.com/apache/iceberg/pull/9223#issuecomment-1841356581 Thanks @bartash . I think let's get https://github.com/apache/iceberg/pull/9221 in first and then we can rebase this change? The change for returning a new empty set was not just

Re: [PR] Fixed certain operations failing to add new data files during retries [iceberg]

jasonf20 closed pull request #9222: Fixed certain operations failing to add new data files during retries URL: https://github.com/apache/iceberg/pull/9222 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Fixed certain operations failing to add new data files during retries [iceberg]

jasonf20 commented on PR #9222: URL: https://github.com/apache/iceberg/pull/9222#issuecomment-1841354140 I was not able to reproduce this issue with internal retries so I'm closing this and converting it to a bug that isn't as critical: https://github.com/apache/iceberg/issues/9227 --

[I] Manually retrying commits can result in partially applied changes [iceberg]

jasonf20 opened a new issue, #9227: URL: https://github.com/apache/iceberg/issues/9227 ### Apache Iceberg version 1.4.2 (latest release) ### Query engine Athena ### Please describe the bug 🐞 Since the following PR: https://github.com/apache/iceberg/pull/6335

Re: [PR] Core: Fix logic in BaseTransaction for determining set of committed files when there are no new snapshots [iceberg]

rdblue commented on code in PR #9221: URL: https://github.com/apache/iceberg/pull/9221#discussion_r1416089227 ## core/src/main/java/org/apache/iceberg/BaseTransaction.java: ## @@ -446,16 +447,18 @@ private void commitSimpleTransaction() { } Set committedFiles = c

Re: [PR] API, Core, Spark 3.5: Parallelize reading of deletes and cache them on executors [iceberg]

aokolnychyi commented on code in PR #8755: URL: https://github.com/apache/iceberg/pull/8755#discussion_r1416089345 ## core/src/main/java/org/apache/iceberg/deletes/BitmapPositionDeleteIndex.java: ## @@ -27,6 +27,15 @@ class BitmapPositionDeleteIndex implements PositionDeleteInd

Re: [PR] API, Core, Spark 3.5: Parallelize reading of deletes and cache them on executors [iceberg]

aokolnychyi commented on code in PR #8755: URL: https://github.com/apache/iceberg/pull/8755#discussion_r1416085857 ## core/src/main/java/org/apache/iceberg/SystemConfigs.java: ## @@ -43,14 +43,14 @@ private SystemConfigs() {} Integer::parseUnsignedInt); /** - *

Re: [PR] Core: Handle IAE in default error handler [iceberg]

rdblue merged PR #9225: URL: https://github.com/apache/iceberg/pull/9225 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Api: Track partition statistics via TableMetadata [iceberg]

aokolnychyi commented on PR #8502: URL: https://github.com/apache/iceberg/pull/8502#issuecomment-1841336236 I started looking on Friday but got distracted. I will try to finish by end of Wed. -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] API, Core, Spark 3.5: Parallelize reading of deletes and cache them on executors [iceberg]

aokolnychyi commented on code in PR #8755: URL: https://github.com/apache/iceberg/pull/8755#discussion_r1416084451 ## core/src/main/java/org/apache/iceberg/deletes/EmptyPositionDeleteIndex.java: ## @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Flink: Fix IcebergSource tableloader lifecycle management in batch mode [iceberg]

mas-chen commented on code in PR #9173: URL: https://github.com/apache/iceberg/pull/9173#discussion_r1416071311 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/IcebergSource.java: ## @@ -105,12 +107,12 @@ public class IcebergSource implements Sourcehttps://git

Re: [PR] Flink: Fix IcebergSource tableloader lifecycle management in batch mode [iceberg]

mas-chen commented on code in PR #9173: URL: https://github.com/apache/iceberg/pull/9173#discussion_r1416071311 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/IcebergSource.java: ## @@ -105,12 +107,12 @@ public class IcebergSource implements Sourcehttps://git

Re: [PR] Flink: Adds support for 1.18 version [iceberg]

rodmeneses commented on code in PR #9211: URL: https://github.com/apache/iceberg/pull/9211#discussion_r1416064067 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/TestHelpers.java: ## @@ -416,7 +416,7 @@ private static void assertAvroEquals( Collection expect

Re: [PR] Build: Bump actions/setup-java from 3 to 4 [iceberg]

nastra merged PR #9200: URL: https://github.com/apache/iceberg/pull/9200 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Flink: Adds support for 1.18 version [iceberg]

stevenzwu commented on code in PR #9211: URL: https://github.com/apache/iceberg/pull/9211#discussion_r1416061615 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/TestHelpers.java: ## @@ -416,7 +416,7 @@ private static void assertAvroEquals( Collection expecte

Re: [PR] Flink: backport PR #9216 for disabling classloader check [iceberg]

nastra merged PR #9226: URL: https://github.com/apache/iceberg/pull/9226 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Fixed certain operations failing to add new data files during retries [iceberg]

jasonf20 commented on PR #9222: URL: https://github.com/apache/iceberg/pull/9222#issuecomment-1841301960 Now that I think about it, it's possible that `cleanUncomittedFiles` is only called after all the internal retries fail, so doing manual retries will not work but automatic ones would.

Re: [PR] Fixed certain operations failing to add new data files during retries [iceberg]

jasonf20 commented on PR #9222: URL: https://github.com/apache/iceberg/pull/9222#issuecomment-1841290690 Hi @nastra I’m not very familiar with this codebase and adding tests that will fail at the right time would take me a long time. Unfortunately, I can’t do that right now.

Re: [PR] Build: Bump datamodel-code-generator from 0.24.2 to 0.25.0 [iceberg]

Fokko commented on PR #9189: URL: https://github.com/apache/iceberg/pull/9189#issuecomment-1841281189 Thanks @nastra 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] Build: Bump datamodel-code-generator from 0.24.2 to 0.25.0 [iceberg]

Fokko merged PR #9189: URL: https://github.com/apache/iceberg/pull/9189 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Fixed certain operations failing to add new data files during retries [iceberg]

nastra commented on PR #9222: URL: https://github.com/apache/iceberg/pull/9222#issuecomment-1841277926 @jasonf20 can you please add a test that reproduces the issue you're seeing? This makes it easier to reason about the problem -- This is an automated message from the Apache Git Service.

Re: [PR] Add SQLite support [iceberg-python]

Fokko commented on PR #178: URL: https://github.com/apache/iceberg-python/pull/178#issuecomment-1841269395 Thanks for the suggestions @bitsondatadev 🙌 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Build: Bump aiohttp from 3.8.6 to 3.9.0 [iceberg-python]

Fokko commented on PR #168: URL: https://github.com/apache/iceberg-python/pull/168#issuecomment-1841266994 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Core: Add PartitionMap [iceberg]

aokolnychyi commented on code in PR #9194: URL: https://github.com/apache/iceberg/pull/9194#discussion_r1416017260 ## core/src/main/java/org/apache/iceberg/util/PartitionMap.java: ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

[PR] Core: Handle IAE in default error handler [iceberg]

nastra opened a new pull request, #9225: URL: https://github.com/apache/iceberg/pull/9225 This came up in https://github.com/apache/iceberg/pull/7913#discussion_r1414686074 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Spark: Bump Spark minor versions for 3.3 and 3.4 [iceberg]

ajantha-bhat commented on PR #9187: URL: https://github.com/apache/iceberg/pull/9187#issuecomment-1841230595 ping -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

Re: [PR] Flink: Fix IcebergSource tableloader lifecycle management in batch mode [iceberg]

stevenzwu commented on code in PR #9173: URL: https://github.com/apache/iceberg/pull/9173#discussion_r1415990252 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/IcebergSource.java: ## @@ -105,12 +107,12 @@ public class IcebergSource implements Source

Re: [PR] Flink: Document watermark generation feature [iceberg]

stevenzwu commented on PR #9179: URL: https://github.com/apache/iceberg/pull/9179#issuecomment-1841204296 thanks @pvary for the documentation. thanks @mas-chen for the review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Flink: Document watermark generation feature [iceberg]

pvary commented on PR #9179: URL: https://github.com/apache/iceberg/pull/9179#issuecomment-1841209354 Thanks for the review and the merge @stevenzwu and @mas-chen ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] Flink: Document watermark generation feature [iceberg]

stevenzwu merged PR #9179: URL: https://github.com/apache/iceberg/pull/9179 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] Core: Expired Snapshot files in a transaction should be deleted. [iceberg]

bartash commented on PR #9183: URL: https://github.com/apache/iceberg/pull/9183#issuecomment-1841164689 > Once this gets in, can you create a PR against the `1.4.x` branch? Clean cherry-pick to 1.4.x is in #9223 -- This is an automated message from the Apache Git Service. To respond

Re: [I] Duplicate file name in Iceberg's metadata [iceberg]

amogh-jahagirdar commented on issue #8953: URL: https://github.com/apache/iceberg/issues/8953#issuecomment-1841135980 Thanks for the details, yeah I agree a UUID would of course essentially guarantee uniqueness, I'm just not sure of all the implications of changing the output paths. There

Re: [I] Parquet file overwritten by spark streaming job in subsequent execution with same spark streaming checkpoint location [iceberg]

amogh-jahagirdar commented on issue #9172: URL: https://github.com/apache/iceberg/issues/9172#issuecomment-1841116845 I think https://github.com/apache/iceberg/issues/8953 is related. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

  1   2   >