Re: [PR] Core: FileSystemTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

nastra commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1575749329 ## core/src/test/java/org/apache/iceberg/hadoop/TestHadoopCommits.java: ## @@ -21,9 +21,12 @@ import static org.apache.iceberg.TableProperties.COMMIT_NUM_RETRIES; impo

Re: [I] does iceberg can run on k8s? [iceberg]

nastra closed issue #10196: does iceberg can run on k8s? URL: https://github.com/apache/iceberg/issues/10196 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

Re: [I] does iceberg can run on k8s? [iceberg]

nastra commented on issue #10196: URL: https://github.com/apache/iceberg/issues/10196#issuecomment-2071584139 > Can I put this data into our Company storage as well using rest api? I'm not exactly sure I understand. Are you trying to replicate the data that lands in S3/GCS/Minio? Can

Re: [PR] Build: Bump deptry from 0.14.0 to 0.16.1 [iceberg-python]

Fokko merged PR #649: URL: https://github.com/apache/iceberg-python/pull/649 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

[PR] Fix incorrect metrics calculation for iceberg table due to column name transformation with special characters. [iceberg]

lintingbin opened a new pull request, #10204: URL: https://github.com/apache/iceberg/pull/10204 Here are the main changes to fix [#2623](https://github.com/apache/iceberg/issues/10115) 1. change MetricsConfig.columnModes from map[colName, MetricsMode] to map[fieldId, MetricsMode] 2. a

Re: [PR] Core: FileSystemTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1575783911 ## core/src/test/java/org/apache/iceberg/hadoop/TestHadoopCommits.java: ## @@ -21,9 +21,12 @@ import static org.apache.iceberg.TableProperties.COMMIT_NUM_RETRIES; imp

Re: [PR] Add Pagination To List Apis [iceberg]

nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1575784851 ## core/src/test/java/org/apache/iceberg/rest/TestRESTViewCatalog.java: ## @@ -144,6 +157,65 @@ public void closeCatalog() throws Exception { } } + @Test + p

Re: [I] does iceberg can run on k8s? [iceberg]

JunseoChoJJ commented on issue #10196: URL: https://github.com/apache/iceberg/issues/10196#issuecomment-2071620951 @nastra I am using naver cloud storage. it is same like s3, gcs. I want to figure out besides using s3, gcs, or minio. Can I use other storage to configure with iceberg -- T

Re: [PR] Add Pagination To List Apis [iceberg]

nastra commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1575784851 ## core/src/test/java/org/apache/iceberg/rest/TestRESTViewCatalog.java: ## @@ -144,6 +157,65 @@ public void closeCatalog() throws Exception { } } + @Test + p

Re: [I] does iceberg can run on k8s? [iceberg]

nastra commented on issue #10196: URL: https://github.com/apache/iceberg/issues/10196#issuecomment-2071634712 If Naver Cloud Storage has an S3-compatible API, then you can use it with Iceberg by using `S3FileIO` in your Spark catalog config: `--conf spark.sql.catalog.my_catalog.io-impl=org.

Re: [I] does iceberg can run on k8s? [iceberg]

JunseoChoJJ commented on issue #10196: URL: https://github.com/apache/iceberg/issues/10196#issuecomment-2071683150 @nastra Thanks a lot -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[I] REST Catalog to support custom-catalog like HMS/Glue [iceberg]

osscm opened a new issue, #10205: URL: https://github.com/apache/iceberg/issues/10205 ### Query engine Trino ### Question We are thinking it would be good feature to support custom catalog name in REST Catalog. Before creating an issue, thought will discuss here, as w

Re: [I] Incorrect Metrics Calculation for Iceberg Table Due to Column Name Transformation with Special Characters [iceberg]

lintingbin commented on issue #10115: URL: https://github.com/apache/iceberg/issues/10115#issuecomment-2071811784 > I see, is it because of the dot character? I havent looked deeply at the problem but both solutions make sense, I think the first one may be preferable because then there is l

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

pvary commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1575983673 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java: ## @@ -84,7 +85,8 @@ public RowDataRewriter( format,

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

pvary commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1575982248 ## data/src/test/java/org/apache/iceberg/io/TestTaskEqualityDeltaWriter.java: ## @@ -409,6 +421,55 @@ public void testUpsertDataWithFullRowSchema() throws IOException

Re: [PR] Spec: Clarify missing fields when writing [iceberg]

Fokko commented on code in PR #8672: URL: https://github.com/apache/iceberg/pull/8672#discussion_r1575985072 ## format/spec.md: ## @@ -128,12 +128,12 @@ Tables do not require rename, except for tables that use atomic rename to implem Writer requirements -Some tables i

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

pvary commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1575985174 ## data/src/test/java/org/apache/iceberg/io/TestTaskEqualityDeltaWriter.java: ## @@ -409,6 +421,55 @@ public void testUpsertDataWithFullRowSchema() throws IOException

Re: [PR] Spec: Clarify missing fields when writing [iceberg]

Fokko commented on code in PR #8672: URL: https://github.com/apache/iceberg/pull/8672#discussion_r1575985773 ## format/spec.md: ## @@ -128,12 +128,12 @@ Tables do not require rename, except for tables that use atomic rename to implem Writer requirements -Some tables i

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

pvary commented on code in PR #10151: URL: https://github.com/apache/iceberg/pull/10151#discussion_r1576007535 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkFileIO.java: ## @@ -0,0 +1,183 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

nastra commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1576011200 ## core/src/test/java/org/apache/iceberg/TestDataTaskParser.java: ## @@ -0,0 +1,249 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

nastra commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1576012052 ## core/src/test/java/org/apache/iceberg/TestDataTaskParser.java: ## @@ -0,0 +1,249 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more con

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

pvary commented on code in PR #10151: URL: https://github.com/apache/iceberg/pull/10151#discussion_r1576012292 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkFileIO.java: ## @@ -0,0 +1,183 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Flink: FlinkFileIO implementation [iceberg]

pvary commented on code in PR #10151: URL: https://github.com/apache/iceberg/pull/10151#discussion_r1576011008 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkFileIO.java: ## @@ -0,0 +1,183 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

nastra commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1576014466 ## core/src/main/java/org/apache/iceberg/DataTaskParser.java: ## @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contribu

Re: [PR] Hive: turn off the stats gathering when iceberg.hive.keep.stats is false [iceberg]

pvary merged PR #10148: URL: https://github.com/apache/iceberg/pull/10148 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Hive: turn off the stats gathering when iceberg.hive.keep.stats is false [iceberg]

pvary commented on PR #10148: URL: https://github.com/apache/iceberg/pull/10148#issuecomment-2071944795 Thanks for @stargrey102 for the PR and @deniskuzZ for the help during the review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

nastra commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1576017301 ## core/src/main/java/org/apache/iceberg/FileScanTaskParser.java: ## @@ -40,16 +40,39 @@ public class FileScanTaskParser { private FileScanTaskParser() {} + /**

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

nastra commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1576019460 ## core/src/main/java/org/apache/iceberg/FileScanTaskParser.java: ## @@ -40,16 +40,39 @@ public class FileScanTaskParser { private FileScanTaskParser() {} + /**

Re: [PR] Flink: Move ParquetReader to LogicalTypeAnnotationVisitor [iceberg]

pvary commented on PR #9719: URL: https://github.com/apache/iceberg/pull/9719#issuecomment-2071950025 Thanks @Fokko for the PR! Please do not forget the PR for the other Flink versions as well. -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

nastra commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1576025475 ## core/src/test/java/org/apache/iceberg/TestFileScanTaskParser.java: ## @@ -33,23 +33,64 @@ public void testNullArguments() { .isInstanceOf(IllegalArgumentExce

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

nastra commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1576026814 ## core/src/test/java/org/apache/iceberg/TestFileScanTaskParser.java: ## @@ -33,23 +33,64 @@ public void testNullArguments() { .isInstanceOf(IllegalArgumentExce

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

nastra commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1576036541 ## core/src/main/java/org/apache/iceberg/ScanTaskParser.java: ## @@ -0,0 +1,105 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contrib

Re: [PR] AWS: Retain Glue Catalog table description after updating Iceberg table [iceberg]

aajisaka commented on PR #10199: URL: https://github.com/apache/iceberg/pull/10199#issuecomment-2072339019 Thank you for your suggestion @geruh I'll test this patch manually and update the class. -- This is an automated message from the Apache Git Service. To respond to the message

Re: [I] Implement the position delete writer [iceberg-rust]

s-akhtar-baig commented on issue #340: URL: https://github.com/apache/iceberg-rust/issues/340#issuecomment-2072353948 Hi @ZENOTME, can I take on either this task or https://github.com/apache/iceberg-rust/issues/341? -- This is an automated message from the Apache Git Service. To respond t

Re: [I] Implement the equality delta writer [iceberg-rust]

liurenjie1024 commented on issue #343: URL: https://github.com/apache/iceberg-rust/issues/343#issuecomment-2072357168 Hi, @ZENOTME Since we have finished the framework, I'm thinking about providing a high level api such as [append in python](https://github.com/apache/iceberg-python/blob/2ee

Re: [I] Implement the equality delta writer [iceberg-rust]

ZENOTME commented on issue #343: URL: https://github.com/apache/iceberg-rust/issues/343#issuecomment-2072374369 > Hi, @ZENOTME Since we have finished the framework, I'm thinking about providing a high level api such as [append in python](https://github.com/apache/iceberg-python/blob/2ee2d19

Re: [I] Discussion: Next steps / requirements to support `append` files [iceberg-rust]

liurenjie1024 commented on issue #329: URL: https://github.com/apache/iceberg-rust/issues/329#issuecomment-2072455753 I've compiled a [doc](https://docs.google.com/document/d/1YncDX-qQ1T9jBGQmJNtRcPU1trRi00cB8eykv5diKw4/edit?usp=sharing) for discussing roadmaps and features for iceberg-rust

Re: [PR] Implement manifest filtering in `TableScan` [iceberg-rust]

liurenjie1024 commented on code in PR #323: URL: https://github.com/apache/iceberg-rust/pull/323#discussion_r1576369286 ## crates/iceberg/src/scan.rs: ## @@ -186,6 +239,27 @@ impl TableScan { .boxed()) } +fn create_manifest_eval_factory( +//&self, +

Re: [I] Implement size-based task planning [iceberg-rust]

s-akhtar-baig commented on issue #344: URL: https://github.com/apache/iceberg-rust/issues/344#issuecomment-2072785869 I am looking into this issue. @liurenjie1024, can you please assign it to me? Thanks! -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

nastra commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1576411506 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -209,11 +262,63 @@ public boolean dropTable(TableIdentifier identifier, boolean purge) {

Re: [I] Support to append file on table [iceberg-rust]

ZENOTME commented on issue #345: URL: https://github.com/apache/iceberg-rust/issues/345#issuecomment-2072819533 I can send a PR for this later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [I] Implement the position delete writer [iceberg-rust]

ZENOTME commented on issue #340: URL: https://github.com/apache/iceberg-rust/issues/340#issuecomment-2072825313 > Hi @ZENOTME, can I take on either this task or #341? Sure! Thanks! Before we move to more writers, maybe we can complete #345 first so that we can add the e2e test for lat

[I] Tracking:: Writing iceberg tables [iceberg-rust]

ZENOTME opened a new issue, #346: URL: https://github.com/apache/iceberg-rust/issues/346 Inspired by https://github.com/apache/iceberg-rust/pull/275#issuecomment-2041304545, I created this issue to track all our write task. It's based on [doc](https://docs.google.com/document/d/1YncDX-qQ1T

Re: [I] Implement partition writer [iceberg-rust]

gupteaj commented on issue #342: URL: https://github.com/apache/iceberg-rust/issues/342#issuecomment-2072845233 @liurenjie1024 , I can look into one of this task , can you please assign me ? -- This is an automated message from the Apache Git Service. To respond to the message, please log

[PR] feat: add `RollingManifestWriter` [iceberg-python]

felixscherz opened a new pull request, #650: URL: https://github.com/apache/iceberg-python/pull/650 Hi, this is in regards to #596 and still WIP. The `RollingManifestWriter` implementation closely follows the java [implementation](https://github.com/apache/iceberg/blob/main/core/src/

Re: [PR] Flink: Fix bounded source state restore record duplication [iceberg]

stevenzwu commented on code in PR #10208: URL: https://github.com/apache/iceberg/pull/10208#discussion_r1576552272 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/source/IcebergSource.java: ## @@ -201,8 +201,11 @@ private SplitEnumerator createEnumer return ne

[I] Add support for BigLake Metastore [iceberg-python]

sashkaw opened a new issue, #651: URL: https://github.com/apache/iceberg-python/issues/651 ### Feature Request / Improvement Feature request to add support for loading and interacting with catalogs from BigLake Metastore. -- This is an automated message from the Apache Git Service.

Re: [PR] Flink: Prevent setting endTag/endSnapshotId for streaming source [iceberg]

stevenzwu commented on code in PR #10207: URL: https://github.com/apache/iceberg/pull/10207#discussion_r1576603768 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java: ## @@ -155,6 +155,13 @@ private void validate() { Preconditions.checkArg

Re: [PR] Flink: Fix bounded source state restore record duplication [iceberg]

pvary commented on code in PR #10208: URL: https://github.com/apache/iceberg/pull/10208#discussion_r1576638673 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java: ## @@ -40,24 +44,27 @@ import org.apache.iceberg.FileFormat; import

Re: [PR] Flink: Prevent setting endTag/endSnapshotId for streaming source [iceberg]

pvary commented on code in PR #10207: URL: https://github.com/apache/iceberg/pull/10207#discussion_r1576658083 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java: ## @@ -155,6 +155,13 @@ private void validate() { Preconditions.checkArgumen

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

stevenzwu commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1576699869 ## core/src/main/java/org/apache/iceberg/DataTaskParser.java: ## @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contr

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

stevenzwu commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1576706517 ## core/src/test/java/org/apache/iceberg/TestFileScanTaskParser.java: ## @@ -33,23 +33,64 @@ public void testNullArguments() { .isInstanceOf(IllegalArgumentE

Re: [PR] Improve CLI Text by Adding Verbose Text for Commands [iceberg-go]

zeroshade commented on PR #68: URL: https://github.com/apache/iceberg-go/pull/68#issuecomment-2073179846 @rtadepalli I like the idea in general, but could you address my question when you have a chance? -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] Add the build from source section [iceberg-go]

zeroshade commented on code in PR #70: URL: https://github.com/apache/iceberg-go/pull/70#discussion_r1576744685 ## README.md: ## @@ -21,6 +21,19 @@ `iceberg` is a Golang implementation of the [Iceberg table spec](https://iceberg.apache.org/spec/). +## Build From Source + +

Re: [PR] feat: glue table creation with some docs on testing [iceberg-go]

zeroshade commented on PR #59: URL: https://github.com/apache/iceberg-go/pull/59#issuecomment-2073185271 any updates @wolfeidau? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

[PR] [WIP]: Add `InclusiveMetricsEvaluator` [iceberg-rust]

sdd opened a new pull request, #347: URL: https://github.com/apache/iceberg-rust/pull/347 `InclusiveMetricsEvaluator` is used inside table scans to filter `DataFile` entries within a Manifest, rejecting any of them if their metrics indicate that they cannot contain any rows that match the p

Re: [PR] Add `ManifestEvaluator`, used to filter manifests in table scans [iceberg-rust]

Fokko commented on code in PR #322: URL: https://github.com/apache/iceberg-rust/pull/322#discussion_r1576798073 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -0,0 +1,466 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Add `ManifestEvaluator`, used to filter manifests in table scans [iceberg-rust]

Fokko commented on code in PR #322: URL: https://github.com/apache/iceberg-rust/pull/322#discussion_r1576800710 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -0,0 +1,459 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Add `ManifestEvaluator`, used to filter manifests in table scans [iceberg-rust]

Fokko commented on code in PR #322: URL: https://github.com/apache/iceberg-rust/pull/322#discussion_r1576805382 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -0,0 +1,466 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] feat: add `RollingManifestWriter` [iceberg-python]

geruh commented on PR #650: URL: https://github.com/apache/iceberg-python/pull/650#issuecomment-2073414087 I believe in the Java implementation we have a concept of a [PositionOutputStream](https://github.com/apache/iceberg/blob/866021d7d34f274349ce7de1f29d113395e7f28c/api/src/main/java/org/

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

stevenzwu commented on code in PR #9728: URL: https://github.com/apache/iceberg/pull/9728#discussion_r1576706517 ## core/src/test/java/org/apache/iceberg/TestFileScanTaskParser.java: ## @@ -33,23 +33,64 @@ public void testNullArguments() { .isInstanceOf(IllegalArgumentE

[PR] Build: Bump pyparsing from 3.1.1 to 3.1.2 [iceberg-python]

dependabot[bot] opened a new pull request, #652: URL: https://github.com/apache/iceberg-python/pull/652 Bumps [pyparsing](https://github.com/pyparsing/pyparsing) from 3.1.1 to 3.1.2. Changelog Sourced from https://github.com/pyparsing/pyparsing/blob/master/CHANGES";>pyparsing's ch

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

chenjunjiedada commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1576141345 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkWriteConf.java: ## @@ -167,6 +168,18 @@ public DistributionMode distributionMode() { retu

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

aokolnychyi commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1577028939 ## core/src/main/java/org/apache/iceberg/deletes/ContinuousFileScopedPositionDeleteWriter.java: ## @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

aokolnychyi commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1577029896 ## core/src/main/java/org/apache/iceberg/io/BaseTaskWriter.java: ## @@ -109,18 +112,30 @@ protected abstract class BaseEqualityDeltaWriter implements Closeable {

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

aokolnychyi commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1577030203 ## core/src/main/java/org/apache/iceberg/deletes/ContinuousFileScopedPositionDeleteWriter.java: ## @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

aokolnychyi commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1577029896 ## core/src/main/java/org/apache/iceberg/io/BaseTaskWriter.java: ## @@ -109,18 +112,30 @@ protected abstract class BaseEqualityDeltaWriter implements Closeable {

Re: [PR] Flink: Apply DeleteGranularity for writes [iceberg]

aokolnychyi commented on code in PR #10200: URL: https://github.com/apache/iceberg/pull/10200#discussion_r1577030981 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/sink/BaseDeltaTaskWriter.java: ## @@ -56,7 +58,8 @@ abstract class BaseDeltaTaskWriter extends BaseTa

Re: [I] NPE while closing OrcFileAppender [iceberg]

github-actions[bot] closed issue #2470: NPE while closing OrcFileAppender URL: https://github.com/apache/iceberg/issues/2470 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] NPE while processing data in flink job [iceberg]

github-actions[bot] commented on issue #2471: URL: https://github.com/apache/iceberg/issues/2471#issuecomment-2073690197 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] NPE while closing OrcFileAppender [iceberg]

github-actions[bot] commented on issue #2470: URL: https://github.com/apache/iceberg/issues/2470#issuecomment-2073690169 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] NPE while processing data in flink job [iceberg]

github-actions[bot] closed issue #2471: NPE while processing data in flink job URL: https://github.com/apache/iceberg/issues/2471 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Hive: Check the increased usage of HMSClients in TestHiveIcebergStorageHandlerWithEngine tests [iceberg]

github-actions[bot] commented on issue #2474: URL: https://github.com/apache/iceberg/issues/2474#issuecomment-2073690227 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Hive: Check the increased usage of HMSClients in TestHiveIcebergStorageHandlerWithEngine tests [iceberg]

github-actions[bot] closed issue #2474: Hive: Check the increased usage of HMSClients in TestHiveIcebergStorageHandlerWithEngine tests URL: https://github.com/apache/iceberg/issues/2474 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [I] Improve hidden partition support for Hive CREATE TABLE [iceberg]

github-actions[bot] commented on issue #2681: URL: https://github.com/apache/iceberg/issues/2681#issuecomment-2073690420 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Add Pagination To List Apis [iceberg]

rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1577061740 ## core/src/test/java/org/apache/iceberg/rest/TestRESTViewCatalog.java: ## @@ -144,6 +157,65 @@ public void closeCatalog() throws Exception { } } + @Test +

Re: [PR] Add `ManifestEvaluator`, used to filter manifests in table scans [iceberg-rust]

liurenjie1024 commented on PR #322: URL: https://github.com/apache/iceberg-rust/pull/322#issuecomment-2073844403 > There are intentionally a lot of `todo!`s in here. The aim is to get this PR merged so that more people can contribute implementations for the different visitor methods that ne

Re: [PR] Add `ManifestEvaluator`, used to filter manifests in table scans [iceberg-rust]

liurenjie1024 commented on code in PR #322: URL: https://github.com/apache/iceberg-rust/pull/322#discussion_r1577106941 ## crates/iceberg/src/expr/visitors/manifest_evaluator.rs: ## @@ -0,0 +1,466 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more cont

Re: [I] Implement size-based task planning [iceberg-rust]

liurenjie1024 commented on issue #344: URL: https://github.com/apache/iceberg-rust/issues/344#issuecomment-2073860929 > I am looking into this issue. @liurenjie1024, can you please assign it to me? Thanks! Done, thanks! -- This is an automated message from the Apache Git Service. T

Re: [PR] Add the build from source section [iceberg-go]

git-hulk commented on code in PR #70: URL: https://github.com/apache/iceberg-go/pull/70#discussion_r1577141430 ## README.md: ## @@ -21,6 +21,19 @@ `iceberg` is a Golang implementation of the [Iceberg table spec](https://iceberg.apache.org/spec/). +## Build From Source + +#

Re: [PR] Infra: Track subtasks from Iceberg improvement proposal [iceberg]

ajantha-bhat commented on PR #10183: URL: https://github.com/apache/iceberg/pull/10183#issuecomment-2073868413 ping @danielcweeks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] Infra: Track subtasks from Iceberg improvement proposal [iceberg]

danielcweeks commented on PR #10183: URL: https://github.com/apache/iceberg/pull/10183#issuecomment-2073903547 I'm not sure I follow the reasoning here. At the point your creating an issue, you shouldn't try to define subtasks as nothing has been decided. You can always go back and edit t

Re: [PR] MR: iceberg storage handler should set common projection pruning config [iceberg]

ludlows commented on PR #10188: URL: https://github.com/apache/iceberg/pull/10188#issuecomment-2073942683 @pvary could you review this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] Build: Bump pyparsing from 3.1.1 to 3.1.2 [iceberg-python]

Fokko merged PR #652: URL: https://github.com/apache/iceberg-python/pull/652 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] MR: iceberg storage handler should set common projection pruning config [iceberg]

pvary commented on code in PR #10188: URL: https://github.com/apache/iceberg/pull/10188#discussion_r1577249432 ## mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java: ## @@ -111,8 +111,15 @@ public void configureTableJobProperties(TableDesc tableDesc, Map

Re: [PR] MR: iceberg storage handler should set common projection pruning config [iceberg]

pvary commented on PR #10188: URL: https://github.com/apache/iceberg/pull/10188#issuecomment-2074024030 @ludlows: Could you please add a unit test as well? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Build: Bump sqlalchemy from 2.0.27 to 2.0.29 [iceberg-python]

Fokko merged PR #645: URL: https://github.com/apache/iceberg-python/pull/645 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] MR: iceberg storage handler should set common projection pruning config [iceberg]

ludlows commented on code in PR #10188: URL: https://github.com/apache/iceberg/pull/10188#discussion_r1577300578 ## mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java: ## @@ -111,8 +111,15 @@ public void configureTableJobProperties(TableDesc tableDesc, M

Re: [PR] feat: add `RollingManifestWriter` [iceberg-python]

felixscherz commented on PR #650: URL: https://github.com/apache/iceberg-python/pull/650#issuecomment-2074101193 Sounds good, I will have a look at the implementation and make a suggestion. Thank you! -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Build: Bump mkdocs-material from 9.5.10 to 9.5.18 [iceberg-python]

Fokko merged PR #646: URL: https://github.com/apache/iceberg-python/pull/646 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Implement partition writer [iceberg-rust]

ZENOTME commented on issue #342: URL: https://github.com/apache/iceberg-rust/issues/342#issuecomment-2074162966 There are rust implementation in icelake can as a reference: - fanout partition writer: https://github.com/icelake-io/icelake/blob/main/icelake/src/io_v2/functional_writer/part

[PR] Refactor GlueCatalog's _commit_table [iceberg-python]

HonahX opened a new pull request, #653: URL: https://github.com/apache/iceberg-python/pull/653 In https://github.com/apache/iceberg-python/pull/498, the `createTableTransaction` API requires the `_commit_table` to handle both table update and table creation. This PR refactors Glue's `_commi

Re: [I] Implement the equality delta writer [iceberg-rust]

QuakeWang commented on issue #343: URL: https://github.com/apache/iceberg-rust/issues/343#issuecomment-2074182349 Please assign it to me, I will do it later : ) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

nastra commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1577350170 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCommits.java: ## @@ -0,0 +1,437 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

nastra commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1577351385 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCommits.java: ## @@ -0,0 +1,437 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

nastra commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1577355581 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCommits.java: ## @@ -0,0 +1,437 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

nastra commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1577368517 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCommits.java: ## @@ -0,0 +1,437 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one