Re: [PR] Core, Spark 3.5: Support file and partition delete granularity [iceberg]

aokolnychyi commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1436831824 ## core/src/main/java/org/apache/iceberg/deletes/DeleteGranularity.java: ## @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Core, Spark 3.5: Support file and partition delete granularity [iceberg]

aokolnychyi commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1436832637 ## core/src/main/java/org/apache/iceberg/deletes/DeleteGranularity.java: ## @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

[I] EOF: read 1 bytes when load manifest write by icelake [iceberg-python]

ZENOTME opened a new issue, #241: URL: https://github.com/apache/iceberg-python/issues/241 ### Question [Icelake](https://github.com/icelake-io/icelake) is a rust sdk and it use avro-rs. I find that it will report `EOF: read 1 bytes` (caused by: https://github.com/apache/iceberg-pyth

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

aokolnychyi commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1436842325 ## core/src/main/java/org/apache/iceberg/deletes/TargetedPositionDeleteWriter.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

aokolnychyi commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1436845094 ## core/src/main/java/org/apache/iceberg/TableProperties.java: ## @@ -334,6 +335,9 @@ private TableProperties() {} public static final String MAX_REF_AGE_MS = "h

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

aokolnychyi commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1436845094 ## core/src/main/java/org/apache/iceberg/TableProperties.java: ## @@ -334,6 +335,9 @@ private TableProperties() {} public static final String MAX_REF_AGE_MS = "h

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

aokolnychyi commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1436845094 ## core/src/main/java/org/apache/iceberg/TableProperties.java: ## @@ -334,6 +335,9 @@ private TableProperties() {} public static final String MAX_REF_AGE_MS = "h

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

zinking commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1436845981 ## core/src/main/java/org/apache/iceberg/deletes/TargetedPositionDeleteWriter.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

aokolnychyi commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1436846492 ## core/src/main/java/org/apache/iceberg/deletes/TargetedPositionDeleteWriter.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

aokolnychyi commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1436847989 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkWriteConf.java: ## @@ -708,4 +709,15 @@ private long sparkAdvisoryPartitionSize() { private doubl

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

jerqi commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1436854871 ## core/src/main/java/org/apache/iceberg/TableProperties.java: ## @@ -334,6 +335,9 @@ private TableProperties() {} public static final String MAX_REF_AGE_MS = "history

Re: [I] EOF: read 1 bytes when load manifest write by icelake [iceberg-python]

Fokko commented on issue #241: URL: https://github.com/apache/iceberg-python/issues/241#issuecomment-1870117506 @ZENOTME Thanks for raising this issue! Can you share an example Avro file that has the bug? -- This is an automated message from the Apache Git Service. To respond to the messa

[PR] Set `ghp_{pages,path}` properties [iceberg-rust]

Fokko opened a new pull request, #138: URL: https://github.com/apache/iceberg-rust/pull/138 In the ASF yaml. I don't think we should make it a protected branch because I think they overwrite it every time with a force push. -- This is an automated message from the Apache Git Service. To r

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

aokolnychyi commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1436886944 ## core/src/main/java/org/apache/iceberg/deletes/TargetedPositionDeleteWriter.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

aokolnychyi commented on code in PR #9384: URL: https://github.com/apache/iceberg/pull/9384#discussion_r1436887408 ## core/src/main/java/org/apache/iceberg/TableProperties.java: ## @@ -334,6 +335,9 @@ private TableProperties() {} public static final String MAX_REF_AGE_MS = "h

Re: [PR] Set `ghp_{pages,path}` properties [iceberg-rust]

Xuanwo commented on PR #138: URL: https://github.com/apache/iceberg-rust/pull/138#issuecomment-1870131218 The same setup works on [opendal](https://github.com/apache/incubator-opendal/blob/main/.asf.yaml), I'm not sure why it doesn't work there. -- This is an automated message from the A

Re: [PR] Core: Close the MetricsReporter when the Catalog is closed. [iceberg]

aokolnychyi commented on code in PR #9353: URL: https://github.com/apache/iceberg/pull/9353#discussion_r1436890848 ## aws/src/main/java/org/apache/iceberg/aws/dynamodb/DynamoDbCatalog.java: ## @@ -487,6 +486,7 @@ public Configuration getConf() { @Override public void clo

Re: [PR] Core: Remove deprecated method from BaseMetadataTable [iceberg]

ajantha-bhat commented on PR #9298: URL: https://github.com/apache/iceberg/pull/9298#issuecomment-1870198207 Just rebased to resolve conflict. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[I] Flink API rewriteDataFile How to set up scanning based on file size [iceberg]

GuoZhaoY opened a new issue, #9386: URL: https://github.com/apache/iceberg/issues/9386 ### Query engine flink code : Actions.forTable(executionEnvironment,table) .rewriteDataFiles() .maxParallelism(maxParallelism)

Re: [PR] Set `ghp_{pages,path}` properties [iceberg-rust]

Fokko commented on PR #138: URL: https://github.com/apache/iceberg-rust/pull/138#issuecomment-1870212450 @Xuanwo Probably Github Pages had been enabled through an infra ticket in the past. The `.asf.yaml` is not declarative, if you leave anything out it will not disable it, you need to do t

Re: [PR] Set `ghp_{pages,path}` properties [iceberg-rust]

Xuanwo commented on PR #138: URL: https://github.com/apache/iceberg-rust/pull/138#issuecomment-1870226415 > @Xuanwo Probably Github Pages had been enabled through an infra ticket in the past. The `.asf.yaml` is not declarative, if you leave anything out it will not disable it, you need to d

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

Fokko commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1436375930 ## core/src/main/java/org/apache/iceberg/avro/AvroIterable.java: ## @@ -78,7 +79,8 @@ public CloseableIterator iterator() { if (start != null) { if (reader in

Re: [I] EOF: read 1 bytes when load manifest write by icelake [iceberg-python]

ZENOTME commented on issue #241: URL: https://github.com/apache/iceberg-python/issues/241#issuecomment-1870247267 > @ZENOTME Thanks for raising this issue! Can you share an example Avro file that has the bug? Sure, Thanks for your help! [b404ad7a-4de1-409c-be71-8c799a88e98b-m

Re: [PR] Core: Close the MetricsReporter when the Catalog is closed. [iceberg]

huyuanfeng2018 commented on code in PR #9353: URL: https://github.com/apache/iceberg/pull/9353#discussion_r1437005760 ## aws/src/main/java/org/apache/iceberg/aws/dynamodb/DynamoDbCatalog.java: ## @@ -487,6 +486,7 @@ public Configuration getConf() { @Override public void

Re: [PR] Spark: support replace equality deletes to position deletes [iceberg]

chenjunjiedada commented on PR #2216: URL: https://github.com/apache/iceberg/pull/2216#issuecomment-1870393661 @chenwyi2 This was split into #2864 and #2364. Please see https://github.com/apache/iceberg/pull/2372 to check why this is closed. -- This is an automated message from the Apach

Re: [PR] Set `ghp_{pages,path}` properties [iceberg-rust]

amogh-jahagirdar merged PR #138: URL: https://github.com/apache/iceberg-rust/pull/138 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...

Re: [I] doc: rust.iceberg.apache.org is not resolved [iceberg-rust]

amogh-jahagirdar closed issue #137: doc: rust.iceberg.apache.org is not resolved URL: https://github.com/apache/iceberg-rust/issues/137 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] Core, Data, Spark 3.5: Support file and partition delete granularity [iceberg]

jerqi commented on PR #9384: URL: https://github.com/apache/iceberg/pull/9384#issuecomment-1870412679 One question: Iceberg have rewritePositionDeletesAction. Will this pr influence this action? -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [I] When will the 0.6.0 version be released? [iceberg-python]

Michal-Bobrowsky commented on issue #192: URL: https://github.com/apache/iceberg-python/issues/192#issuecomment-1870429884 still realistic? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Spark Streaming: Fix clobbering of files across streaming epochs [iceberg]

amogh-jahagirdar commented on code in PR #9255: URL: https://github.com/apache/iceberg/pull/9255#discussion_r1437173563 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java: ## @@ -673,11 +673,11 @@ public DataWriter createWriter(int partitionId, lo

Re: [I] doc: rust.iceberg.apache.org is not resolved [iceberg-rust]

amogh-jahagirdar commented on issue #137: URL: https://github.com/apache/iceberg-rust/issues/137#issuecomment-1870604564 Reopening, the domain still seems to be an issue, although maybe a different issue. after merging #138 I got the following email: ``` An error occurred while run

Re: [PR] Core: Iceberg streaming streaming-skip-overwrite-snapshots SparkMicroBatchStream only skips over one file per trigger [iceberg]

cccs-jc commented on code in PR #8980: URL: https://github.com/apache/iceberg/pull/8980#discussion_r1437294256 ## core/src/main/java/org/apache/iceberg/MicroBatches.java: ## @@ -92,7 +92,7 @@ private static List> indexManifests( for (ManifestFile manifest : manifestFiles

Re: [I] Spark: migrate hive table to iceberg table(using hive metastore) [iceberg]

github-actions[bot] commented on issue #7328: URL: https://github.com/apache/iceberg/issues/7328#issuecomment-1870705923 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Spark: migrate hive table to iceberg table(using hive metastore) [iceberg]

github-actions[bot] closed issue #7328: Spark: migrate hive table to iceberg table(using hive metastore) URL: https://github.com/apache/iceberg/issues/7328 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] Cannot write incompatible data [iceberg]

github-actions[bot] closed issue #6979: Cannot write incompatible data URL: https://github.com/apache/iceberg/issues/6979 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [I] Cannot write incompatible data [iceberg]

github-actions[bot] commented on issue #6979: URL: https://github.com/apache/iceberg/issues/6979#issuecomment-1870705955 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [PR] Core, Spark: Fix delete with filter on nested columns [iceberg]

bluzy commented on PR #7132: URL: https://github.com/apache/iceberg/pull/7132#issuecomment-1870762007 PTAL @rdblue @RussellSpitzer @aokolnychyi @szehon-ho -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] doc: rust.iceberg.apache.org is not resolved [iceberg-rust]

liurenjie1024 commented on issue #137: URL: https://github.com/apache/iceberg-rust/issues/137#issuecomment-1870791312 Should we involve apache infra team to help debugging? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[I] Can iceberg support truncating table? [iceberg]

coolderli opened a new issue, #9387: URL: https://github.com/apache/iceberg/issues/9387 ### Query engine truncate table ### Question Can iceberg support truncating a table? I have some users who want to clear all data in the iceberg table. If I use `delete from` to drop

Re: [I] Can iceberg support truncating table? [iceberg]

coolderli commented on issue #9387: URL: https://github.com/apache/iceberg/issues/9387#issuecomment-1870798132 By the way, I don't want to drop the table. Because this table is just for testing, I can reuse it later without recreating it. -- This is an automated message from the Apache Gi

Re: [PR] Spark 3.5: Parallelize reading files in add_files procedure [iceberg]

amogh-jahagirdar commented on code in PR #9274: URL: https://github.com/apache/iceberg/pull/9274#discussion_r1437431367 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java: ## @@ -374,14 +376,16 @@ private static Iterator buildManifest( * @param pa

Re: [PR] Spark 3.5: Parallelize reading files in add_files procedure [iceberg]

amogh-jahagirdar commented on code in PR #9274: URL: https://github.com/apache/iceberg/pull/9274#discussion_r1437431367 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java: ## @@ -374,14 +376,16 @@ private static Iterator buildManifest( * @param pa

Re: [PR] Flink: Create JUnit5 version of TestFlinkScan [iceberg]

cgpoh commented on PR #9185: URL: https://github.com/apache/iceberg/pull/9185#issuecomment-1870902422 > @cgpoh I took the liberty to address the remaining items so that we can get this in. @pvary could you do a final review on this one please? I think we would also want to apply this to Fli