Re: [PR] [WIP] Sync manifest entry schema [iceberg-rust]

2025-04-29 Thread via GitHub
dentiny commented on PR #1276: URL: https://github.com/apache/iceberg-rust/pull/1276#issuecomment-2840997182 Sorry there're some test compilation failures, let me fix them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] feat: Add deletion vector related fields in spec types [iceberg-rust]

2025-04-29 Thread via GitHub
Xuanwo commented on PR #1276: URL: https://github.com/apache/iceberg-rust/pull/1276#issuecomment-2840992046 Hi @dentiny, I have updated the PR title to better reflect the changes. I have to say that I completely agree with your "upstream first" philosophy, and I appreciate any kind of help.

Re: [PR] Core, Data: File Format API interfaces [iceberg]

2025-04-29 Thread via GitHub
liurenjie1024 commented on code in PR #12774: URL: https://github.com/apache/iceberg/pull/12774#discussion_r2067958054 ## data/src/main/java/org/apache/iceberg/data/PositionDeleteWriterBuilder.java: ## @@ -0,0 +1,49 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

[PR] [WIP] Sync manifest entry schema [iceberg-rust]

2025-04-29 Thread via GitHub
dentiny opened a new pull request, #1276: URL: https://github.com/apache/iceberg-rust/pull/1276 ## Which issue does this PR close? This PR sync up with upstream iceberg spec on deletion vector related fields. Reference PR: https://github.com/apache/iceberg/pull/11240 Reference fi

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067934911 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreClientImpl.java: ## @@ -0,0 +1,594 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067931567 ## bigquery/src/test/java/org/apache/iceberg/gcp/bigquery/BigQueryTableOperationsTest.java: ## @@ -0,0 +1,274 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067930498 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java: ## @@ -0,0 +1,371 @@ +/* + * Licensed to the Apache Software Foundation (

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067928742 ## bigquery/src/test/java/org/apache/iceberg/gcp/bigquery/TestBigQueryTableOperations.java: ## @@ -0,0 +1,291 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [I] [bug] REST catalog `namespace_exists` fails with 400 Bad Request [iceberg-rust]

2025-04-29 Thread via GitHub
liurenjie1024 commented on issue #1271: URL: https://github.com/apache/iceberg-rust/issues/1271#issuecomment-2840904533 > cc [@liurenjie1024](https://github.com/liurenjie1024), [@Fokko](https://github.com/Fokko) Do you think it's worth making this compatible here? Should we fallback to `get

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
nastra commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067914332 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreClientImpl.java: ## @@ -0,0 +1,594 @@ +/* + * Licensed to the Apache Software Foundation (AS

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
nastra commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067912871 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreClientImpl.java: ## @@ -0,0 +1,594 @@ +/* + * Licensed to the Apache Software Foundation (AS

Re: [PR] refactor!: Change FileIO::remove_all to remove_dir_all [iceberg-rust]

2025-04-29 Thread via GitHub
liurenjie1024 commented on code in PR #1275: URL: https://github.com/apache/iceberg-rust/pull/1275#discussion_r2067912038 ## crates/iceberg/src/io/file_io.rs: ## @@ -95,9 +95,14 @@ impl FileIO { /// # Arguments /// /// * path: It should be *absolute* path starting

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
nastra commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067912014 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java: ## @@ -0,0 +1,371 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
nastra commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067907024 ## bigquery/src/test/java/org/apache/iceberg/gcp/bigquery/TestBigQueryTableOperations.java: ## @@ -0,0 +1,291 @@ +/* + * Licensed to the Apache Software Foundation (AS

[PR] Build, Core: Let RevAPI compare against 1.9.0 / Fix API breakage around StorageCredential [iceberg]

2025-04-29 Thread via GitHub
nastra opened a new pull request, #12930: URL: https://github.com/apache/iceberg/pull/12930 https://github.com/apache/iceberg/pull/12882 went in right in between voting on 1.9.0 happened, which means that the API breakage wasn't detected as a breakage when it was merged. After upgrading Rev

Re: [PR] Build, Core: Let RevAPI compare against 1.9.0 / Fix API breakage around StorageCredential [iceberg]

2025-04-29 Thread via GitHub
nastra closed pull request #12930: Build, Core: Let RevAPI compare against 1.9.0 / Fix API breakage around StorageCredential URL: https://github.com/apache/iceberg/pull/12930 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

[I] Support CRETA TABLE LIKE with Spark [iceberg]

2025-04-29 Thread via GitHub
manuzhang opened a new issue, #12936: URL: https://github.com/apache/iceberg/issues/12936 ### Feature Request / Improvement It looks `CREATE TABLE ... LIKE ... USING iceberg` isn't supported with Spark. ```sql CREATE TABLE default.test (id BIGINT, data STRING) USING iceberg;

Re: [PR] Build: Bump mypy-boto3-glue from 1.37.31 to 1.38.0 [iceberg-python]

2025-04-29 Thread via GitHub
Fokko merged PR #1956: URL: https://github.com/apache/iceberg-python/pull/1956 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Spark : Derive Stats From Manifest on the Fly [iceberg]

2025-04-29 Thread via GitHub
jeesou commented on PR #11615: URL: https://github.com/apache/iceberg/pull/11615#issuecomment-2840872077 Hi @RussellSpitzer could we please check this PR once? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Core: Support incremental compute for partition stats [iceberg]

2025-04-29 Thread via GitHub
ajantha-bhat commented on PR #12629: URL: https://github.com/apache/iceberg/pull/12629#issuecomment-2840855588 > After discussing this with @deniskuzZ, I feel that we probably would want to move the PartitionStatsUtil and the PartitionStatsHandler into the same package, and deprecate the me

Re: [PR] Spark: Update RewriteDataFilesSparkAction and RewritePositionDeleteFilesSparkAction to use the new APIs [iceberg]

2025-04-29 Thread via GitHub
pvary commented on code in PR #12692: URL: https://github.com/apache/iceberg/pull/12692#discussion_r2067879419 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/SparkBinPackFileRewriteRunner.java: ## @@ -56,16 +58,18 @@ protected void doRewrite(String groupId, L

Re: [PR] Spark: Update RewriteDataFilesSparkAction and RewritePositionDeleteFilesSparkAction to use the new APIs [iceberg]

2025-04-29 Thread via GitHub
pvary commented on code in PR #12692: URL: https://github.com/apache/iceberg/pull/12692#discussion_r2067879186 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/SparkShufflingDataRewritePlanner.java: ## @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] Spark: Update RewriteDataFilesSparkAction and RewritePositionDeleteFilesSparkAction to use the new APIs [iceberg]

2025-04-29 Thread via GitHub
pvary commented on code in PR #12692: URL: https://github.com/apache/iceberg/pull/12692#discussion_r2067872681 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteDataFilesProcedure.java: ## @@ -638,7 +638,7 @@ public void testRewriteDataF

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067872969 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreClientImpl.java: ## @@ -0,0 +1,594 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] Spark: Update RewriteDataFilesSparkAction and RewritePositionDeleteFilesSparkAction to use the new APIs [iceberg]

2025-04-29 Thread via GitHub
pvary commented on code in PR #12692: URL: https://github.com/apache/iceberg/pull/12692#discussion_r2067872681 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteDataFilesProcedure.java: ## @@ -638,7 +638,7 @@ public void testRewriteDataF

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067872740 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java: ## @@ -0,0 +1,405 @@ +/* + * Licensed to the Apache Software Foundation (

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067872433 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java: ## @@ -0,0 +1,405 @@ +/* + * Licensed to the Apache Software Foundation (

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067871268 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java: ## @@ -0,0 +1,394 @@ +/* + * Licensed to the Apache Software Foundation (

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067868284 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java: ## @@ -0,0 +1,405 @@ +/* + * Licensed to the Apache Software Foundation (

[I] table.doCommit call fails when metadata file is already removed [iceberg-go]

2025-04-29 Thread via GitHub
jwtryg opened a new issue, #411: URL: https://github.com/apache/iceberg-go/issues/411 ### Apache Iceberg version main (development) ### Please describe the bug 🐞 When `write.metadata.delete-after-commit.enabled` is set to true, the [deleteOldMetadata](https://github.com/

Re: [PR] Core: Support incremental compute for partition stats [iceberg]

2025-04-29 Thread via GitHub
ajantha-bhat commented on code in PR #12629: URL: https://github.com/apache/iceberg/pull/12629#discussion_r2067851119 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsHandler.java: ## @@ -135,20 +142,114 @@ public static PartitionStatisticsFile computeAndWriteStatsFi

Re: [PR] Spark: Update RewriteDataFilesSparkAction and RewritePositionDeleteFilesSparkAction to use the new APIs [iceberg]

2025-04-29 Thread via GitHub
stevenzwu commented on code in PR #12692: URL: https://github.com/apache/iceberg/pull/12692#discussion_r2067584705 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: ## @@ -120,36 +111,39 @@ protected RewriteDataFilesSparkAction

Re: [PR] Spark 3.4: Migrate tests in sql [iceberg]

2025-04-29 Thread via GitHub
tomtongue commented on PR #12934: URL: https://github.com/apache/iceberg/pull/12934#issuecomment-2840768748 @nastra Sorry for the delay, could you review this when you have a chance? I believe one or two PRs will complete the `spark.sql` package migration. -- This is an automated message

Re: [PR] Spark: support rewrite on specified target branch [iceberg]

2025-04-29 Thread via GitHub
amitgilad3 commented on PR #12257: URL: https://github.com/apache/iceberg/pull/12257#issuecomment-2840760461 @lliangyu-lin - I have 1 fix todo, will work on it today and if all is good we can merge 👍 -- This is an automated message from the Apache Git Service. To respond to the mess

Re: [PR] Docs: Add versioned Javadocs for 1.9.0 [iceberg]

2025-04-29 Thread via GitHub
manuzhang commented on PR #12920: URL: https://github.com/apache/iceberg/pull/12920#issuecomment-2840720548 As a follow-up, can we let GitHub action to create a PR to update javadocs on a new tag? -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] Infra: Add 1.8.x to protected branch [iceberg]

2025-04-29 Thread via GitHub
manuzhang closed pull request #12830: Infra: Add 1.8.x to protected branch URL: https://github.com/apache/iceberg/pull/12830 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[I] feature to add timestamp in data and metadata file names in iceberg [iceberg]

2025-04-29 Thread via GitHub
yadav-ankit opened a new issue, #12935: URL: https://github.com/apache/iceberg/issues/12935 ### Feature Request / Improvement I am using iceberg sink connector to write files to s3 using HMS. At present its not possible to search data or metadata files in s3 as file names of ic

Re: [I] Create table should take in sort order/ distribution mode [iceberg]

2025-04-29 Thread via GitHub
manuzhang commented on issue #8179: URL: https://github.com/apache/iceberg/issues/8179#issuecomment-2840693797 Hi @fbertsch, As per discussion in https://github.com/apache/iceberg/issues/3547, it would be better supported by engines, in your case, Spark. -- This is an automated mes

Re: [PR] Core: Broaden exception handling in writer clean up logic [iceberg]

2025-04-29 Thread via GitHub
amogh-jahagirdar commented on code in PR #12863: URL: https://github.com/apache/iceberg/pull/12863#discussion_r2067768617 ## core/src/main/java/org/apache/iceberg/io/BaseTaskWriter.java: ## @@ -347,9 +346,9 @@ private void closeCurrent() throws IOException { if (curre

Re: [PR] Spec: Update partition stats for V3 [iceberg]

2025-04-29 Thread via GitHub
ajantha-bhat commented on PR #12098: URL: https://github.com/apache/iceberg/pull/12098#issuecomment-2840659790 @aokolnychyi: Looks like we had enough approvals and you can merge this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] Create table should take in sort order/ distribution mode [iceberg]

2025-04-29 Thread via GitHub
fbertsch commented on issue #8179: URL: https://github.com/apache/iceberg/issues/8179#issuecomment-2840609092 Raising this again, to discuss why it's useful. We’ve started using DBT w/ Spark + Iceberg. When using DBT, you write a `SQL SELECT` statement that DBT uses to either create t

Re: [I] How to build and package into jar file for Kafka Connect? [iceberg]

2025-04-29 Thread via GitHub
jesumyip commented on issue #12895: URL: https://github.com/apache/iceberg/issues/12895#issuecomment-2840602948 I just tested the build and it works great locally in Docker. Should be quite easy to put this into our ci/cd pipeline. I had to make some minor changes to your Dockerfile and sc

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067703701 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreClientImpl.java: ## @@ -0,0 +1,594 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067703309 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreUtils.java: ## @@ -0,0 +1,95 @@ +/* + * Licensed to the Apache Software Foundation (ASF

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067702487 ## bigquery/src/test/java/org/apache/iceberg/gcp/bigquery/BigQueryTableOperationsTest.java: ## @@ -0,0 +1,274 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067701790 ## bigquery/src/test/java/org/apache/iceberg/gcp/bigquery/BigQueryTableOperationsTest.java: ## @@ -0,0 +1,274 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067701639 ## bigquery/src/test/java/org/apache/iceberg/gcp/bigquery/BigQueryTableOperationsTest.java: ## @@ -0,0 +1,274 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067701528 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java: ## @@ -0,0 +1,405 @@ +/* + * Licensed to the Apache Software Foundation (

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067701133 ## gradle/libs.versions.toml: ## @@ -51,6 +51,7 @@ guava = "33.4.7-jre" hadoop3 = "3.4.1" httpcomponents-httpclient5 = "5.4.3" hive2 = { strictly = "2.3.10"} #

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067700635 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java: ## @@ -0,0 +1,405 @@ +/* + * Licensed to the Apache Software Foundation (

Re: [I] Implement partition writer [iceberg-rust]

2025-04-29 Thread via GitHub
ZENOTME commented on issue #342: URL: https://github.com/apache/iceberg-rust/issues/342#issuecomment-2840564972 > Hello. I've gone through ZENOTME's PR and experimented with getting a fanout partition writer working. It is not too much more effort. I was wondering if I could help out with a

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067695000 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java: ## @@ -0,0 +1,405 @@ +/* + * Licensed to the Apache Software Foundation (

Re: [I] Document table properties [iceberg-python]

2025-04-29 Thread via GitHub
github-actions[bot] closed issue #1231: Document table properties URL: https://github.com/apache/iceberg-python/issues/1231 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067652720 ## build.gradle: ## @@ -657,6 +657,47 @@ project(':iceberg-delta-lake') { } } +project(':iceberg-bigquery') { + test { +useJUnitPlatform() + } + + de

Re: [I] Expose PyIceberg table as PyArrow Dataset [iceberg-python]

2025-04-29 Thread via GitHub
github-actions[bot] commented on issue #30: URL: https://github.com/apache/iceberg-python/issues/30#issuecomment-2840513172 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occu

Re: [I] [feature request] Improve integration test reliance on docker [iceberg-python]

2025-04-29 Thread via GitHub
github-actions[bot] commented on issue #637: URL: https://github.com/apache/iceberg-python/issues/637#issuecomment-2840513126 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity oc

Re: [I] Document table properties [iceberg-python]

2025-04-29 Thread via GitHub
github-actions[bot] commented on issue #1231: URL: https://github.com/apache/iceberg-python/issues/1231#issuecomment-2840513035 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the A

Re: [PR] Build: Bump org.apache.hadoop.thirdparty:hadoop-shaded-guava from 1.3.0 to 1.4.0 [iceberg]

2025-04-29 Thread via GitHub
github-actions[bot] commented on PR #12684: URL: https://github.com/apache/iceberg/pull/12684#issuecomment-2840509046 This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pul

Re: [I] Add support for multiple table DataStream FlinkSink [iceberg]

2025-04-29 Thread via GitHub
github-actions[bot] commented on issue #11436: URL: https://github.com/apache/iceberg/issues/11436#issuecomment-2840508893 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] org.apache.iceberg.parquet.PruneColumns does not project properly for depth 3 structs. [iceberg]

2025-04-29 Thread via GitHub
github-actions[bot] commented on issue #11332: URL: https://github.com/apache/iceberg/issues/11332#issuecomment-2840508843 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] How to query table by partition filed? [iceberg]

2025-04-29 Thread via GitHub
github-actions[bot] closed issue #11329: How to query table by partition filed? URL: https://github.com/apache/iceberg/issues/11329 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [I] [feature request] Allow Java Iceberg library to write parquet files with special character column names [iceberg]

2025-04-29 Thread via GitHub
github-actions[bot] commented on issue #10120: URL: https://github.com/apache/iceberg/issues/10120#issuecomment-2840508572 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] RewriteManifest with more options [iceberg]

2025-04-29 Thread via GitHub
github-actions[bot] commented on issue #9615: URL: https://github.com/apache/iceberg/issues/9615#issuecomment-2840508526 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Max number of columns [iceberg]

2025-04-29 Thread via GitHub
github-actions[bot] commented on issue #9220: URL: https://github.com/apache/iceberg/issues/9220#issuecomment-2840508482 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Can't select table If drop the corresponding column after replacing or dropping partition spec field [iceberg]

2025-04-29 Thread via GitHub
github-actions[bot] closed issue #11314: Can't select table If drop the corresponding column after replacing or dropping partition spec field URL: https://github.com/apache/iceberg/issues/11314 -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [I] org.apache.iceberg.parquet.PruneColumns does not project properly for depth 3 structs. [iceberg]

2025-04-29 Thread via GitHub
github-actions[bot] closed issue #11332: org.apache.iceberg.parquet.PruneColumns does not project properly for depth 3 structs. URL: https://github.com/apache/iceberg/issues/11332 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [I] How to use iceberg metadata or something to get which partition of table does spark sql hit? [iceberg]

2025-04-29 Thread via GitHub
github-actions[bot] commented on issue #11331: URL: https://github.com/apache/iceberg/issues/11331#issuecomment-2840508809 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] How to query table by partition filed? [iceberg]

2025-04-29 Thread via GitHub
github-actions[bot] commented on issue #11329: URL: https://github.com/apache/iceberg/issues/11329#issuecomment-2840508774 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] How to use iceberg metadata or something to get which partition of table does spark sql hit? [iceberg]

2025-04-29 Thread via GitHub
github-actions[bot] closed issue #11331: How to use iceberg metadata or something to get which partition of table does spark sql hit? URL: https://github.com/apache/iceberg/issues/11331 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [I] Can't select table If drop the corresponding column after replacing or dropping partition spec field [iceberg]

2025-04-29 Thread via GitHub
github-actions[bot] commented on issue #11314: URL: https://github.com/apache/iceberg/issues/11314#issuecomment-2840508743 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Flink: Maintenance - Add support for more kinds of scheduling [iceberg]

2025-04-29 Thread via GitHub
github-actions[bot] commented on issue #11246: URL: https://github.com/apache/iceberg/issues/11246#issuecomment-2840508698 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] [RewriteDataFiles] add option from-snapshot to support minor compaction [iceberg]

2025-04-29 Thread via GitHub
github-actions[bot] commented on issue #10824: URL: https://github.com/apache/iceberg/issues/10824#issuecomment-2840508649 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Drop table purge issue for parquet tables with SparkSessionCatalog [iceberg]

2025-04-29 Thread via GitHub
github-actions[bot] closed issue #10157: Drop table purge issue for parquet tables with SparkSessionCatalog URL: https://github.com/apache/iceberg/issues/10157 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] Drop table purge issue for parquet tables with SparkSessionCatalog [iceberg]

2025-04-29 Thread via GitHub
github-actions[bot] commented on issue #10157: URL: https://github.com/apache/iceberg/issues/10157#issuecomment-2840508625 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] RewriteManifest with more options [iceberg]

2025-04-29 Thread via GitHub
github-actions[bot] closed issue #9615: RewriteManifest with more options URL: https://github.com/apache/iceberg/issues/9615 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Spark: Update RewriteDataFilesSparkAction and RewritePositionDeleteFilesSparkAction to use the new APIs [iceberg]

2025-04-29 Thread via GitHub
stevenzwu commented on code in PR #12692: URL: https://github.com/apache/iceberg/pull/12692#discussion_r2067626037 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/SparkShufflingDataRewritePlanner.java: ## @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Softwar

Re: [PR] Spark: Update RewriteDataFilesSparkAction and RewritePositionDeleteFilesSparkAction to use the new APIs [iceberg]

2025-04-29 Thread via GitHub
stevenzwu commented on code in PR #12692: URL: https://github.com/apache/iceberg/pull/12692#discussion_r2067626037 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/SparkShufflingDataRewritePlanner.java: ## @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Softwar

Re: [PR] Add table property to disable/enable parquet column statistics #12770 [iceberg]

2025-04-29 Thread via GitHub
huaxiangsun commented on code in PR #12771: URL: https://github.com/apache/iceberg/pull/12771#discussion_r2067603465 ## docs/docs/configuration.md: ## @@ -52,6 +52,8 @@ Iceberg tables support table properties to configure table behavior, like the de | write.parquet.bloom-filte

Re: [PR] AWS: add view support for Glue Catalog [iceberg]

2025-04-29 Thread via GitHub
lawofcycles commented on code in PR #12544: URL: https://github.com/apache/iceberg/pull/12544#discussion_r2067587100 ## aws/src/integration/java/org/apache/iceberg/aws/glue/TestGlueCatalogView.java: ## @@ -0,0 +1,883 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] Spark: Update RewriteDataFilesSparkAction and RewritePositionDeleteFilesSparkAction to use the new APIs [iceberg]

2025-04-29 Thread via GitHub
stevenzwu commented on code in PR #12692: URL: https://github.com/apache/iceberg/pull/12692#discussion_r2067584705 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: ## @@ -120,36 +111,39 @@ protected RewriteDataFilesSparkAction

Re: [PR] Spark: Update RewriteDataFilesSparkAction and RewritePositionDeleteFilesSparkAction to use the new APIs [iceberg]

2025-04-29 Thread via GitHub
stevenzwu commented on code in PR #12692: URL: https://github.com/apache/iceberg/pull/12692#discussion_r2067584229 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: ## @@ -120,36 +111,39 @@ protected RewriteDataFilesSparkAction

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
danielcweeks commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067559631 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java: ## @@ -0,0 +1,405 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
danielcweeks commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067560229 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java: ## @@ -0,0 +1,405 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
danielcweeks commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067558777 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java: ## @@ -0,0 +1,405 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] [Spark]Add max files rewrite option for RewriteAction [iceberg]

2025-04-29 Thread via GitHub
coderfender commented on code in PR #12824: URL: https://github.com/apache/iceberg/pull/12824#discussion_r2067555906 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: ## @@ -407,15 +409,49 @@ private Builder doExecuteWithPartial

Re: [PR] [Spark]Add max files rewrite option for RewriteAction [iceberg]

2025-04-29 Thread via GitHub
sririshindra commented on code in PR #12824: URL: https://github.com/apache/iceberg/pull/12824#discussion_r2067557994 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: ## @@ -407,15 +409,49 @@ private Builder doExecuteWithPartia

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
danielcweeks commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067556600 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java: ## @@ -0,0 +1,405 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] [Spark]Add max files rewrite option for RewriteAction [iceberg]

2025-04-29 Thread via GitHub
coderfender commented on code in PR #12824: URL: https://github.com/apache/iceberg/pull/12824#discussion_r2067555906 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: ## @@ -407,15 +409,49 @@ private Builder doExecuteWithPartial

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
danielcweeks commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067550907 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java: ## @@ -0,0 +1,405 @@ +/* + * Licensed to the Apache Software Foundation

Re: [I] How to build and package into jar file for Kafka Connect? [iceberg]

2025-04-29 Thread via GitHub
jesumyip commented on issue #12895: URL: https://github.com/apache/iceberg/issues/12895#issuecomment-2840412709 Thank you @fuzing !!! You are a godsend! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] feat: delete orphaned files [iceberg-python]

2025-04-29 Thread via GitHub
jayceslesar commented on code in PR #1958: URL: https://github.com/apache/iceberg-python/pull/1958#discussion_r2067548613 ## pyiceberg/table/__init__.py: ## @@ -1371,6 +1375,45 @@ def to_polars(self) -> pl.LazyFrame: return pl.scan_iceberg(self) +def delete_orph

Re: [PR] feat: delete orphaned files [iceberg-python]

2025-04-29 Thread via GitHub
jayceslesar commented on code in PR #1958: URL: https://github.com/apache/iceberg-python/pull/1958#discussion_r2067540543 ## pyiceberg/table/inspect.py: ## @@ -645,10 +645,10 @@ def data_files(self, snapshot_id: Optional[int] = None) -> "pa.Table": def delete_files(self, s

Re: [PR] [Spark]Add max files rewrite option for RewriteAction [iceberg]

2025-04-29 Thread via GitHub
sririshindra commented on code in PR #12824: URL: https://github.com/apache/iceberg/pull/12824#discussion_r2067539142 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: ## @@ -407,15 +409,49 @@ private Builder doExecuteWithPartia

Re: [PR] Spec: Update partition stats for V3 [iceberg]

2025-04-29 Thread via GitHub
aokolnychyi commented on PR #12098: URL: https://github.com/apache/iceberg/pull/12098#issuecomment-2840400875 This became stale. I reopened it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[PR] Spec: Update partition stats for V3 [iceberg]

2025-04-29 Thread via GitHub
aokolnychyi opened a new pull request, #12098: URL: https://github.com/apache/iceberg/pull/12098 This PR adapts our partition stats spec to accommodate DVs: - `position_delete_record_count` includes a sum of position deletes across position delete files and DVs - `position_delete_f

Re: [PR] feat: delete orphaned files [iceberg-python]

2025-04-29 Thread via GitHub
jayceslesar commented on code in PR #1958: URL: https://github.com/apache/iceberg-python/pull/1958#discussion_r2067532346 ## pyiceberg/table/__init__.py: ## @@ -1371,6 +1375,45 @@ def to_polars(self) -> pl.LazyFrame: return pl.scan_iceberg(self) +def delete_orph

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-04-29 Thread via GitHub
danielcweeks commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2067530383 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java: ## @@ -0,0 +1,405 @@ +/* + * Licensed to the Apache Software Foundation

[PR] feat: delete orphaned files [iceberg-python]

2025-04-29 Thread via GitHub
jayceslesar opened a new pull request, #1958: URL: https://github.com/apache/iceberg-python/pull/1958 Closes #1200 # Rationale for this change Ability to do more table maintenance from pyiceberg (iceberg-python?) # Are these changes tested? Added a test! # Are ther

Re: [PR] feat: Add commit table support for Glue Catalog [iceberg-go]

2025-04-29 Thread via GitHub
zeroshade merged PR #403: URL: https://github.com/apache/iceberg-go/pull/403 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] [Spark]Add max files rewrite option for RewriteAction [iceberg]

2025-04-29 Thread via GitHub
RussellSpitzer commented on code in PR #12824: URL: https://github.com/apache/iceberg/pull/12824#discussion_r2067521513 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: ## @@ -407,15 +409,49 @@ private Builder doExecuteWithPart

  1   2   3   >