Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-05-08 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2081050614 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryTableOperations.java: ## @@ -0,0 +1,296 @@ +/* + * Licensed to the Apache Software Foundation (A

[I] Logically incorrect `await` statement could cause deadlock at some point [iceberg-rust]

2025-05-08 Thread via GitHub
sdd opened a new issue, #1305: URL: https://github.com/apache/iceberg-rust/issues/1305 ### Apache Iceberg Rust version None ### Describe the bug We're `await`ing on a spawned task that should proceed concurrently rather than being awaited upon. This could cause deadlock

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-05-08 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2081045506 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreClientImpl.java: ## @@ -0,0 +1,579 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-05-08 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2081049091 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryTableOperations.java: ## @@ -0,0 +1,296 @@ +/* + * Licensed to the Apache Software Foundation (A

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-05-08 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2081048540 ## build.gradle: ## @@ -657,6 +657,46 @@ project(':iceberg-delta-lake') { } } +project(':iceberg-bigquery') { + test { +useJUnitPlatform() + } + + de

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-05-08 Thread via GitHub
talatuyarer commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2081046612 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java: ## @@ -0,0 +1,385 @@ +/* + * Licensed to the Apache Software Foundation (

Re: [PR] Core: Enhance remove snapshots efficiency by executing them in bulk [iceberg]

2025-05-08 Thread via GitHub
ajantha-bhat commented on code in PR #12670: URL: https://github.com/apache/iceberg/pull/12670#discussion_r2081015914 ## core/src/main/java/org/apache/iceberg/TableMetadata.java: ## @@ -1450,6 +1452,10 @@ private Builder rewriteSnapshotsInternal(Collection idsToRemove, boolean

Re: [PR] Materialized View Spec [iceberg]

2025-05-08 Thread via GitHub
hashhar commented on code in PR #11041: URL: https://github.com/apache/iceberg/pull/11041#discussion_r2080999014 ## format/view-spec.md: ## @@ -160,6 +179,57 @@ Each entry in `version-log` is a struct with the following fields: | _required_ | `timestamp-ms` | Timestamp when t

[I] Illegal domain name [iceberg]

2025-05-08 Thread via GitHub
jesumyip opened a new issue, #13013: URL: https://github.com/apache/iceberg/issues/13013 ### Apache Iceberg version None ### Query engine None ### Please describe the bug šŸž I get this error `Caused by: javax.net.ssl.SSLHandshakeException: Illegal given doma

Re: [PR] [Spark]Add max files rewrite option for RewriteAction [iceberg]

2025-05-08 Thread via GitHub
pvary commented on code in PR #12824: URL: https://github.com/apache/iceberg/pull/12824#discussion_r2080967454 ## core/src/main/java/org/apache/iceberg/actions/BinPackRewriteFilePlanner.java: ## @@ -199,30 +214,48 @@ protected long defaultTargetFileSize() { public FileRewrite

Re: [PR] Core: Support incremental compute for partition stats [iceberg]

2025-05-08 Thread via GitHub
ajantha-bhat commented on PR #12629: URL: https://github.com/apache/iceberg/pull/12629#issuecomment-2865166306 @pvary, @gaborkaszab, @deniskuzZ, @nastra: PR is ready for review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [I] Nessie should throw a NoSuchNamespaceException when listing a non-existing namespace [iceberg]

2025-05-08 Thread via GitHub
coderfender commented on issue #12875: URL: https://github.com/apache/iceberg/issues/12875#issuecomment-2865151106 @akshatmardia Please let me know if you are still working on this and I could continue to work on my Draft PR , fix unit tests and open a pull request -- This is an automate

Re: [PR] Core: Support incremental compute for partition stats [iceberg]

2025-05-08 Thread via GitHub
ajantha-bhat commented on code in PR #12629: URL: https://github.com/apache/iceberg/pull/12629#discussion_r2080927538 ## core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java: ## @@ -40,27 +44,48 @@ public class PartitionStatsUtil { private PartitionStatsUtil() {}

Re: [PR] feat: add name mapping [iceberg-cpp]

2025-05-08 Thread via GitHub
lidavidm commented on code in PR #93: URL: https://github.com/apache/iceberg-cpp/pull/93#discussion_r2080931836 ## src/iceberg/name_mapping.cc: ## @@ -0,0 +1,267 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. Se

Re: [PR] Spark: Add 'skip_file_list' option to RewriteTablePathProcedure for optional file-list generation [iceberg]

2025-05-08 Thread via GitHub
slfan1989 commented on PR #12844: URL: https://github.com/apache/iceberg/pull/12844#issuecomment-2865095776 > @slfan1989 FYI, Spark 4.0 integration [will be redone](https://github.com/apache/iceberg/pull/13006). Anyway, could you please create a separate PR for API change only? @manu

Re: [PR] Revert "Spark 4.0 integration" [iceberg]

2025-05-08 Thread via GitHub
huaxingao commented on PR #13006: URL: https://github.com/apache/iceberg/pull/13006#issuecomment-2865076169 @manuzhang > Please update the PR description to summarize all the changes. Updated. Thanks! -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Support retrieving the latest Iceberg table on table scan [iceberg-rust]

2025-05-08 Thread via GitHub
phillipleblanc commented on code in PR #1297: URL: https://github.com/apache/iceberg-rust/pull/1297#discussion_r2080814466 ## crates/integrations/datafusion/src/table/mod.rs: ## @@ -130,8 +131,19 @@ impl TableProvider for IcebergTableProvider { filters: &[Expr],

Re: [I] Question: can I use puffin file to store any customized blobs? [iceberg]

2025-05-08 Thread via GitHub
dentiny commented on issue #13011: URL: https://github.com/apache/iceberg/issues/13011#issuecomment-2865060129 > Yes, I think you can define your blob type, https://iceberg.apache.org/puffin-spec/#blob-types. Thank you @manuzhang for the quick reply! I assume query engines (like spar

Re: [I] Question: can I use puffin file to store any customized blobs? [iceberg]

2025-05-08 Thread via GitHub
manuzhang commented on issue #13011: URL: https://github.com/apache/iceberg/issues/13011#issuecomment-2865054263 Yes, I think you can define your blob type, https://iceberg.apache.org/puffin-spec/#blob-types. -- This is an automated message from the Apache Git Service. To respond to the m

Re: [PR] Spark: Add 'skip_file_list' option to RewriteTablePathProcedure for optional file-list generation [iceberg]

2025-05-08 Thread via GitHub
manuzhang commented on PR #12844: URL: https://github.com/apache/iceberg/pull/12844#issuecomment-2865004582 @slfan1989 FYI, Spark 4.0 integration [will be redone](https://github.com/apache/iceberg/pull/13006). Anyway, could you please create a separate PR for API change only? -- This is

Re: [PR] AWS: Fix NullPointerException for glue tables without storage descriptor [iceberg]

2025-05-08 Thread via GitHub
github-actions[bot] commented on PR #12551: URL: https://github.com/apache/iceberg/pull/12551#issuecomment-2864751598 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Spark: Add 'skip_file_list' option to RewriteTablePathProcedure for optional file-list generation [iceberg]

2025-05-08 Thread via GitHub
slfan1989 commented on PR #12844: URL: https://github.com/apache/iceberg/pull/12844#issuecomment-2864981462 @szehon-ho @manuzhang @dramaticlly Could you please review this PR again? Thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] Core, OpenAPI: Add commit timestamp to CommitReport [iceberg]

2025-05-08 Thread via GitHub
manuzhang commented on code in PR #12990: URL: https://github.com/apache/iceberg/pull/12990#discussion_r2080809718 ## open-api/rest-catalog-open-api.yaml: ## @@ -3775,6 +3775,9 @@ components: properties: table-name: type: string +timestamp-mill

Re: [PR] introduce a new config class to manage configurable options following design ideas from Velox [iceberg-cpp]

2025-05-08 Thread via GitHub
lidavidm commented on code in PR #92: URL: https://github.com/apache/iceberg-cpp/pull/92#discussion_r2080854477 ## src/iceberg/util/config.h: ## @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See

[PR] Enable withCommitProperties for snapshot update operations [iceberg]

2025-05-08 Thread via GitHub
guykhazma opened a new pull request, #13012: URL: https://github.com/apache/iceberg/pull/13012 This PR adds support to allow custom snapshot properties to be specified when doing snapshot update operations such as rewrite data files by leveraging the `CommitMetadata.withCommitProperties` me

Re: [PR] introduce a new config class to manage configurable options following design ideas from Velox [iceberg-cpp]

2025-05-08 Thread via GitHub
wgtmac commented on PR #92: URL: https://github.com/apache/iceberg-cpp/pull/92#issuecomment-2864952785 What about renaming the title to `feat: add base config implementation`? We can mention Velox as a comment to the config class. -- This is an automated message from the Apache Git Servic

Re: [PR] introduce a new config class to manage configurable options following design ideas from Velox [iceberg-cpp]

2025-05-08 Thread via GitHub
wgtmac commented on code in PR #92: URL: https://github.com/apache/iceberg-cpp/pull/92#discussion_r2080820923 ## src/iceberg/util/config.h: ## @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See th

Re: [I] How do I get this to work with Snowflake Polaris? [iceberg]

2025-05-08 Thread via GitHub
jesumyip commented on issue #13009: URL: https://github.com/apache/iceberg/issues/13009#issuecomment-2864943846 Just fyi that in my testing I have to provide both: `iceberg.catalog.oauth2-server-uri` and `iceberg.catalog.rest.client.oauth2-server-uri` to get rid of the warning about

Re: [PR] feat: add manifest related structures [iceberg-cpp]

2025-05-08 Thread via GitHub
wgtmac commented on code in PR #91: URL: https://github.com/apache/iceberg-cpp/pull/91#discussion_r2079944001 ## src/iceberg/manifest_reader.h: ## @@ -26,13 +26,15 @@ #include #include "iceberg/file_reader.h" +#include "iceberg/iceberg_export.h" +#include "iceberg/type_fwd.

Re: [PR] Revert "Spark 4.0 integration" [iceberg]

2025-05-08 Thread via GitHub
manuzhang commented on PR #13006: URL: https://github.com/apache/iceberg/pull/13006#issuecomment-2864912636 @huaxingao Please update the PR description to summarize all the changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [I] Compilation failure with only storage-fs feature used [iceberg-rust]

2025-05-08 Thread via GitHub
dentiny commented on issue #1303: URL: https://github.com/apache/iceberg-rust/issues/1303#issuecomment-2864874589 A followup question: what's the idiomatic way to test different combination of features in CI, to prevent it happen? Local `cargo test` and `cargo build` only build for defaul

[PR] fix: Fix compilation failure when only storage-fs feature included [iceberg-rust]

2025-05-08 Thread via GitHub
dentiny opened a new pull request, #1304: URL: https://github.com/apache/iceberg-rust/pull/1304 ## Which issue does this PR close? - Closes https://github.com/apache/iceberg-rust/issues/1303 ## What changes are included in this PR? This PR makes return type `Return<>` exp

Re: [PR] feat: validate snapshot write compatibility [iceberg-python]

2025-05-08 Thread via GitHub
kaushiksrini commented on PR #1772: URL: https://github.com/apache/iceberg-python/pull/1772#issuecomment-2864831706 @sungwy, I implemented your feedback in the PR. Let me know what you think! -- This is an automated message from the Apache Git Service. To respond to the message, please log

[I] Compilation failure with only storage-fs feature used [iceberg-rust]

2025-05-08 Thread via GitHub
dentiny opened a new issue, #1303: URL: https://github.com/apache/iceberg-rust/issues/1303 ### Apache Iceberg Rust version 0.4.0 (latest version) ### Describe the bug I'm using main branch, and suffers compilation failure when only storage-fs feature included. ###

Re: [PR] Revert "Spark 4.0 integration" [iceberg]

2025-05-08 Thread via GitHub
huaxingao commented on PR #13006: URL: https://github.com/apache/iceberg/pull/13006#issuecomment-2864821184 @amogh-jahagirdar @wypoon > in https://github.com/apache/iceberg/pull/12494, there are 584 changed files, while in this PR, there are 580. The following 4 files are chan

Re: [PR] Revert "Spark 4.0 integration" [iceberg]

2025-05-08 Thread via GitHub
wypoon commented on PR #13006: URL: https://github.com/apache/iceberg/pull/13006#issuecomment-2864793091 I see that in https://github.com/apache/iceberg/pull/12494, there are 584 changed files, while in this PR, there are 580. Are the 4 files that are not reverted the build.gradle and Comet

Re: [I] Equality delete column constraints are not enforced [iceberg]

2025-05-08 Thread via GitHub
weijiii commented on issue #12971: URL: https://github.com/apache/iceberg/issues/12971#issuecomment-2864714088 > It seems that In your case someone is producing the equality delete file with custom code using Iceberg Java SDK. Is the identifier fields set for the table schema? Validation ab

Re: [PR] AWS: Fix NullPointerException for glue tables without storage descriptor [iceberg]

2025-05-08 Thread via GitHub
github-actions[bot] closed pull request #12551: AWS: Fix NullPointerException for glue tables without storage descriptor URL: https://github.com/apache/iceberg/pull/12551 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] chore: improve coordinator election logging [iceberg]

2025-05-08 Thread via GitHub
github-actions[bot] closed pull request #12609: chore: improve coordinator election logging URL: https://github.com/apache/iceberg/pull/12609 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [I] Implement list_views Method for hive catalog [iceberg-python]

2025-05-08 Thread via GitHub
github-actions[bot] closed issue #1240: Implement list_views Method for hive catalog URL: https://github.com/apache/iceberg-python/issues/1240 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [I] Implement list_views Method for hive catalog [iceberg-python]

2025-05-08 Thread via GitHub
github-actions[bot] commented on issue #1240: URL: https://github.com/apache/iceberg-python/issues/1240#issuecomment-2864754484 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the A

Re: [PR] Core: Make TestBase abstract [iceberg]

2025-05-08 Thread via GitHub
github-actions[bot] commented on PR #12678: URL: https://github.com/apache/iceberg/pull/12678#issuecomment-2864751678 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] chore: improve coordinator election logging [iceberg]

2025-05-08 Thread via GitHub
github-actions[bot] commented on PR #12609: URL: https://github.com/apache/iceberg/pull/12609#issuecomment-2864751652 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] Core: Make TestBase abstract [iceberg]

2025-05-08 Thread via GitHub
github-actions[bot] closed pull request #12678: Core: Make TestBase abstract URL: https://github.com/apache/iceberg/pull/12678 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] Revert "Spark 4.0 integration" [iceberg]

2025-05-08 Thread via GitHub
huaxingao commented on PR #13006: URL: https://github.com/apache/iceberg/pull/13006#issuecomment-2864709122 @amogh-jahagirdar @pan3793 @wypoon @szehon-ho All checks have passed. Could you please check the PR when you have a min? -- This is an automated message from the Apache Git Servi

Re: [PR] fix(table): Desanitize struct partition filter name [iceberg-go]

2025-05-08 Thread via GitHub
loicalleyne commented on PR #412: URL: https://github.com/apache/iceberg-go/pull/412#issuecomment-2864670704 Closing this PR in favour of #427 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] fix(table): Desanitize struct partition filter name [iceberg-go]

2025-05-08 Thread via GitHub
loicalleyne closed pull request #412: fix(table): Desanitize struct partition filter name URL: https://github.com/apache/iceberg-go/pull/412 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [I] Equality delete column constraints are not enforced [iceberg]

2025-05-08 Thread via GitHub
weijiii commented on issue #12971: URL: https://github.com/apache/iceberg/issues/12971#issuecomment-2864647911 > The problem is that the identifier fields rule (e.g. excluding FLOAT and DOUBLE types) is not enforced in your case? Yes, I would agree that it should be enforced, as floating nu

Re: [PR] Add all filles metadata tables [iceberg-python]

2025-05-08 Thread via GitHub
soumya-ghosh commented on code in PR #1626: URL: https://github.com/apache/iceberg-python/pull/1626#discussion_r2080552494 ## pyiceberg/table/inspect.py: ## @@ -523,7 +523,67 @@ def history(self) -> "pa.Table": return pa.Table.from_pylist(history, schema=history_schem

Re: [I] Equality delete column constraints are not enforced [iceberg]

2025-05-08 Thread via GitHub
JeonDaehong commented on issue #12971: URL: https://github.com/apache/iceberg/issues/12971#issuecomment-2864559667 > > Hello, just checking—are you planning to work on a PR for this yourself? > > @JeonDaehong Yes I would like to give it a try. Thanks for checking! I was cheering

Re: [I] Remove JUnit4 dependency from Flink [iceberg]

2025-05-08 Thread via GitHub
JeonDaehong commented on issue #12937: URL: https://github.com/apache/iceberg/issues/12937#issuecomment-2864557470 > @JeonDaehong ideally what we want is to be able to remove the JUnit4 dependency from Flink via the diff outlined in the issue description. So I would use the diff + the chang

Re: [PR] feat: Add initial support for create table CLI through json schema [iceberg-go]

2025-05-08 Thread via GitHub
zeroshade commented on code in PR #424: URL: https://github.com/apache/iceberg-go/pull/424#discussion_r2080536584 ## schema.go: ## @@ -51,6 +51,17 @@ type Schema struct { lazyNameMapping func() NameMapping } +// NewSchemaFromJson constructs a new schema with the provi

Re: [I] Remove JUnit4 dependency from Flink [iceberg]

2025-05-08 Thread via GitHub
JeonDaehong commented on issue #12937: URL: https://github.com/apache/iceberg/issues/12937#issuecomment-2864556106 > @JeonDaehong ideally what we want is to be able to remove the JUnit4 dependency from Flink via the diff outlined in the issue description. So I would use the diff + the chang

Re: [I] Equality delete column constraints are not enforced [iceberg]

2025-05-08 Thread via GitHub
stevenzwu commented on issue #12971: URL: https://github.com/apache/iceberg/issues/12971#issuecomment-2864446931 let me confirm the problem. The problem is that the identifier fields rule (e.g. excluding FLOAT and DOUBLE types) is not enforced? Yes, I would agree that it should be enforced,

Re: [PR] feat: Add initial support for create table CLI through json schema [iceberg-go]

2025-05-08 Thread via GitHub
lliangyu-lin commented on code in PR #424: URL: https://github.com/apache/iceberg-go/pull/424#discussion_r2080512350 ## schema.go: ## @@ -51,6 +51,17 @@ type Schema struct { lazyNameMapping func() NameMapping } +// NewSchemaFromJson constructs a new schema with the pr

Re: [PR] feat: Add initial support for create table CLI through json schema [iceberg-go]

2025-05-08 Thread via GitHub
lliangyu-lin commented on code in PR #424: URL: https://github.com/apache/iceberg-go/pull/424#discussion_r2080512350 ## schema.go: ## @@ -51,6 +51,17 @@ type Schema struct { lazyNameMapping func() NameMapping } +// NewSchemaFromJson constructs a new schema with the pr

Re: [I] table.doCommit call fails when metadata file is already removed [iceberg-go]

2025-05-08 Thread via GitHub
zeroshade closed issue #411: table.doCommit call fails when metadata file is already removed URL: https://github.com/apache/iceberg-go/issues/411 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Core: Refactor and use InternalData for partition stats [iceberg]

2025-05-08 Thread via GitHub
pvary merged PR #12946: URL: https://github.com/apache/iceberg/pull/12946 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-08 Thread via GitHub
Guosmilesmile commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2079919363 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/CommittableToTableChangeConverter.java: ## @@ -0,0 +1,178 @@ +/* + * Licensed to the Apache Sof

Re: [PR] Revert "Spark 4.0 integration" [iceberg]

2025-05-08 Thread via GitHub
huaxingao commented on PR #13006: URL: https://github.com/apache/iceberg/pull/13006#issuecomment-2864295828 Thanks @wypoon for the suggestion! I will try to resolve the conflicts on my side and see how it goes -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Build and test hive-metastore with Hive 2, 3 and 4 with a single source set [iceberg]

2025-05-08 Thread via GitHub
wypoon commented on PR #12721: URL: https://github.com/apache/iceberg/pull/12721#issuecomment-2864298245 > This mean we have the following options: > > 1. Limit the supported Hive versions > 2. Create hive-metastore jars to the different Hive versions, and remove them from the runt

[I] Question: can I use puffin file to store any customized blobs? [iceberg]

2025-05-08 Thread via GitHub
dentiny opened a new issue, #13011: URL: https://github.com/apache/iceberg/issues/13011 ### Query engine _No response_ ### Question Puffin file is introduced in iceberg V3, which serves as a container file to store any types of blobs. According to the [spec](https://i

Re: [PR] Core, OpenAPI: Add commit timestamp to CommitReport [iceberg]

2025-05-08 Thread via GitHub
flyrain commented on code in PR #12990: URL: https://github.com/apache/iceberg/pull/12990#discussion_r2080446157 ## open-api/rest-catalog-open-api.yaml: ## @@ -3775,6 +3775,9 @@ components: properties: table-name: type: string +timestamp-millis

Re: [PR] Revert "Spark 4.0 integration" [iceberg]

2025-05-08 Thread via GitHub
wypoon commented on PR #13006: URL: https://github.com/apache/iceberg/pull/13006#issuecomment-2864264962 Perhaps revert https://github.com/apache/iceberg/commit/809a2327f15dd0f5f9a20f43b3dbb1632bb00828 first, then this, followed by redo of Spark 4.0 support and redo of https://github.com/a

Re: [PR] Revert "Spark 4.0 integration" [iceberg]

2025-05-08 Thread via GitHub
wypoon commented on PR #13006: URL: https://github.com/apache/iceberg/pull/13006#issuecomment-2864251352 > I had an offline discussion with @amogh-jahagirdar. We have decided to revert the integration PR and redo it, so we can keep all the commit history. Thanks for the heads up! --

Re: [PR] Revert "Spark 4.0 integration" [iceberg]

2025-05-08 Thread via GitHub
huaxingao commented on PR #13006: URL: https://github.com/apache/iceberg/pull/13006#issuecomment-2864240727 I had an offline discussion with @amogh-jahagirdar. We have decided to revert the integration PR and redo it, so we can keep all the commit history. -- This is an automated message

Re: [PR] Revert "Spark 4.0 integration" [iceberg]

2025-05-08 Thread via GitHub
huaxingao commented on PR #13006: URL: https://github.com/apache/iceberg/pull/13006#issuecomment-2864233797 @wypoon right, we should always have 3 commits. I forgot to squash all the remaining commits into one and was trying to figure out whether the commit history was still preserved. -

Re: [PR] Spark: Avoid closing deserialized copies of shared resources like FileIO [iceberg]

2025-05-08 Thread via GitHub
xiaoxuandev commented on code in PR #12868: URL: https://github.com/apache/iceberg/pull/12868#discussion_r2080374719 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/source/SerializableTableWithSize.java: ## @@ -65,8 +66,7 @@ public static Table copyOf(Table table) {

Re: [PR] Spark: Custom snapshot property from session configuration [iceberg]

2025-05-08 Thread via GitHub
cccs-jory commented on PR #12999: URL: https://github.com/apache/iceberg/pull/12999#issuecomment-2864088601 > Similar requirements did come up in the past : #4956 is this not sufficient ? or you want this for specifically with the SQLConfs ? > > Can you please elaborate your use case

Re: [PR] Spark: Avoid closing deserialized copies of shared resources like FileIO [iceberg]

2025-05-08 Thread via GitHub
xiaoxuandev commented on code in PR #12868: URL: https://github.com/apache/iceberg/pull/12868#discussion_r2080365598 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SerializableTableWithSize.java: ## @@ -33,8 +33,9 @@ * * This class also implements AutoClos

Re: [I] Format-versioned `Snapshot`s in light of V3 additions [iceberg-python]

2025-05-08 Thread via GitHub
smaheshwar-pltr commented on issue #1973: URL: https://github.com/apache/iceberg-python/issues/1973#issuecomment-2864101431 > @smaheshwar-pltr Are you interested in adding those fields? Happy for someone else to take a stab! -- This is an automated message from the Apache Git Servi

Re: [PR] Write small decimals as INTs [iceberg-python]

2025-05-08 Thread via GitHub
Fokko commented on PR #1983: URL: https://github.com/apache/iceberg-python/pull/1983#issuecomment-2864053383 CI failing due to the release of snowballstemmer: https://pypi.org/project/snowballstemmer/#history -- This is an automated message from the Apache Git Service. To respond to the m

Re: [PR] feat(catalog/sql): Add views related ops [iceberg-go]

2025-05-08 Thread via GitHub
zeroshade commented on code in PR #414: URL: https://github.com/apache/iceberg-go/pull/414#discussion_r2080217572 ## catalog/sql/sql.go: ## @@ -147,6 +149,19 @@ type sqlIcebergNamespaceProps struct { PropertyValue sql.NullString } +type sqlIcebergView struct { +

Re: [PR] feat(catalog/sql): Add views related ops [iceberg-go]

2025-05-08 Thread via GitHub
zeroshade commented on code in PR #414: URL: https://github.com/apache/iceberg-go/pull/414#discussion_r2080242144 ## catalog/sql/sql.go: ## @@ -844,3 +871,235 @@ func (c *Catalog) UpdateNamespaceProperties(ctx context.Context, namespace table func (c *Catalog) CheckNamespaceEx

Re: [I] table.ParseMetadataBytes fails if table has no columns [iceberg-go]

2025-05-08 Thread via GitHub
zeroshade closed issue #405: table.ParseMetadataBytes fails if table has no columns URL: https://github.com/apache/iceberg-go/issues/405 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] build(deps): bump the gomod_updates group with 2 updates [iceberg-go]

2025-05-08 Thread via GitHub
zeroshade merged PR #418: URL: https://github.com/apache/iceberg-go/pull/418 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] table.ParseMetadataBytes fails if table has no columns [iceberg-go]

2025-05-08 Thread via GitHub
zeroshade commented on issue #405: URL: https://github.com/apache/iceberg-go/issues/405#issuecomment-2863919086 Fixed by #421 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] Revert "Spark 4.0 integration" [iceberg]

2025-05-08 Thread via GitHub
wypoon commented on PR #13006: URL: https://github.com/apache/iceberg/pull/13006#issuecomment-2863895232 > @pan3793 I think this over, actually, the key to preserve history is what happens in step 1, right? Ideally, I should have 3 commits, but as long as I have 1 and 2, it's OK to have mor

Re: [PR] feat(catalog/sql): Add views related ops [iceberg-go]

2025-05-08 Thread via GitHub
zeroshade commented on code in PR #414: URL: https://github.com/apache/iceberg-go/pull/414#discussion_r2080224366 ## catalog/sql/sql.go: ## @@ -844,3 +871,235 @@ func (c *Catalog) UpdateNamespaceProperties(ctx context.Context, namespace table func (c *Catalog) CheckNamespaceEx

Re: [PR] feat(catalog/sql): Add views related ops [iceberg-go]

2025-05-08 Thread via GitHub
zeroshade commented on code in PR #414: URL: https://github.com/apache/iceberg-go/pull/414#discussion_r2080217572 ## catalog/sql/sql.go: ## @@ -147,6 +149,19 @@ type sqlIcebergNamespaceProps struct { PropertyValue sql.NullString } +type sqlIcebergView struct { +

Re: [PR] feat: Add initial support for create table CLI through json schema [iceberg-go]

2025-05-08 Thread via GitHub
zeroshade commented on code in PR #424: URL: https://github.com/apache/iceberg-go/pull/424#discussion_r2080201687 ## schema.go: ## @@ -51,6 +51,17 @@ type Schema struct { lazyNameMapping func() NameMapping } +// NewSchemaFromJson constructs a new schema with the provi

Re: [PR] feat: Add initial support for create table CLI through json schema [iceberg-go]

2025-05-08 Thread via GitHub
zeroshade commented on code in PR #424: URL: https://github.com/apache/iceberg-go/pull/424#discussion_r2080199422 ## cmd/iceberg/main.go: ## @@ -79,7 +79,8 @@ Options: --warehouse TEXT specify the warehouse to use --config TEXT specify the path to the configuration

Re: [PR] feat: Add initial support for create table CLI through json schema [iceberg-go]

2025-05-08 Thread via GitHub
zeroshade commented on PR #424: URL: https://github.com/apache/iceberg-go/pull/424#issuecomment-2863845647 > An alternative idea is to use SQL like form like `id bigint NOT NULL COMMENT 'unique id'`, but the parsing may get complicated with nestings. Yea, let's not attempt to parse SQ

Re: [I] List CLI command fails when using Glue catalog with namespace [iceberg-go]

2025-05-08 Thread via GitHub
zeroshade closed issue #425: List CLI command fails when using Glue catalog with namespace URL: https://github.com/apache/iceberg-go/issues/425 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] fix: Improve list logic to fallback from tables to namespaces [iceberg-go]

2025-05-08 Thread via GitHub
zeroshade merged PR #426: URL: https://github.com/apache/iceberg-go/pull/426 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Equality delete column constraints are not enforced [iceberg]

2025-05-08 Thread via GitHub
weijiii commented on issue #12971: URL: https://github.com/apache/iceberg/issues/12971#issuecomment-2863811968 > Hello, just checking—are you planning to work on a PR for this yourself? @JeonDaehong Yes I would like to give it a try. Let me assign it to myself now. Thanks for checking

Re: [I] Equality delete column constraints are not enforced [iceberg]

2025-05-08 Thread via GitHub
RussellSpitzer commented on issue #12971: URL: https://github.com/apache/iceberg/issues/12971#issuecomment-2863793513 Do we say in the spec you can’t use those types? I was checking and didn’t see anything specific On Thu, May 8, 2025 at 8:24 AM JeonDaehong ***@***.***> wrote: >

[PR] [REST SPEC] Add first-row-id in the data files for Row Lineage [iceberg]

2025-05-08 Thread via GitHub
singhpk234 opened a new pull request, #13010: URL: https://github.com/apache/iceberg/pull/13010 ## About the change includes first_row_id in the DataFile for the rest spec used in scan planning, this information IMHO needs to passed from server to client for reads, as its inferred du

[I] How can I achieve fast writes with pyiceberg and s3 tables? [iceberg-python]

2025-05-08 Thread via GitHub
potatochipcoconut opened a new issue, #1984: URL: https://github.com/apache/iceberg-python/issues/1984 ### Question Hello pyicebergers, I am new to iceberg/s3 tables and am experimenting with using it as part of an IDP pipeline where we would store ocr data in s3 tables, that r

Re: [PR] fix(table): logging error when deleting old metadata [iceberg-go]

2025-05-08 Thread via GitHub
zeroshade merged PR #422: URL: https://github.com/apache/iceberg-go/pull/422 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-08 Thread via GitHub
Guosmilesmile commented on PR #12979: URL: https://github.com/apache/iceberg/pull/12979#issuecomment-2862105685 Rebased based on main -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] table.doCommit call fails when metadata file is already removed [iceberg-go]

2025-05-08 Thread via GitHub
zeroshade closed issue #411: table.doCommit call fails when metadata file is already removed URL: https://github.com/apache/iceberg-go/issues/411 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] logging error when deleting old metadata [iceberg-go]

2025-05-08 Thread via GitHub
zeroshade commented on PR #422: URL: https://github.com/apache/iceberg-go/pull/422#issuecomment-2863480690 Just needs to fix the linting issue first -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-05-08 Thread via GitHub
danielcweeks commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2080090426 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryTableOperations.java: ## @@ -0,0 +1,296 @@ +/* + * Licensed to the Apache Software Foundation (

Re: [PR] Catalog: Add BigQuery Metastore Catalog Support [iceberg]

2025-05-08 Thread via GitHub
danielcweeks commented on code in PR #12808: URL: https://github.com/apache/iceberg/pull/12808#discussion_r2080088898 ## bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java: ## @@ -0,0 +1,366 @@ +/* + * Licensed to the Apache Software Foundation

Re: [I] Remove JUnit4 dependency from Flink [iceberg]

2025-05-08 Thread via GitHub
nastra commented on issue #12937: URL: https://github.com/apache/iceberg/issues/12937#issuecomment-2863444737 @JeonDaehong ideally what we want is to be able to remove the JUnit4 dependency from Flink via the diff outlined in the issue description. So I would use the diff + the changes requ

Re: [PR] Spark: Avoid closing deserialized copies of shared resources like FileIO [iceberg]

2025-05-08 Thread via GitHub
mgmarino commented on code in PR #12868: URL: https://github.com/apache/iceberg/pull/12868#discussion_r2079911493 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SerializableTableWithSize.java: ## @@ -33,8 +33,9 @@ * * This class also implements AutoCloseab

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-08 Thread via GitHub
Guosmilesmile commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2079923441 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/CommittableToTableChangeConverter.java: ## @@ -0,0 +1,178 @@ +/* + * Licensed to the Apache Sof

Re: [PR] REST spec: Add encryption keys [iceberg]

2025-05-08 Thread via GitHub
rdblue commented on code in PR #12987: URL: https://github.com/apache/iceberg/pull/12987#discussion_r2079921543 ## open-api/rest-catalog-open-api.py: ## @@ -399,6 +406,16 @@ class EnableRowLineageUpdate(BaseUpdate): action: str = Field('enable-row-lineage', const=True)

Re: [PR] Flink: Support compact in iceberg sink v2 [iceberg]

2025-05-08 Thread via GitHub
Guosmilesmile commented on code in PR #12979: URL: https://github.com/apache/iceberg/pull/12979#discussion_r2079926365 ## flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/CommittableToTableChangeConverter.java: ## @@ -0,0 +1,178 @@ +/* + * Licensed to the Apache Sof

  1   2   >