Re: [I] Make the OAuth2 request scope configurable [iceberg-python]

2024-03-04 Thread via GitHub
Fokko commented on issue #462: URL: https://github.com/apache/iceberg-python/issues/462#issuecomment-1978155414 #484 went in, closing this one ๐Ÿ‘ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [I] Make the OAuth2 request scope configurable [iceberg-python]

2024-03-04 Thread via GitHub
Fokko closed issue #462: Make the OAuth2 request scope configurable URL: https://github.com/apache/iceberg-python/issues/462 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Fix uuid partition internal representation from str to bytes [iceberg-python]

2024-03-04 Thread via GitHub
Fokko commented on PR #489: URL: https://github.com/apache/iceberg-python/pull/489#issuecomment-1978153309 @jqin61 sorry for the confusion there. We do this for the Literal itself: https://github.com/apache/iceberg-python/blob/9d01b02f14e761b36fe0b88a82274037f7ced5a7/pyiceberg/expressions/li

Re: [PR] [Bug Fix] cast None `current-snapshot-id` as -1 for Backwards Compatibility [iceberg-python]

2024-03-04 Thread via GitHub
Fokko commented on code in PR #473: URL: https://github.com/apache/iceberg-python/pull/473#discussion_r1512278316 ## pyiceberg/table/metadata.py: ## @@ -263,6 +264,12 @@ def sort_order_by_id(self, sort_order_id: int) -> Optional[SortOrder]: """Get the sort order by sor

Re: [PR] chord: Add IssueNavigationLink for RustRover [iceberg-rust]

2024-03-04 Thread via GitHub
stream2000 commented on PR #230: URL: https://github.com/apache/iceberg-rust/pull/230#issuecomment-1978144885 > Ok, but I prefer put it in `.idea/.gitignore`. Sure and done, thanks for your review~ -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [I] feat: Implement expression rewriter to remove `NOT` operator. [iceberg-rust]

2024-03-04 Thread via GitHub
liurenjie1024 commented on issue #150: URL: https://github.com/apache/iceberg-rust/issues/150#issuecomment-1978143748 > @liurenjie1024 could you assign this task to me please? :-) Yes, sure. Thanks for contributing! -- This is an automated message from the Apache Git Service. To res

Re: [PR] Tests should explicitly check for `schema_id` [iceberg-python]

2024-03-04 Thread via GitHub
Fokko merged PR #487: URL: https://github.com/apache/iceberg-python/pull/487 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Issue with CALL parsing [iceberg]

2024-03-04 Thread via GitHub
arvindeybram commented on issue #8343: URL: https://github.com/apache/iceberg/issues/8343#issuecomment-1978136859 I am facing the same issue in pyspark - when creating external tables in hive using ICEBERG format. ```ParseException: [PARSE_SYNTAX_ERROR] Syntax error at or near 'ICEBER

Re: [PR] Make optional oauth configurable [iceberg-python]

2024-03-04 Thread via GitHub
Fokko commented on PR #486: URL: https://github.com/apache/iceberg-python/pull/486#issuecomment-1978136175 @himadripal Can you fix the issue here? And update the doc in the same PR? Apart from that, this looks good. Thanks for working on this ๐Ÿ‘ -- This is an automated message from the Ap

Re: [PR] Make scope configurable [iceberg-python]

2024-03-04 Thread via GitHub
Fokko commented on PR #484: URL: https://github.com/apache/iceberg-python/pull/484#issuecomment-1978135226 Thanks for working on this @himadripal and for the review @flyrain -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] add rest scope in the config documentation [iceberg-python]

2024-03-04 Thread via GitHub
Fokko commented on PR #495: URL: https://github.com/apache/iceberg-python/pull/495#issuecomment-1978134798 Thanks for working on this @himadripal -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Sql catalog [iceberg-rust]

2024-03-04 Thread via GitHub
sdd commented on code in PR #229: URL: https://github.com/apache/iceberg-rust/pull/229#discussion_r1512266440 ## crates/catalog/sql/src/catalog.rs: ## @@ -0,0 +1,397 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. Se

Re: [PR] Make scope configurable [iceberg-python]

2024-03-04 Thread via GitHub
Fokko merged PR #484: URL: https://github.com/apache/iceberg-python/pull/484 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] feat: Implement expression rewriter to remove `NOT` operator. [iceberg-rust]

2024-03-04 Thread via GitHub
sdd commented on issue #150: URL: https://github.com/apache/iceberg-rust/issues/150#issuecomment-1978133744 @liurenjie1024 could you assign this task to me please? :-) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] add rest scope in the config documentation [iceberg-python]

2024-03-04 Thread via GitHub
Fokko merged PR #495: URL: https://github.com/apache/iceberg-python/pull/495 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Update bug isse template release list [iceberg-python]

2024-03-04 Thread via GitHub
Fokko merged PR #496: URL: https://github.com/apache/iceberg-python/pull/496 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] chord: Add IssueNavigationLink for RustRover [iceberg-rust]

2024-03-04 Thread via GitHub
Xuanwo commented on PR #230: URL: https://github.com/apache/iceberg-rust/pull/230#issuecomment-1978122647 > I think excluding `.idea/vcs.xml` in this way should be enough. Ok, but I prefer put it in `.idea/.gitignore`. -- This is an automated message from the Apache Git Service. To

Re: [PR] Sql catalog [iceberg-rust]

2024-03-04 Thread via GitHub
sdd commented on code in PR #229: URL: https://github.com/apache/iceberg-rust/pull/229#discussion_r1512249728 ## crates/catalog/sql/src/catalog.rs: ## @@ -0,0 +1,397 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. Se

Re: [PR] Flink: Supports specifying comment for iceberg fields in create table and addcolumn syntax using flinksql [iceberg]

2024-03-04 Thread via GitHub
huyuanfeng2018 commented on PR #9606: URL: https://github.com/apache/iceberg/pull/9606#issuecomment-1978115143 > Thanks @huyuanfeng2018 for the PR and @stevenzwu for the review! > > @huyuanfeng2018: Could you please backport the changes to Flink 1.17 and 1.16? > > Thanks, Peter

Re: [PR] chord: Add IssueNavigationLink for RustRover [iceberg-rust]

2024-03-04 Thread via GitHub
stream2000 commented on PR #230: URL: https://github.com/apache/iceberg-rust/pull/230#issuecomment-1978109062 > https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore ```shell .idea/* !.idea/vcs.xml ``` I think excluding `.idea/vcs.xml` in this way should

Re: [PR] Sql catalog [iceberg-rust]

2024-03-04 Thread via GitHub
sdd commented on code in PR #229: URL: https://github.com/apache/iceberg-rust/pull/229#discussion_r1512233865 ## crates/catalog/sql/Cargo.toml: ## @@ -0,0 +1,52 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the N

[PR] Fix pagination spec description [iceberg]

2024-03-04 Thread via GitHub
rahil-c opened a new pull request, #9866: URL: https://github.com/apache/iceberg/pull/9866 Minor revision for comments from @rdblue on the following pr: https://github.com/apache/iceberg/pull/9660 cc @jackye1995 -- This is an automated message from the Apache Git Service. To respo

Re: [PR] feat: add `UnboundPredicate::negate()` [iceberg-rust]

2024-03-04 Thread via GitHub
sdd commented on PR #228: URL: https://github.com/apache/iceberg-rust/pull/228#issuecomment-1978098358 @liurenjie1024: this is ready for review ๐Ÿ‘๐Ÿผ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] chord: Add IssueNavigationLink for RustRover [iceberg-rust]

2024-03-04 Thread via GitHub
stream2000 commented on PR #230: URL: https://github.com/apache/iceberg-rust/pull/230#issuecomment-1978092643 > Sorry, I'm not sure if adding an IDE specific configuration is a good choice. cc @Xuanwo @ZENOTME @Fokko WDYT? @liurenjie1024 this is just a minor improvement aimed at enhan

Re: [PR] chord: Add IssueNavigationLink for RustRover [iceberg-rust]

2024-03-04 Thread via GitHub
liurenjie1024 commented on PR #230: URL: https://github.com/apache/iceberg-rust/pull/230#issuecomment-1978074123 Sorry, I'm not sure if adding an IDE specific configuration is a good choice. cc @Xuanwo @ZENOTME @Fokko WDYT? -- This is an automated message from the Apache Git Service. To r

Re: [PR] Migrate Read in Core to JUnit5 [iceberg]

2024-03-04 Thread via GitHub
tomtongue commented on code in PR #9849: URL: https://github.com/apache/iceberg/pull/9849#discussion_r1512208471 ## core/src/test/java/org/apache/iceberg/TestIncrementalDataTableScan.java: ## @@ -196,56 +193,66 @@ public void testRollbacks() { transaction.commitTransaction(

Re: [PR] Migrate Read in Core to JUnit5 [iceberg]

2024-03-04 Thread via GitHub
tomtongue commented on code in PR #9849: URL: https://github.com/apache/iceberg/pull/9849#discussion_r1512205833 ## core/src/test/java/org/apache/iceberg/TestBase.java: ## @@ -745,6 +745,12 @@ void assertEquals(String context, Object expected, Object actual) { assertTh

Re: [PR] Migrate Read in Core to JUnit5 [iceberg]

2024-03-04 Thread via GitHub
tomtongue commented on code in PR #9849: URL: https://github.com/apache/iceberg/pull/9849#discussion_r1512205470 ## core/src/test/java/org/apache/iceberg/TestBatchScans.java: ## @@ -51,27 +48,27 @@ public void testDataTableScan() { BatchScan scan = table.newBatchScan();

Re: [PR] Flink: Supports specifying comment for iceberg fields in create table and addcolumn syntax using flinksql [iceberg]

2024-03-04 Thread via GitHub
pvary commented on PR #9606: URL: https://github.com/apache/iceberg/pull/9606#issuecomment-1978057949 Thanks @huyuanfeng2018 for the PR and @stevenzwu for the review! @huyuanfeng2018: Could you please backport the changes to Flink 1.17 and 1.16? Thanks, Peter -- This is an

Re: [I] UUID write requires different record in Parquet and ORC/Avro [iceberg]

2024-03-04 Thread via GitHub
pvary commented on issue #1881: URL: https://github.com/apache/iceberg/issues/1881#issuecomment-1978055418 Is this still the case? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

[PR] chord: Add IssueNavigationLink for RustRover [iceberg-rust]

2024-03-04 Thread via GitHub
stream2000 opened a new pull request, #230: URL: https://github.com/apache/iceberg-rust/pull/230 Add IssueNavigationLink for RustRover -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Migrate Read in Core to JUnit5 [iceberg]

2024-03-04 Thread via GitHub
tomtongue commented on code in PR #9849: URL: https://github.com/apache/iceberg/pull/9849#discussion_r1512160426 ## core/src/test/java/org/apache/iceberg/TestMetadataTableScans.java: ## @@ -356,14 +353,14 @@ public void testPartitionsTableScanNoFilter() { Ty

Re: [PR] Migrate Read in Core to JUnit5 [iceberg]

2024-03-04 Thread via GitHub
tomtongue commented on code in PR #9849: URL: https://github.com/apache/iceberg/pull/9849#discussion_r1512157166 ## core/src/test/java/org/apache/iceberg/TestIncrementalDataTableScan.java: ## @@ -196,56 +193,66 @@ public void testRollbacks() { transaction.commitTransaction(

Re: [I] iceberg-core: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

2024-03-04 Thread via GitHub
nk1506 commented on issue #9085: URL: https://github.com/apache/iceberg/issues/9085#issuecomment-1977995330 @tomtongue , I have reverted changed for the above files. As recommended by @nastra I will work with `TestBase` for the related files. -- This is an automated message from the Apac

Re: [PR] Migrate Read in Core to JUnit5 [iceberg]

2024-03-04 Thread via GitHub
tomtongue commented on code in PR #9849: URL: https://github.com/apache/iceberg/pull/9849#discussion_r1512150409 ## core/src/test/java/org/apache/iceberg/TestIncrementalDataTableScan.java: ## @@ -31,49 +35,42 @@ import org.apache.iceberg.relocated.com.google.common.collect.Iter

Re: [PR] Migrate Read in Core to JUnit5 [iceberg]

2024-03-04 Thread via GitHub
tomtongue commented on code in PR #9849: URL: https://github.com/apache/iceberg/pull/9849#discussion_r1512141524 ## core/src/test/java/org/apache/iceberg/TestEntriesMetadataTable.java: ## @@ -68,20 +63,19 @@ public void testEntriesTableScan() { Schema expectedSchema =

Re: [I] Why does FlinkSink writes position deletes in append-mode if identifier fields are specified? [iceberg]

2024-03-04 Thread via GitHub
tibercus closed issue #9773: Why does FlinkSink writes position deletes in append-mode if identifier fields are specified? URL: https://github.com/apache/iceberg/issues/9773 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [I] Why does FlinkSink writes position deletes in append-mode if identifier fields are specified? [iceberg]

2024-03-04 Thread via GitHub
tibercus commented on issue #9773: URL: https://github.com/apache/iceberg/issues/9773#issuecomment-1977975169 @pvary Hi Peter! Thanks, now that I know about changelog stream in Flink, the behaviour makes perfect sense. -- This is an automated message from the Apache Git Service. To

Re: [PR] Flink: Made IcebergFilesCommitter work with single phase commit [iceberg]

2024-03-04 Thread via GitHub
mudit-97 commented on PR #9694: URL: https://github.com/apache/iceberg/pull/9694#issuecomment-1977933726 sure @stevenzwu , thanks for the inputs, in our case, we were okay with data duplication in non happy path scenarios but given your inputs, it might not be okay with everyone. We can dis

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-03-04 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1512074114 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -309,24 +312,6 @@ private List writePartitionedManifests(

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-03-04 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1512074114 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -309,24 +312,6 @@ private List writePartitionedManifests(

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-03-04 Thread via GitHub
zachdisc commented on PR #9731: URL: https://github.com/apache/iceberg/pull/9731#issuecomment-1977849542 @nastra nastra > @zachdisc Hey Zach, sorry I was trying to experiment with refactoring the current PR, but somehow the branch is public and I ended up directly committing to it..

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-03-04 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1512037464 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/CustomDataFileSorterUdf.java: ## @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-03-04 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1512056511 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -628,4 +613,50 @@ private Table table() { return tabl

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-03-04 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1512056069 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -628,4 +613,50 @@ private Table table() { return tabl

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-04 Thread via GitHub
szehon-ho commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1512046593 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -269,6 +284,160 @@ public void renameTable(TableIdentifier from, TableIdentifier orig

Re: [PR] Flink: Supports specifying comment for iceberg fields in create table and addcolumn syntax using flinksql [iceberg]

2024-03-04 Thread via GitHub
stevenzwu commented on PR #9606: URL: https://github.com/apache/iceberg/pull/9606#issuecomment-1977835880 thanks @huyuanfeng2018 for the contribution and @pvary for the review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] Flink: Supports specifying comment for iceberg fields in create table and addcolumn syntax using flinksql [iceberg]

2024-03-04 Thread via GitHub
stevenzwu merged PR #9606: URL: https://github.com/apache/iceberg/pull/9606 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [I] Add hive metastore catalog support [iceberg-rust]

2024-03-04 Thread via GitHub
Xuanwo commented on issue #113: URL: https://github.com/apache/iceberg-rust/issues/113#issuecomment-1977830450 > I've done a simple prototype with my hive-metastore running locally via docker-compose, which works just fine. So testing via integration tests seems to be no problem. It'

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-04 Thread via GitHub
szehon-ho commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1511994436 ## core/src/main/java/org/apache/iceberg/BaseMetastoreTableOperations.java: ## @@ -309,65 +304,20 @@ protected enum CommitStatus { * @return Commit Status of Succ

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-04 Thread via GitHub
szehon-ho commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1511970931 ## core/src/main/java/org/apache/iceberg/BaseMetastoreOperations.java: ## @@ -0,0 +1,114 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-03-04 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1512037464 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/CustomDataFileSorterUdf.java: ## @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software Foundation

Re: [PR] Flink: Supports specifying comment for iceberg fields in create table and addcolumn syntax using flinksql [iceberg]

2024-03-04 Thread via GitHub
huyuanfeng2018 commented on PR #9606: URL: https://github.com/apache/iceberg/pull/9606#issuecomment-1977807670 > LGTM. just 2 nit comments to be addressed done. @stevenzwu -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] Doc: Clarify table properties on metadata metrics inferred column defaults [iceberg]

2024-03-04 Thread via GitHub
stevenzwu commented on PR #9865: URL: https://github.com/apache/iceberg/pull/9865#issuecomment-1977787041 thanks @dramaticlly for clarifying the doc. I have always thought the 100 limit is applicable to nested fields. didn't know it is limited to the top level. -- This is an automated me

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-03-04 Thread via GitHub
zachdisc commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1512015577 ## api/src/main/java/org/apache/iceberg/actions/RewriteManifests.java: ## @@ -44,6 +47,38 @@ public interface RewriteManifests */ RewriteManifests rewriteIf(Pre

Re: [PR] Make optional oauth configurable [iceberg-python]

2024-03-04 Thread via GitHub
flyrain commented on code in PR #486: URL: https://github.com/apache/iceberg-python/pull/486#discussion_r1512006182 ## pyiceberg/catalog/rest.py: ## @@ -289,12 +291,25 @@ def auth_url(self) -> str: else: return self.url(Endpoints.get_token, prefixed=False)

[PR] Doc: Clarify table properties on metadata metrics inferred column defaults [iceberg]

2024-03-04 Thread via GitHub
dramaticlly opened a new pull request, #9865: URL: https://github.com/apache/iceberg/pull/9865 Based on comment from https://github.com/apache/iceberg/pull/5215/files#r915475413 and https://github.com/apache/iceberg/pull/5215/files#r915276052 100 as default limit is counted only

Re: [PR] Core: Support appending files with different specs [iceberg]

2024-03-04 Thread via GitHub
amogh-jahagirdar commented on code in PR #9860: URL: https://github.com/apache/iceberg/pull/9860#discussion_r1511996190 ## core/src/test/java/org/apache/iceberg/TestMergeAppend.java: ## @@ -92,6 +94,74 @@ public void testEmptyTableAppend() { statuses(Status.ADDED, Statu

Re: [PR] Feature make oauth `audience` configurable [iceberg]

2024-03-04 Thread via GitHub
flyrain commented on code in PR #9839: URL: https://github.com/apache/iceberg/pull/9839#discussion_r1511897858 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -178,15 +178,18 @@ public void initialize(String name, Map unresolved) { ConfigRespon

Re: [PR] Core: Support appending files with different specs [iceberg]

2024-03-04 Thread via GitHub
amogh-jahagirdar commented on code in PR #9860: URL: https://github.com/apache/iceberg/pull/9860#discussion_r1511975142 ## core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java: ## @@ -878,7 +886,7 @@ public Object updateEvent() { @SuppressWarnings("checkstyle:

Re: [I] UUID write requires different record in Parquet and ORC/Avro [iceberg]

2024-03-04 Thread via GitHub
github-actions[bot] commented on issue #1881: URL: https://github.com/apache/iceberg/issues/1881#issuecomment-1977708432 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-03-04 Thread via GitHub
jackye1995 commented on PR #9731: URL: https://github.com/apache/iceberg/pull/9731#issuecomment-1977706910 There is a more general question of why this refactoring cannot be done at API level, the key issue is that `RewriteManifests` cannot access the `Table` object so it cannot construct t

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-03-04 Thread via GitHub
jackye1995 commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1511961587 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/CustomDataFileSorterUdf.java: ## @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-03-04 Thread via GitHub
jackye1995 commented on code in PR #9731: URL: https://github.com/apache/iceberg/pull/9731#discussion_r1511961199 ## api/src/main/java/org/apache/iceberg/actions/RewriteManifests.java: ## @@ -44,6 +47,38 @@ public interface RewriteManifests */ RewriteManifests rewriteIf(P

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-03-04 Thread via GitHub
jackye1995 commented on PR #9731: URL: https://github.com/apache/iceberg/pull/9731#issuecomment-1977701904 @zachdisc Hey Zach, sorry I was trying to experiment with refactoring the current PR, but somehow the branch is public and I ended up directly committing to it... Let me know if

Re: [PR] Flink: Made IcebergFilesCommitter work with single phase commit [iceberg]

2024-03-04 Thread via GitHub
stevenzwu commented on PR #9694: URL: https://github.com/apache/iceberg/pull/9694#issuecomment-1977646231 In general, I feel unease of changing the Flink Iceberg sink behavior from 2pc to 1pc. Would at least have more broader community input before we think it is good to add this option. Mi

Re: [PR] Update specs from hugo to mkdocs format [iceberg]

2024-03-04 Thread via GitHub
bitsondatadev commented on code in PR #9861: URL: https://github.com/apache/iceberg/pull/9861#discussion_r1511891884 ## format/spec.md: ## @@ -791,7 +788,7 @@ Each version of table metadata is stored in a metadata folder under the tableโ€™ Notes: -1. The file system table sc

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-04 Thread via GitHub
jackye1995 commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1511883182 ## open-api/rest-catalog-open-api.yaml: ## @@ -2574,7 +2810,7 @@ components: allOf: - $ref: "#/components/schemas/TableRequirement" descriptio

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-04 Thread via GitHub
jackye1995 commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1511880091 ## open-api/rest-catalog-open-api.yaml: ## @@ -2106,6 +2208,139 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile'

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-04 Thread via GitHub
jackye1995 commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1511879160 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile'

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-04 Thread via GitHub
jackye1995 commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1511873297 ## open-api/rest-catalog-open-api.yaml: ## @@ -2068,6 +2162,145 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile'

Re: [I] Import statement unexpectedly executes functions in __init__.py [iceberg-python]

2024-03-04 Thread via GitHub
ndrluis commented on issue #497: URL: https://github.com/apache/iceberg-python/issues/497#issuecomment-1977552814 After reading the `__init__.py` file in each namespace, I believe that this problem exists only in the table, io and catalog namespaces. Does anyone have suggestions on ho

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-04 Thread via GitHub
jackye1995 commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1511858632 ## open-api/rest-catalog-open-api.yaml: ## @@ -537,6 +537,108 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-04 Thread via GitHub
jackye1995 commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1511857247 ## open-api/rest-catalog-open-api.yaml: ## @@ -2838,6 +3093,59 @@ components: additionalProperties: type: string +PreplanTableRequest:

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-04 Thread via GitHub
jackye1995 commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1511852671 ## open-api/rest-catalog-open-api.yaml: ## @@ -537,6 +537,108 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-04 Thread via GitHub
jackye1995 commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1511851647 ## open-api/rest-catalog-open-api.yaml: ## @@ -537,6 +537,108 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-04 Thread via GitHub
jackye1995 commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1511851647 ## open-api/rest-catalog-open-api.yaml: ## @@ -537,6 +537,108 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-04 Thread via GitHub
jackye1995 commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1509256679 ## open-api/rest-catalog-open-api.yaml: ## @@ -537,6 +537,108 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-04 Thread via GitHub
jackye1995 commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1509256679 ## open-api/rest-catalog-open-api.yaml: ## @@ -537,6 +537,108 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-03-04 Thread via GitHub
jackye1995 commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1509256679 ## open-api/rest-catalog-open-api.yaml: ## @@ -537,6 +537,108 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}

[I] The import statement is running the functions [iceberg-python]

2024-03-04 Thread via GitHub
ndrluis opened a new issue, #497: URL: https://github.com/apache/iceberg-python/issues/497 ### Apache Iceberg version None ### Please describe the bug ๐Ÿž When we import any functions that reside in `__init__.py`, it triggers the execution of the code instead of merely imp

Re: [PR] API,Core: Support Conditional Commits [iceberg]

2024-03-04 Thread via GitHub
fqaiser94 commented on code in PR #6513: URL: https://github.com/apache/iceberg/pull/6513#discussion_r1142716546 ## core/src/test/java/org/apache/iceberg/TestCustomValidations.java: ## @@ -0,0 +1,643 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

[PR] Update bug isse template release list [iceberg-python]

2024-03-04 Thread via GitHub
ndrluis opened a new pull request, #496: URL: https://github.com/apache/iceberg-python/pull/496 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-03-04 Thread via GitHub
rahil-c commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1511704319 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1608,17 @@ components: type: string example: [ "accounting", "tax" ] +PageToken: + des

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-03-04 Thread via GitHub
rahil-c commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1511701936 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1608,17 @@ components: type: string example: [ "accounting", "tax" ] +PageToken: + des

Re: [I] Implement Remaining Catalog operations for REST catalog [iceberg-go]

2024-03-04 Thread via GitHub
zeroshade commented on issue #63: URL: https://github.com/apache/iceberg-go/issues/63#issuecomment-1977305156 > Is that correct? If so, I don't see createTable implemented yet. I think that is required to test RenameTable. Should that be implemented first? Most likely, yes. -- This

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-03-04 Thread via GitHub
jackye1995 commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1511679559 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1608,17 @@ components: type: string example: [ "accounting", "tax" ] +PageToken: +

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-03-04 Thread via GitHub
jackye1995 commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1511679559 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1608,17 @@ components: type: string example: [ "accounting", "tax" ] +PageToken: +

Re: [PR] add support for glue.id [iceberg-python]

2024-03-04 Thread via GitHub
jrouly commented on PR #490: URL: https://github.com/apache/iceberg-python/pull/490#issuecomment-1977246026 @HonahX appreciate the review! Pushed a new commit with your suggestions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] Feature make oauth `audience` configurable [iceberg]

2024-03-04 Thread via GitHub
flyrain commented on PR #9839: URL: https://github.com/apache/iceberg/pull/9839#issuecomment-1977235781 Thanks @himadripal for working on it. Let me take a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] add support for glue.id [iceberg-python]

2024-03-04 Thread via GitHub
jrouly commented on code in PR #490: URL: https://github.com/apache/iceberg-python/pull/490#discussion_r1511608985 ## pyiceberg/catalog/glue.py: ## @@ -93,6 +93,13 @@ if TYPE_CHECKING: import pyarrow as pa + +# There is a unique Glue metastore in each AWS account and eac

Re: [PR] add support for glue.id [iceberg-python]

2024-03-04 Thread via GitHub
jrouly commented on code in PR #490: URL: https://github.com/apache/iceberg-python/pull/490#discussion_r1511604893 ## pyiceberg/catalog/glue.py: ## @@ -266,6 +289,10 @@ def __init__(self, name: str, **properties: Any): ) self.glue: GlueClient = session.client("

Re: [PR] add support for glue.id [iceberg-python]

2024-03-04 Thread via GitHub
jrouly commented on code in PR #490: URL: https://github.com/apache/iceberg-python/pull/490#discussion_r1511604071 ## pyiceberg/catalog/glue.py: ## @@ -252,6 +259,22 @@ def _construct_database_input(database_name: str, properties: Properties) -> Dat return database_input

Re: [I] Add hive metastore catalog support [iceberg-rust]

2024-03-04 Thread via GitHub
marvinlanhenke commented on issue #113: URL: https://github.com/apache/iceberg-rust/issues/113#issuecomment-1977202916 @Xuanwo I've started looking into this (just started yesterday taking a look at the codebase in general) - how would you go about unit testing the catalog trait for hms? Th

Re: [PR] Flink: Supports specifying comment for iceberg fields in create table and addcolumn syntax using flinksql [iceberg]

2024-03-04 Thread via GitHub
stevenzwu commented on code in PR #9606: URL: https://github.com/apache/iceberg/pull/9606#discussion_r1511504695 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/FlinkSchemaUtil.java: ## @@ -64,26 +68,59 @@ public static Schema convert(TableSchema schema) { RowTy

Re: [PR] Core: Support appending files with different specs [iceberg]

2024-03-04 Thread via GitHub
danielcweeks commented on code in PR #9860: URL: https://github.com/apache/iceberg/pull/9860#discussion_r1511419189 ## core/src/main/java/org/apache/iceberg/BaseReplacePartitions.java: ## @@ -87,31 +87,40 @@ public BaseReplacePartitions toBranch(String branch) { @Override

Re: [PR] Core: Support appending files with different specs [iceberg]

2024-03-04 Thread via GitHub
nastra commented on code in PR #9860: URL: https://github.com/apache/iceberg/pull/9860#discussion_r1511474175 ## core/src/test/java/org/apache/iceberg/TestMergeAppend.java: ## @@ -92,6 +94,74 @@ public void testEmptyTableAppend() { statuses(Status.ADDED, Status.ADDED));

Re: [PR] Sql catalog [iceberg-rust]

2024-03-04 Thread via GitHub
JanKaul commented on PR #229: URL: https://github.com/apache/iceberg-rust/pull/229#issuecomment-1977013448 PTAL @liurenjie1024 @Xuanwo @ZENOTME @Fokko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Update specs from hugo to mkdocs format [iceberg]

2024-03-04 Thread via GitHub
bitsondatadev commented on code in PR #9861: URL: https://github.com/apache/iceberg/pull/9861#discussion_r1511441289 ## format/spec.md: ## @@ -791,7 +788,7 @@ Each version of table metadata is stored in a metadata folder under the tableโ€™ Notes: -1. The file system table sc

Re: [PR] Update specs from hugo to mkdocs format [iceberg]

2024-03-04 Thread via GitHub
bitsondatadev commented on code in PR #9861: URL: https://github.com/apache/iceberg/pull/9861#discussion_r1511432119 ## format/spec.md: ## @@ -791,7 +788,7 @@ Each version of table metadata is stored in a metadata folder under the tableโ€™ Notes: -1. The file system table sc

  1   2   >