[PR] feat: Implement binding expression [iceberg-rust]

2024-03-06 Thread via GitHub
liurenjie1024 opened a new pull request, #231: URL: https://github.com/apache/iceberg-rust/pull/231 Close #149 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscri

Re: [I] hive catalog drop table XX purge not delete hdfs path [iceberg]

2024-03-06 Thread via GitHub
mengshangxun commented on issue #9869: URL: https://github.com/apache/iceberg/issues/9869#issuecomment-1980316480 when create a hive catalog table ,it will create hdfs path in the spark warehouse,but now neither `drop table` or` drop table purge` can delete this path. if i use drop table a

Re: [PR] Core, Spark, Flink: Migrate tests that depend on TableTestBase to JUnit5 [iceberg]

2024-03-06 Thread via GitHub
nastra commented on code in PR #9862: URL: https://github.com/apache/iceberg/pull/9862#discussion_r1514068452 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestStreamingReaderOperator.java: ## @@ -34,59 +37,45 @@ import org.apache.iceberg.FileFormat; impor

Re: [PR] Core, Spark, Flink: Migrate tests that depend on TableTestBase to JUnit5 [iceberg]

2024-03-06 Thread via GitHub
nk1506 commented on code in PR #9862: URL: https://github.com/apache/iceberg/pull/9862#discussion_r1514064008 ## core/src/test/java/org/apache/iceberg/TestCreateTransaction.java: ## @@ -29,61 +31,62 @@ import org.apache.iceberg.types.TypeUtil; import org.apache.iceberg.types.T

Re: [PR] Core, Spark, Flink: Migrate tests that depend on TableTestBase to JUnit5 [iceberg]

2024-03-06 Thread via GitHub
nastra commented on PR #9862: URL: https://github.com/apache/iceberg/pull/9862#issuecomment-1980374933 @nk1506 can you please split this PR up into smaller and more manageble pieces so that it's easier reviewing the changes? -- This is an automated message from the Apache Git Service. To

[PR] Nessie: Gracefully handle empty namespace lookup [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat opened a new pull request, #9877: URL: https://github.com/apache/iceberg/pull/9877 Without this change, empty namespace lookup fails with below callstack. It is good to return false in this case like other catalogs. ``` org.projectnessie.error.NessieBadRequestExcep

Re: [PR] Nessie: Gracefully handle empty namespace lookup [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on PR #9877: URL: https://github.com/apache/iceberg/pull/9877#issuecomment-1980407013 cc: @adutra, @dimas-b, @snazy -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Nessie: Gracefully handle empty namespace lookup [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on code in PR #9877: URL: https://github.com/apache/iceberg/pull/9877#discussion_r1514102226 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNamespace.java: ## @@ -132,6 +133,15 @@ public void testSettingProperties() { .hasMessage("Namespace

Re: [PR] Nessie: Gracefully handle empty namespace lookup [iceberg]

2024-03-06 Thread via GitHub
nastra commented on code in PR #9877: URL: https://github.com/apache/iceberg/pull/9877#discussion_r1514142672 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNamespace.java: ## @@ -132,6 +133,15 @@ public void testSettingProperties() { .hasMessage("Namespace does

Re: [PR] Nessie: Gracefully handle empty namespace lookup [iceberg]

2024-03-06 Thread via GitHub
nastra commented on code in PR #9877: URL: https://github.com/apache/iceberg/pull/9877#discussion_r1514143315 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNamespace.java: ## @@ -132,6 +133,15 @@ public void testSettingProperties() { .hasMessage("Namespace does

Re: [PR] Nessie: Gracefully handle empty namespace lookup [iceberg]

2024-03-06 Thread via GitHub
nastra commented on code in PR #9877: URL: https://github.com/apache/iceberg/pull/9877#discussion_r1514144573 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNamespace.java: ## @@ -132,6 +133,15 @@ public void testSettingProperties() { .hasMessage("Namespace does

Re: [PR] Update specs from hugo to mkdocs format [iceberg]

2024-03-06 Thread via GitHub
nastra merged PR #9861: URL: https://github.com/apache/iceberg/pull/9861 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Docs: Add Daft into Iceberg documentation [iceberg]

2024-03-06 Thread via GitHub
nastra commented on code in PR #9836: URL: https://github.com/apache/iceberg/pull/9836#discussion_r1514218998 ## docs/docs/daft.md: ## @@ -0,0 +1,146 @@ +--- +title: "Daft" +--- + + +# Daft + +[Daft](www.getdaft.io) is a distributed query engine written in Python and Rust, two

Re: [PR] Docs: Add Daft into Iceberg documentation [iceberg]

2024-03-06 Thread via GitHub
nastra commented on code in PR #9836: URL: https://github.com/apache/iceberg/pull/9836#discussion_r1514219952 ## docs/docs/daft.md: ## @@ -0,0 +1,146 @@ +--- +title: "Daft" +--- + + +# Daft + +[Daft](www.getdaft.io) is a distributed query engine written in Python and Rust, two

Re: [PR] Docs: Add Daft into Iceberg documentation [iceberg]

2024-03-06 Thread via GitHub
nastra commented on code in PR #9836: URL: https://github.com/apache/iceberg/pull/9836#discussion_r1514225357 ## docs/docs/daft.md: ## @@ -0,0 +1,146 @@ +--- +title: "Daft" +--- + + +# Daft Review Comment: so it seems the site can't be actually built when running the docs lo

Re: [PR] Docs: Add Daft into Iceberg documentation [iceberg]

2024-03-06 Thread via GitHub
nastra commented on code in PR #9836: URL: https://github.com/apache/iceberg/pull/9836#discussion_r1514225357 ## docs/docs/daft.md: ## @@ -0,0 +1,146 @@ +--- +title: "Daft" +--- + + +# Daft Review Comment: so it seems the site can't be actually built when serving the docs lo

Re: [PR] Docs: Add Daft into Iceberg documentation [iceberg]

2024-03-06 Thread via GitHub
nastra commented on code in PR #9836: URL: https://github.com/apache/iceberg/pull/9836#discussion_r1514231912 ## docs/docs/daft.md: ## @@ -0,0 +1,146 @@ +--- +title: "Daft" +--- + + +# Daft + +[Daft](www.getdaft.io) is a distributed query engine written in Python and Rust, two

Re: [PR] Docs: Add Daft into Iceberg documentation [iceberg]

2024-03-06 Thread via GitHub
nastra commented on code in PR #9836: URL: https://github.com/apache/iceberg/pull/9836#discussion_r1514232735 ## docs/docs/daft.md: ## @@ -0,0 +1,146 @@ +--- +title: "Daft" +--- + + +# Daft + +[Daft](www.getdaft.io) is a distributed query engine written in Python and Rust, two

Re: [I] Optimize `plan_files` with filter in case whe it is fully evaluated on Iceberg metadata [iceberg-python]

2024-03-06 Thread via GitHub
Fokko commented on issue #491: URL: https://github.com/apache/iceberg-python/issues/491#issuecomment-1980593421 Hey @srilman Thanks for reaching out here. I'm aware of the potential optimization, but most query engines don't optimize to that level. I'm very curious about the PR. Feel free t

Re: [PR] Make optional oauth configurable [iceberg-python]

2024-03-06 Thread via GitHub
Fokko commented on code in PR #486: URL: https://github.com/apache/iceberg-python/pull/486#discussion_r1514258300 ## pyiceberg/catalog/rest.py: ## @@ -289,12 +291,25 @@ def auth_url(self) -> str: else: return self.url(Endpoints.get_token, prefixed=False)

Re: [PR] Flink:backport PR to 1.17 #9606 : Supports specifying comment for iceberg fields in create table and addcolumn syntax using flinksql [iceberg]

2024-03-06 Thread via GitHub
pvary commented on PR #9868: URL: https://github.com/apache/iceberg/pull/9868#issuecomment-1980612048 Thanks @huyuanfeng2018 and @stevenzwu for taking care of this! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

[PR] Docs: Add DDL docs for Views [iceberg]

2024-03-06 Thread via GitHub
nastra opened a new pull request, #9878: URL: https://github.com/apache/iceberg/pull/9878 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Re: [I] Support deletion in Apache Flink [iceberg]

2024-03-06 Thread via GitHub
pvary commented on issue #8718: URL: https://github.com/apache/iceberg/issues/8718#issuecomment-1980627773 If your input stream contains `-D` records, then they will be deleted from the table. Like: ```java Row row = Row.withPositions(RowKind.DELETE, 3); row.setField(

Re: [PR] Use SupportsPrefixOperations for Remove OrphanFile Procedure [iceberg]

2024-03-06 Thread via GitHub
lokeshrdy commented on PR #7914: URL: https://github.com/apache/iceberg/pull/7914#issuecomment-1980642069 same issue here . let me know if anyone solved this with latest version? @carlosescura @domonkosbalogh-seon @rahil-c -- This is an automated message from the Apache Git Service. To r

Re: [PR] Nessie: Gracefully handle empty namespace lookup [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on code in PR #9877: URL: https://github.com/apache/iceberg/pull/9877#discussion_r1514299928 ## nessie/src/test/java/org/apache/iceberg/nessie/TestNamespace.java: ## @@ -132,6 +133,15 @@ public void testSettingProperties() { .hasMessage("Namespace

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-06 Thread via GitHub
pvary commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1514300878 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -269,6 +284,160 @@ public void renameTable(TableIdentifier from, TableIdentifier original

Re: [PR] Docs: Add DDL docs for Views [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on code in PR #9878: URL: https://github.com/apache/iceberg/pull/9878#discussion_r1514308982 ## docs/docs/spark-ddl.md: ## @@ -566,3 +566,85 @@ Tags can be removed via the `DROP TAG` sql ```sql ALTER TABLE prod.db.sample DROP TAG `historical-tag` ``` +

Re: [I] Library public api isolation and import decoupling [iceberg-python]

2024-03-06 Thread via GitHub
ndrluis commented on issue #499: URL: https://github.com/apache/iceberg-python/issues/499#issuecomment-1980691856 @HonahX Yes! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[I] JSON (De)serialization of the commit-metrics is broken. [iceberg]

2024-03-06 Thread via GitHub
snazy opened a new issue, #9879: URL: https://github.com/apache/iceberg/issues/9879 ### Apache Iceberg version main (development) ### Query engine None ### Please describe the bug 🐞 `CommitMetricsResult` defines the attributes `addedFilesSizeInBytes` + `rem

Re: [I] JSON (De)serialization of the commit-metrics is broken. [iceberg]

2024-03-06 Thread via GitHub
snazy commented on issue #9879: URL: https://github.com/apache/iceberg/issues/9879#issuecomment-1980697986 Meh - my (local) fault. All good. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [I] JSON (De)serialization of the commit-metrics is broken. [iceberg]

2024-03-06 Thread via GitHub
snazy closed issue #9879: JSON (De)serialization of the commit-metrics is broken. URL: https://github.com/apache/iceberg/issues/9879 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Docs: Add DDL docs for Views [iceberg]

2024-03-06 Thread via GitHub
nastra commented on code in PR #9878: URL: https://github.com/apache/iceberg/pull/9878#discussion_r1514339962 ## docs/docs/spark-ddl.md: ## @@ -566,3 +566,85 @@ Tags can be removed via the `DROP TAG` sql ```sql ALTER TABLE prod.db.sample DROP TAG `historical-tag` ``` + +### I

Re: [PR] Docs: Add DDL docs for Views [iceberg]

2024-03-06 Thread via GitHub
nastra commented on code in PR #9878: URL: https://github.com/apache/iceberg/pull/9878#discussion_r1514341452 ## docs/docs/spark-ddl.md: ## @@ -566,3 +566,85 @@ Tags can be removed via the `DROP TAG` sql ```sql ALTER TABLE prod.db.sample DROP TAG `historical-tag` ``` + +### I

Re: [PR] Open-api: update prefix param description [iceberg]

2024-03-06 Thread via GitHub
nastra commented on code in PR #9870: URL: https://github.com/apache/iceberg/pull/9870#discussion_r1514349522 ## open-api/rest-catalog-open-api.yaml: ## @@ -1444,7 +1444,7 @@ components: schema: type: string required: true - description: An optional p

Re: [PR] Open-api: update prefix param description [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on code in PR #9870: URL: https://github.com/apache/iceberg/pull/9870#discussion_r1514366199 ## open-api/rest-catalog-open-api.yaml: ## @@ -1444,7 +1444,7 @@ components: schema: type: string required: true - description: An opti

Re: [I] Support S3 Access Points with Access Point to Bucket mapping [iceberg-python]

2024-03-06 Thread via GitHub
JGynther commented on issue #452: URL: https://github.com/apache/iceberg-python/issues/452#issuecomment-1980790119 Testing a very simple wrapper like: ```Python from pyarrow.fs import S3FileSystem class WrappedS3FileSystem(S3FileSystem): def __init__(self, bucket_over

[I] About /v1/config REST API endpoint [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat opened a new issue, #9880: URL: https://github.com/apache/iceberg/issues/9880 ### Query engine NA ### Question I started going through the REST API spec. I couldn't understand about optional `warehouse` parameter in `/v1/config` endpoint. The use

Re: [I] About /v1/config REST API endpoint [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on issue #9880: URL: https://github.com/apache/iceberg/issues/9880#issuecomment-1980862061 maybe cc: @nastra, @danielcweeks, @rdblue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] Nessie: Gracefully handle empty namespace lookup [iceberg]

2024-03-06 Thread via GitHub
adutra commented on code in PR #9877: URL: https://github.com/apache/iceberg/pull/9877#discussion_r1514477117 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -308,6 +308,10 @@ public List listNamespaces(Namespace namespace) throws NoSuchNamespa

[I] Spark query pushdown error in SQL query optimization phase [iceberg]

2024-03-06 Thread via GitHub
burriad opened a new issue, #9881: URL: https://github.com/apache/iceberg/issues/9881 ### Apache Iceberg version 1.4.3 (latest release) ### Query engine Spark ### Please describe the bug 🐞 When upgrading from Iceberg version 1.3.1 to 1.4.3 we get the followi

Re: [PR] Nessie: Gracefully handle empty namespace lookup [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on code in PR #9877: URL: https://github.com/apache/iceberg/pull/9877#discussion_r1514489991 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -308,6 +308,10 @@ public List listNamespaces(Namespace namespace) throws NoSuchN

Re: [PR] Nessie: Gracefully handle empty namespace lookup [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on code in PR #9877: URL: https://github.com/apache/iceberg/pull/9877#discussion_r1514493828 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -380,6 +389,10 @@ public boolean removeProperties(Namespace namespace, Set prope

Re: [PR] Nessie: Gracefully handle empty namespace lookup [iceberg]

2024-03-06 Thread via GitHub
nastra commented on code in PR #9877: URL: https://github.com/apache/iceberg/pull/9877#discussion_r1514502547 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -380,6 +389,10 @@ public boolean removeProperties(Namespace namespace, Set properties)

Re: [PR] Nessie: Gracefully handle empty namespace lookup [iceberg]

2024-03-06 Thread via GitHub
nastra commented on code in PR #9877: URL: https://github.com/apache/iceberg/pull/9877#discussion_r1514509086 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java: ## @@ -308,6 +308,10 @@ public List listNamespaces(Namespace namespace) throws NoSuchNamespa

Re: [I] Spark query pushdown error in SQL query optimization phase [iceberg]

2024-03-06 Thread via GitHub
RussellSpitzer commented on issue #9881: URL: https://github.com/apache/iceberg/issues/9881#issuecomment-1980957406 Could you share the query as well as the table schema? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[PR] revert apache 9726 [iceberg]

2024-03-06 Thread via GitHub
artyrian opened a new pull request, #9882: URL: https://github.com/apache/iceberg/pull/9882 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

Re: [PR] revert apache 9726 [iceberg]

2024-03-06 Thread via GitHub
nastra commented on PR #9882: URL: https://github.com/apache/iceberg/pull/9882#issuecomment-1981000150 @artyrian can you please add some context why this revert is necessary? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
jbonofre opened a new pull request, #9884: URL: https://github.com/apache/iceberg/pull/9884 This PR adds a way to load custom schemes (in addition of the default ones) in `ResolvingFileIO` using properties prefixed with `resolving-io.schemes.`. This closes #9883 -- This is an

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
adutra commented on code in PR #9884: URL: https://github.com/apache/iceberg/pull/9884#discussion_r1514756075 ## core/src/main/java/org/apache/iceberg/io/ResolvingFileIO.java: ## @@ -53,20 +53,22 @@ public class ResolvingFileIO implements HadoopConfigurable, DelegateFileIO {

Re: [PR] Migrate Read in Core to JUnit5 [iceberg]

2024-03-06 Thread via GitHub
tomtongue commented on code in PR #9849: URL: https://github.com/apache/iceberg/pull/9849#discussion_r1514762022 ## core/src/test/java/org/apache/iceberg/TestMetadataTableScans.java: ## @@ -1357,27 +1344,27 @@ private void testPositionDeletesBaseTableFilter(boolean transactiona

Re: [PR] Migrate Read in Core to JUnit5 [iceberg]

2024-03-06 Thread via GitHub
tomtongue commented on code in PR #9849: URL: https://github.com/apache/iceberg/pull/9849#discussion_r1514762594 ## core/src/test/java/org/apache/iceberg/TestMetadataTableScans.java: ## @@ -1357,27 +1344,27 @@ private void testPositionDeletesBaseTableFilter(boolean transactiona

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on code in PR #9884: URL: https://github.com/apache/iceberg/pull/9884#discussion_r1514759756 ## core/src/main/java/org/apache/iceberg/io/ResolvingFileIO.java: ## @@ -122,6 +124,15 @@ public void initialize(Map newProperties) { close(); // close and di

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
jbonofre commented on PR #9884: URL: https://github.com/apache/iceberg/pull/9884#issuecomment-1981255063 @ajantha-bhat or also new IO implementations, interesting in Europe 😄 (like wasabi, leviia, ovh store, scaleway store, ...) -- This is an automated message from the Apache Git Service

Re: [PR] Make optional oauth configurable [iceberg-python]

2024-03-06 Thread via GitHub
himadripal commented on PR #486: URL: https://github.com/apache/iceberg-python/pull/486#issuecomment-1981317217 @flyrain and @Fokko thank you for reviewing this. Please take a look again when you can. -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on PR #9884: URL: https://github.com/apache/iceberg/pull/9884#issuecomment-1981329539 Testcase failed, probably because of `schemeToFileIo.clear(); ` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
jbonofre commented on PR #9884: URL: https://github.com/apache/iceberg/pull/9884#issuecomment-1981347201 @ajantha-bhat I'm fixing yeah. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] Feature make oauth `audience` configurable [iceberg]

2024-03-06 Thread via GitHub
himadripal commented on code in PR #9839: URL: https://github.com/apache/iceberg/pull/9839#discussion_r1514845097 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -178,15 +178,18 @@ public void initialize(String name, Map unresolved) { ConfigRes

Re: [PR] Feature make oauth `audience` configurable [iceberg]

2024-03-06 Thread via GitHub
himadripal commented on code in PR #9839: URL: https://github.com/apache/iceberg/pull/9839#discussion_r1514845097 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -178,15 +178,18 @@ public void initialize(String name, Map unresolved) { ConfigRes

Re: [PR] Migrate Read in Core to JUnit5 [iceberg]

2024-03-06 Thread via GitHub
tomtongue commented on PR #9849: URL: https://github.com/apache/iceberg/pull/9849#issuecomment-1981376570 @nastra Update with your comments. When you have time, could you review the new commit? -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on code in PR #9884: URL: https://github.com/apache/iceberg/pull/9884#discussion_r1514871547 ## core/src/main/java/org/apache/iceberg/io/ResolvingFileIO.java: ## @@ -219,7 +230,7 @@ DelegateFileIO io(String location) { @VisibleForTesting String imp

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on code in PR #9884: URL: https://github.com/apache/iceberg/pull/9884#discussion_r1514874407 ## core/src/main/java/org/apache/iceberg/io/ResolvingFileIO.java: ## @@ -219,7 +230,7 @@ DelegateFileIO io(String location) { @VisibleForTesting String imp

Re: [PR] Migrate Read in Core to JUnit5 [iceberg]

2024-03-06 Thread via GitHub
nastra commented on code in PR #9849: URL: https://github.com/apache/iceberg/pull/9849#discussion_r1514880737 ## core/src/test/java/org/apache/iceberg/TestFindFiles.java: ## @@ -184,33 +181,33 @@ public void testCaseSensitivity() { .withMetadataMatching(Expressions.

Re: [PR] Spark: Adding simple custom partition sort order option to RewriteManifests Spark Action [iceberg]

2024-03-06 Thread via GitHub
zachdisc commented on PR #9731: URL: https://github.com/apache/iceberg/pull/9731#issuecomment-1981412777 R4 Renamed `sort` to `clusterBy` - more honest and aligns with the core API as well. Have not made progress on using the custom UDF for both standard partition column ord

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on code in PR #9884: URL: https://github.com/apache/iceberg/pull/9884#discussion_r1514921889 ## core/src/main/java/org/apache/iceberg/io/ResolvingFileIO.java: ## @@ -122,11 +124,20 @@ public void initialize(Map newProperties) { close(); // close and

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
ajantha-bhat commented on code in PR #9884: URL: https://github.com/apache/iceberg/pull/9884#discussion_r1514921889 ## core/src/main/java/org/apache/iceberg/io/ResolvingFileIO.java: ## @@ -122,11 +124,20 @@ public void initialize(Map newProperties) { close(); // close and

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
jbonofre commented on code in PR #9884: URL: https://github.com/apache/iceberg/pull/9884#discussion_r1514931349 ## core/src/main/java/org/apache/iceberg/io/ResolvingFileIO.java: ## @@ -219,7 +230,7 @@ DelegateFileIO io(String location) { @VisibleForTesting String implFro

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-06 Thread via GitHub
jbonofre commented on code in PR #9884: URL: https://github.com/apache/iceberg/pull/9884#discussion_r1514932065 ## core/src/main/java/org/apache/iceberg/io/ResolvingFileIO.java: ## @@ -122,11 +124,20 @@ public void initialize(Map newProperties) { close(); // close and disc

Re: [PR] Feature make oauth `audience` configurable [iceberg]

2024-03-06 Thread via GitHub
himadripal commented on code in PR #9839: URL: https://github.com/apache/iceberg/pull/9839#discussion_r1514955976 ## aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java: ## @@ -207,7 +212,13 @@ private AuthSession authSession() { t

Re: [PR] Make optional oauth configurable [iceberg-python]

2024-03-06 Thread via GitHub
flyrain merged PR #486: URL: https://github.com/apache/iceberg-python/pull/486 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [I] Make the OAuth2 request audience configurable [iceberg-python]

2024-03-06 Thread via GitHub
flyrain commented on issue #479: URL: https://github.com/apache/iceberg-python/issues/479#issuecomment-1981523086 This is fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [I] Make the OAuth2 request audience configurable [iceberg-python]

2024-03-06 Thread via GitHub
flyrain closed issue #479: Make the OAuth2 request audience configurable URL: https://github.com/apache/iceberg-python/issues/479 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[I] Can't import pyiceberg.catalog [iceberg-python]

2024-03-06 Thread via GitHub
leonidmakarovsky opened a new issue, #500: URL: https://github.com/apache/iceberg-python/issues/500 ### Apache Iceberg version 0.6.0 (latest release) ### Please describe the bug 🐞 Running on Mac OS Monterey 12.6.7, Spyder IDE (Python 3.9.7 64-bit | Qt 5.9.7 | PyQt5 5.9.2

Re: [PR] Feature make oauth `audience` configurable [iceberg]

2024-03-06 Thread via GitHub
himadripal commented on code in PR #9839: URL: https://github.com/apache/iceberg/pull/9839#discussion_r1514845097 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -178,15 +178,18 @@ public void initialize(String name, Map unresolved) { ConfigRes

Re: [PR] Feature make oauth `audience` configurable [iceberg]

2024-03-06 Thread via GitHub
himadripal commented on PR #9839: URL: https://github.com/apache/iceberg/pull/9839#issuecomment-1981669062 Does all of these optional parameters need to be added in the response as well? @flyrain -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] Docs: Add Daft into Iceberg documentation [iceberg]

2024-03-06 Thread via GitHub
jaychia commented on code in PR #9836: URL: https://github.com/apache/iceberg/pull/9836#discussion_r1515138488 ## docs/docs/daft.md: ## @@ -0,0 +1,146 @@ +--- +title: "Daft" +--- + + +# Daft + +[Daft](www.getdaft.io) is a distributed query engine written in Python and Rust, two

Re: [PR] Docs: Add Daft into Iceberg documentation [iceberg]

2024-03-06 Thread via GitHub
jaychia commented on code in PR #9836: URL: https://github.com/apache/iceberg/pull/9836#discussion_r1515139832 ## docs/docs/daft.md: ## @@ -0,0 +1,146 @@ +--- +title: "Daft" +--- + + +# Daft + +[Daft](www.getdaft.io) is a distributed query engine written in Python and Rust, two

Re: [PR] Docs: Add Daft into Iceberg documentation [iceberg]

2024-03-06 Thread via GitHub
jaychia commented on code in PR #9836: URL: https://github.com/apache/iceberg/pull/9836#discussion_r1515142038 ## docs/docs/daft.md: ## @@ -0,0 +1,146 @@ +--- +title: "Daft" +--- + + +# Daft Review Comment: I was able to successfully build the site with `mkdocs serve`! ``

[PR] Core: Mark 502 and 504 failures as retryable to the exponential retry strategy [iceberg]

2024-03-06 Thread via GitHub
amogh-jahagirdar opened a new pull request, #9885: URL: https://github.com/apache/iceberg/pull/9885 Bad gateway or gateway timeouts should be able to be safely retried. A bad gateway could be caused by a load balancer or proxy between the client and the actual server, when there's some netw

Re: [I] Can't import pyiceberg.catalog [iceberg-python]

2024-03-06 Thread via GitHub
kevinjqliu commented on issue #500: URL: https://github.com/apache/iceberg-python/issues/500#issuecomment-1981880634 seems like an issue related to `pydantic` library. What version of it do you have installed? Here's my env setup using `make install`: ``` (venv) ➜ iceberg-pyth

Re: [I] Can't import pyiceberg.catalog [iceberg-python]

2024-03-06 Thread via GitHub
leonidmakarovsky commented on issue #500: URL: https://github.com/apache/iceberg-python/issues/500#issuecomment-1981884781 My python version is 3.9.7. My colleague told me that pyiceberg works with 3.9.11 and later. It didn't work with 3.9.7 in his env as well. I'm trying to upgrade Python

Re: [PR] Bin-pack Writes Operation into multiple parquet files, and parallelize writing `WriteTask`s [iceberg-python]

2024-03-06 Thread via GitHub
kevinjqliu commented on code in PR #444: URL: https://github.com/apache/iceberg-python/pull/444#discussion_r1515227170 ## tests/integration/test_writes.py: ## @@ -424,6 +424,30 @@ def test_data_files(spark: SparkSession, session_catalog: Catalog, arrow_table_w assert [row.

Re: [I] [feature request] easier API to set table properties [iceberg-python]

2024-03-06 Thread via GitHub
kevinjqliu commented on issue #502: URL: https://github.com/apache/iceberg-python/issues/502#issuecomment-1981976531 Another workaround can be to use ``` with tbl.transaction() as transaction: transaction._apply((SetPropertiesUpdate(updates=properties),)) ``` -- This i

Re: [PR] Kafka Connect: Record converters [iceberg]

2024-03-06 Thread via GitHub
fqaiser94 commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1515260098 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java: ## @@ -0,0 +1,508 @@ +/* + * Licensed to the Apache Software Founda

Re: [I] A logo for iceberg rust! [iceberg-rust]

2024-03-06 Thread via GitHub
caldempsey commented on issue #216: URL: https://github.com/apache/iceberg-rust/issues/216#issuecomment-1982002725 I'd prefer a letter I with the RUST gears surrounding it ⚙️ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [I] A logo for iceberg rust! [iceberg-rust]

2024-03-06 Thread via GitHub
caldempsey commented on issue #216: URL: https://github.com/apache/iceberg-rust/issues/216#issuecomment-1982004560 ![image](https://github.com/apache/iceberg-rust/assets/8885269/ea2a8b46-15fb-4aba-9a06-49c58bc0b821) -- This is an automated message from the Apache Git Service. To respo

Re: [PR] Bin-pack Writes Operation into multiple parquet files, and parallelize writing `WriteTask`s [iceberg-python]

2024-03-06 Thread via GitHub
kevinjqliu commented on PR #444: URL: https://github.com/apache/iceberg-python/pull/444#issuecomment-1982045743 thanks for the review @HonahX. I've rebased off `main` and addressed your comments. I also added more tests after figuring out the fix for #482 -- This is an automated m

Re: [PR] Disable Spark Catalog caching for integration tests [iceberg-python]

2024-03-06 Thread via GitHub
amogh-jahagirdar commented on PR #501: URL: https://github.com/apache/iceberg-python/pull/501#issuecomment-1982063186 Thanks @kevinjqliu I think this change makes sense. I don't think there's ever a reason on the Python side where we want to have the spark caching enabled. On the Iceberg Ja

Re: [PR] Disable Spark Catalog caching for integration tests [iceberg-python]

2024-03-06 Thread via GitHub
amogh-jahagirdar commented on code in PR #501: URL: https://github.com/apache/iceberg-python/pull/501#discussion_r1515300598 ## tests/integration/test_writes.py: ## @@ -355,6 +355,26 @@ def test_data_files(spark: SparkSession, session_catalog: Catalog, arrow_table_w assert

Re: [PR] Disable Spark Catalog caching for integration tests [iceberg-python]

2024-03-06 Thread via GitHub
kevinjqliu commented on code in PR #501: URL: https://github.com/apache/iceberg-python/pull/501#discussion_r1515301588 ## tests/integration/test_writes.py: ## @@ -355,6 +355,26 @@ def test_data_files(spark: SparkSession, session_catalog: Catalog, arrow_table_w assert [row.

Re: [I] The database deletion operation was performed, but the hdfs file directory was not deleted. [iceberg]

2024-03-06 Thread via GitHub
github-actions[bot] commented on issue #1886: URL: https://github.com/apache/iceberg/issues/1886#issuecomment-1982079661 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Flink Table Without Timestamp Cannot Query By Spark [iceberg]

2024-03-06 Thread via GitHub
github-actions[bot] commented on issue #1914: URL: https://github.com/apache/iceberg/issues/1914#issuecomment-1982079679 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Move field into place when adding during schema evolution [iceberg]

2024-03-06 Thread via GitHub
hiloboy0119 commented on PR #8409: URL: https://github.com/apache/iceberg/pull/8409#issuecomment-1982085238 @amogh-jahagirdar sorry for the long delay. Finally had time to write tests for the various edge cases and fix a few things. The examples of schema evolution are all shown in t

Re: [PR] Flink: implement range partitioner for map data statistics [iceberg]

2024-03-06 Thread via GitHub
stevenzwu commented on code in PR #9321: URL: https://github.com/apache/iceberg/pull/9321#discussion_r1515180358 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/MapRangePartitioner.java: ## @@ -0,0 +1,368 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Disable Spark Catalog caching for integration tests [iceberg-python]

2024-03-06 Thread via GitHub
syun64 commented on PR #501: URL: https://github.com/apache/iceberg-python/pull/501#issuecomment-1982096927 Great idea @kevinjqliu ! Thanks for adding this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] [feature request] easier API to set table properties [iceberg-python]

2024-03-06 Thread via GitHub
syun64 commented on issue #502: URL: https://github.com/apache/iceberg-python/issues/502#issuecomment-1982120843 I think this is a great idea @kevinjqliu . Most of the table properties have `.` in the key anyways, and hence we can't pass them as kwargs anyways. Changing the input parameter

Re: [PR] Disable Spark Catalog caching for integration tests [iceberg-python]

2024-03-06 Thread via GitHub
amogh-jahagirdar commented on PR #501: URL: https://github.com/apache/iceberg-python/pull/501#issuecomment-1982123757 Sweet, thanks @kevinjqliu! I'm going to go ahead and merge this now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [I] Spark <> Iceberg bug integration test [iceberg-python]

2024-03-06 Thread via GitHub
amogh-jahagirdar closed issue #482: Spark <> Iceberg bug integration test URL: https://github.com/apache/iceberg-python/issues/482 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] Disable Spark Catalog caching for integration tests [iceberg-python]

2024-03-06 Thread via GitHub
amogh-jahagirdar merged PR #501: URL: https://github.com/apache/iceberg-python/pull/501 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr.

Re: [PR] revert apache 9726 [iceberg]

2024-03-06 Thread via GitHub
artyrian closed pull request #9882: revert apache 9726 URL: https://github.com/apache/iceberg/pull/9882 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[PR] fix minor version for striclty libs versions [iceberg]

2024-03-06 Thread via GitHub
artyrian opened a new pull request, #9886: URL: https://github.com/apache/iceberg/pull/9886 https://github.com/apache/iceberg/pull/9484 and there is also a confilict with jackson 2.15.4 and spark 3.5 on gradle 8.4: `java.lang.IllegalArgumentException: Unsupported class file maj

  1   2   >