Re: [PR] Spark 3.5: Add a procedure to remove corrupt snapshots [iceberg]

2024-02-05 Thread via GitHub
nastra commented on code in PR #9645: URL: https://github.com/apache/iceberg/pull/9645#discussion_r1477793542 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/RemoveSnapshotsProcedure.java: ## @@ -0,0 +1,110 @@ +/* + * Licensed to the Apache Software Foundat

Re: [PR] Kafka Connect: Record converters and delta writers [iceberg]

2024-02-05 Thread via GitHub
nastra commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1477806683 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/IcebergWriter.java: ## @@ -52,20 +51,23 @@ public IcebergWriter(Table table, String tableNam

Re: [I] Migrate RESTCatalogServlet to use jakarta.* package for Spring boot 3 [iceberg]

2024-02-05 Thread via GitHub
nastra commented on issue #9626: URL: https://github.com/apache/iceberg/issues/9626#issuecomment-1926470051 I think that would require updating https://eclipse.dev/jetty/ to a newer version. However, newer Jetty versions require JDK 11+ but Iceberg is on JDK8 due to projects like Hive --

Re: [PR] Core: Add alias for glue, nessie, snowflake, jdbc catalog [iceberg]

2024-02-05 Thread via GitHub
ajantha-bhat closed pull request #8641: Core: Add alias for glue, nessie, snowflake, jdbc catalog URL: https://github.com/apache/iceberg/pull/8641 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Core: Add alias for glue, nessie, snowflake, jdbc catalog [iceberg]

2024-02-05 Thread via GitHub
ajantha-bhat commented on PR #8641: URL: https://github.com/apache/iceberg/pull/8641#issuecomment-1926475947 since there is no response, I handled the testcase and doc updates and opened a new PR with @zhaomin1423 as coauthor closing this in favour of https://github.com/apache/ice

Re: [PR] Docs: Update Nessie URI to API v2 [iceberg]

2024-02-05 Thread via GitHub
ajantha-bhat commented on PR #9648: URL: https://github.com/apache/iceberg/pull/9648#issuecomment-1926482528 cc: @adutra, @dimas-b -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[PR] Docs: Fix listing of catalog implementations [iceberg]

2024-02-05 Thread via GitHub
nastra opened a new pull request, #9649: URL: https://github.com/apache/iceberg/pull/9649 The listing of catalog implementations is currently wrongly displayed as can be seen below ![image](https://github.com/apache/iceberg/assets/271029/d5643a5e-485b-4716-a3f0-efcb8fe66cc2) -- Th

Re: [PR] Docs: Update Nessie URI to API v2 [iceberg]

2024-02-05 Thread via GitHub
nastra merged PR #9648: URL: https://github.com/apache/iceberg/pull/9648 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Core: Add catalog type for glue,jdbc,nessie [iceberg]

2024-02-05 Thread via GitHub
nastra commented on code in PR #9647: URL: https://github.com/apache/iceberg/pull/9647#discussion_r1477878848 ## docs/docs/aws.md: ## @@ -428,7 +428,7 @@ For example, to add S3 delete tags with Spark 3.3, you can start the Spark SQL s ``` sh spark-sql --conf spark.sql.catalo

[I] Cannot use MERGE INTO query on Iceberg table. Getting `java.lang.IllegalArgumentException: Comparison method violates its general contract!` error. [iceberg]

2024-02-05 Thread via GitHub
Mourya1319 opened a new issue, #9650: URL: https://github.com/apache/iceberg/issues/9650 ### Apache Iceberg version 1.4.2 ### Query engine Spark ### Please describe the bug 🐞 I have an Iceberg Table to which I have used MERGE INTO query to update specific r

[I] Enabling schema evolution feature using spark configuration like we have in Delta Lake [iceberg]

2024-02-05 Thread via GitHub
aleenamg21-1 opened a new issue, #9651: URL: https://github.com/apache/iceberg/issues/9651 ### Feature Request / Improvement Currently schema evolution settings can be configured in Delta lake via both [spark write option](https://delta.io/blog/2023-02-08-delta-lake-schema-evolution/

Re: [PR] Spark 3.4: Fix CREATE OR REPLACE VIEW when view doesn't exist [iceberg]

2024-02-05 Thread via GitHub
Fokko merged PR #9646: URL: https://github.com/apache/iceberg/pull/9646 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Docs: Fix listing of catalog implementations [iceberg]

2024-02-05 Thread via GitHub
Fokko merged PR #9649: URL: https://github.com/apache/iceberg/pull/9649 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[PR] Label `site/` as documentation [iceberg]

2024-02-05 Thread via GitHub
Fokko opened a new pull request, #9652: URL: https://github.com/apache/iceberg/pull/9652 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail

Re: [PR] Use `write.parquet.compression-{codec,level}` [iceberg-python]

2024-02-05 Thread via GitHub
Fokko merged PR #358: URL: https://github.com/apache/iceberg-python/pull/358 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Bug Fix: Allow GlueCatalog to create table with TimestampzType [iceberg-python]

2024-02-05 Thread via GitHub
Fokko merged PR #366: URL: https://github.com/apache/iceberg-python/pull/366 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Label `site/` as documentation [iceberg]

2024-02-05 Thread via GitHub
nastra commented on PR #9652: URL: https://github.com/apache/iceberg/pull/9652#issuecomment-1926600320 It also looks like we're running all CI actions for stuff under site/docs, which we should probably also fix (not in this PR) -- This is an automated message from the Apache Git Service.

[I] Iceberg Partition writer converts partition column value to null [iceberg]

2024-02-05 Thread via GitHub
robinsinghstudios opened a new issue, #9653: URL: https://github.com/apache/iceberg/issues/9653 ### Query engine Iceberg Java API 1.4.3 ### Question For context, I am new to Java and might be missing something simple but, after being stuck on this issue for a long while

Re: [PR] docs: Document Parquet write options [iceberg-python]

2024-02-05 Thread via GitHub
nastra commented on code in PR #364: URL: https://github.com/apache/iceberg-python/pull/364#discussion_r1478028951 ## mkdocs/docs/configuration.md: ## @@ -46,7 +46,20 @@ The environment variable picked up by Iceberg starts with `PYICEBERG_` and then For example, `PYICEBERG_C

Re: [I] Cannot use MERGE INTO query on Iceberg table. Getting `java.lang.IllegalArgumentException: Comparison method violates its general contract!` error. [iceberg]

2024-02-05 Thread via GitHub
nastra closed issue #9650: Cannot use MERGE INTO query on Iceberg table. Getting `java.lang.IllegalArgumentException: Comparison method violates its general contract!` error. URL: https://github.com/apache/iceberg/issues/9650 -- This is an automated message from the Apache Git Service. To re

Re: [I] Cannot use MERGE INTO query on Iceberg table. Getting `java.lang.IllegalArgumentException: Comparison method violates its general contract!` error. [iceberg]

2024-02-05 Thread via GitHub
nastra commented on issue #9650: URL: https://github.com/apache/iceberg/issues/9650#issuecomment-1926769179 @Mourya1319 this sounds rather like an issue specific to EMR because `HeadTailBinPackingAMZN` does not exist in the OSS Iceberg codebase. I'd suggest to report this issue to EMR direc

Re: [PR] Core: Add catalog type for glue,jdbc,nessie [iceberg]

2024-02-05 Thread via GitHub
ajantha-bhat commented on code in PR #9647: URL: https://github.com/apache/iceberg/pull/9647#discussion_r1478042865 ## docs/docs/aws.md: ## @@ -428,7 +428,7 @@ For example, to add S3 delete tags with Spark 3.3, you can start the Spark SQL s ``` sh spark-sql --conf spark.sql.

Re: [PR] Label `site/` as documentation [iceberg]

2024-02-05 Thread via GitHub
nastra merged PR #9652: URL: https://github.com/apache/iceberg/pull/9652 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Spark 3.5: Add a procedure to remove corrupt snapshots [iceberg]

2024-02-05 Thread via GitHub
nastra commented on code in PR #9645: URL: https://github.com/apache/iceberg/pull/9645#discussion_r1477793542 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/RemoveSnapshotsProcedure.java: ## @@ -0,0 +1,110 @@ +/* + * Licensed to the Apache Software Foundat

Re: [PR] Spark 3.5: Add a procedure to remove corrupt snapshots [iceberg]

2024-02-05 Thread via GitHub
manuzhang commented on code in PR #9645: URL: https://github.com/apache/iceberg/pull/9645#discussion_r1478061696 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/RemoveSnapshotsProcedure.java: ## @@ -0,0 +1,110 @@ +/* + * Licensed to the Apache Software Foun

Re: [PR] Iceberg site fixes [iceberg]

2024-02-05 Thread via GitHub
bitsondatadev commented on code in PR #9642: URL: https://github.com/apache/iceberg/pull/9642#discussion_r1478097135 ## site/nav.yml: ## @@ -32,18 +32,18 @@ nav: - Talks: talks.md - Vendors: vendors.md - Project: -- Join: community.md +- Community: community.md

Re: [PR] Iceberg site fixes [iceberg]

2024-02-05 Thread via GitHub
bitsondatadev commented on code in PR #9642: URL: https://github.com/apache/iceberg/pull/9642#discussion_r1478097135 ## site/nav.yml: ## @@ -32,18 +32,18 @@ nav: - Talks: talks.md - Vendors: vendors.md - Project: -- Join: community.md +- Community: community.md

Re: [PR] Core: Add catalog type for glue,jdbc,nessie [iceberg]

2024-02-05 Thread via GitHub
ajantha-bhat commented on code in PR #9647: URL: https://github.com/apache/iceberg/pull/9647#discussion_r1478105179 ## docs/docs/aws.md: ## @@ -428,7 +428,7 @@ For example, to add S3 delete tags with Spark 3.3, you can start the Spark SQL s ``` sh spark-sql --conf spark.sql.

Re: [I] DOCS: Report CSS and styling issues on the new site. [iceberg]

2024-02-05 Thread via GitHub
bitsondatadev commented on issue #9643: URL: https://github.com/apache/iceberg/issues/9643#issuecomment-1926888633 @manuzhang could you try loading the site in a private browser. I think the settings on your sessioned browser is relying on old cached items from the old site. Try from

Re: [PR] docs: Document Parquet write options [iceberg-python]

2024-02-05 Thread via GitHub
Fokko commented on code in PR #364: URL: https://github.com/apache/iceberg-python/pull/364#discussion_r1478159369 ## mkdocs/docs/configuration.md: ## @@ -46,7 +46,20 @@ The environment variable picked up by Iceberg starts with `PYICEBERG_` and then For example, `PYICEBERG_CA

Re: [PR] Core: Add catalog type for glue,jdbc,nessie [iceberg]

2024-02-05 Thread via GitHub
nastra commented on code in PR #9647: URL: https://github.com/apache/iceberg/pull/9647#discussion_r1478181814 ## docs/docs/flink-configuration.md: ## @@ -37,7 +37,7 @@ The following properties can be set globally and are not limited to a specific c | Property

Re: [PR] Retry with new Access Token on 419 response [iceberg-python]

2024-02-05 Thread via GitHub
anupam-saini commented on code in PR #340: URL: https://github.com/apache/iceberg-python/pull/340#discussion_r1478222848 ## pyiceberg/catalog/rest.py: ## @@ -438,6 +444,16 @@ def _response_to_table(self, identifier_tuple: Tuple[str, ...], table_response: catalog=se

Re: [PR] Core: Add strictness flag to prevent loss of view representation when replacing a view [iceberg]

2024-02-05 Thread via GitHub
nastra commented on code in PR #9620: URL: https://github.com/apache/iceberg/pull/9620#discussion_r1478246464 ## core/src/main/java/org/apache/iceberg/rest/RESTViewOperations.java: ## @@ -59,6 +60,8 @@ public void commit(ViewMetadata base, ViewMetadata metadata) { // this i

Re: [PR] Core: Add strictness flag to prevent loss of view representation when replacing a view [iceberg]

2024-02-05 Thread via GitHub
nastra commented on code in PR #9620: URL: https://github.com/apache/iceberg/pull/9620#discussion_r1478247434 ## core/src/main/java/org/apache/iceberg/view/ViewProperties.java: ## @@ -26,6 +26,8 @@ public class ViewProperties { public static final String METADATA_COMPRESSION

[I] HMS lock timeout [iceberg]

2024-02-05 Thread via GitHub
ShubhamSharmaCSE opened a new issue, #9654: URL: https://github.com/apache/iceberg/issues/9654 ### Apache Iceberg version None ### Query engine Spark ### Please describe the bug 🐞 iceberg version: iceberg-spark-runtime-3.3_2.12-1.1.0-amzn-0.jar Spark

Re: [PR] Core: Add strictness flag to prevent loss of view representation when replacing a view [iceberg]

2024-02-05 Thread via GitHub
nastra commented on code in PR #9620: URL: https://github.com/apache/iceberg/pull/9620#discussion_r1478249143 ## core/src/main/java/org/apache/iceberg/rest/RESTViewOperations.java: ## @@ -59,6 +60,8 @@ public void commit(ViewMetadata base, ViewMetadata metadata) { // this i

Re: [I] HMS lock timeout [iceberg]

2024-02-05 Thread via GitHub
nastra commented on issue #9654: URL: https://github.com/apache/iceberg/issues/9654#issuecomment-1927024063 https://github.com/apache/iceberg/issues/2301 is a related issue to this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [I] HMS lock timeout [iceberg]

2024-02-05 Thread via GitHub
nastra commented on issue #9654: URL: https://github.com/apache/iceberg/issues/9654#issuecomment-1927028023 You might want to check if https://github.com/apache/iceberg/pull/6570 applies in your case. Note that this was shipped with Iceberg 1.3.0 -- This is an automated message from the A

Re: [PR] Retry with new Access Token on 419 response [iceberg-python]

2024-02-05 Thread via GitHub
anupam-saini commented on code in PR #340: URL: https://github.com/apache/iceberg-python/pull/340#discussion_r1478267337 ## pyiceberg/catalog/rest.py: ## @@ -472,6 +488,7 @@ def create_table( table_response = TableResponse(**response.json()) return self._respon

[I] Support setting a snapshot property in same commit as spark.sql [iceberg-python]

2024-02-05 Thread via GitHub
brianfromoregon opened a new issue, #368: URL: https://github.com/apache/iceberg-python/issues/368 ### Feature Request / Improvement This [cookbook](https://tabular.io/apache-iceberg-cookbook/data-engineering-incremental-processing/) has a java snippet to update a snapshot property a

Re: [PR] Core: Add catalog type for glue,jdbc,nessie [iceberg]

2024-02-05 Thread via GitHub
nastra merged PR #9647: URL: https://github.com/apache/iceberg/pull/9647 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[PR] Release: add instruction to update doap.rdf file as part of release process [iceberg]

2024-02-05 Thread via GitHub
jbonofre opened a new pull request, #9655: URL: https://github.com/apache/iceberg/pull/9655 This closes #9522 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [PR] Release: add instruction to update doap.rdf file as part of release process [iceberg]

2024-02-05 Thread via GitHub
jbonofre commented on PR #9655: URL: https://github.com/apache/iceberg/pull/9655#issuecomment-1927070185 @Fokko @ajantha-bhat can you guys take a look ? Thanks ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Release: add instruction to update doap.rdf file as part of release process [iceberg]

2024-02-05 Thread via GitHub
ajantha-bhat commented on code in PR #9655: URL: https://github.com/apache/iceberg/pull/9655#discussion_r1478319471 ## site/docs/how-to-release.md: ## @@ -308,6 +308,39 @@ Create a PR in the `iceberg` repo to make revapi run on the new release. For an - Create a PR in the `ice

Re: [PR] Release: add instruction to update doap.rdf file as part of release process [iceberg]

2024-02-05 Thread via GitHub
jbonofre commented on code in PR #9655: URL: https://github.com/apache/iceberg/pull/9655#discussion_r1478329855 ## site/docs/how-to-release.md: ## @@ -308,6 +308,39 @@ Create a PR in the `iceberg` repo to make revapi run on the new release. For an - Create a PR in the `iceberg

Re: [PR] Release: add instruction to update doap.rdf file as part of release process [iceberg]

2024-02-05 Thread via GitHub
jbonofre commented on code in PR #9655: URL: https://github.com/apache/iceberg/pull/9655#discussion_r1478330135 ## site/docs/how-to-release.md: ## @@ -308,6 +308,39 @@ Create a PR in the `iceberg` repo to make revapi run on the new release. For an - Create a PR in the `iceberg

Re: [PR] docs: Document Parquet write options [iceberg-python]

2024-02-05 Thread via GitHub
HonahX commented on code in PR #364: URL: https://github.com/apache/iceberg-python/pull/364#discussion_r1478336457 ## mkdocs/docs/configuration.md: ## @@ -46,7 +46,20 @@ The environment variable picked up by Iceberg starts with `PYICEBERG_` and then For example, `PYICEBERG_C

Re: [PR] Docs: Add newline so that subsection is correctly rendered [iceberg]

2024-02-05 Thread via GitHub
nastra commented on PR #9656: URL: https://github.com/apache/iceberg/pull/9656#issuecomment-1927189695 @bitsondatadev I think it would be good to go over the docs and see which pages/sections don't render properly. I've just found this because i was looking for something specific on Metadat

Re: [PR] Kafka Connect: Record converters and delta writers [iceberg]

2024-02-05 Thread via GitHub
bryanck commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1478399558 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/IcebergWriter.java: ## @@ -52,20 +51,23 @@ public IcebergWriter(Table table, String tableNa

Re: [PR] docs: Document Parquet write options [iceberg-python]

2024-02-05 Thread via GitHub
Fokko merged PR #364: URL: https://github.com/apache/iceberg-python/pull/364 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Docs: Add newline so that subsection is correctly rendered [iceberg]

2024-02-05 Thread via GitHub
bitsondatadev commented on PR #9656: URL: https://github.com/apache/iceberg/pull/9656#issuecomment-1927265427 Agreed, I'll make a pass at this some time this week! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-02-05 Thread via GitHub
nastra commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1478448616 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcViewOperations.java: ## @@ -0,0 +1,204 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mor

Re: [PR] partitioned write support [iceberg-python]

2024-02-05 Thread via GitHub
jqin61 commented on code in PR #353: URL: https://github.com/apache/iceberg-python/pull/353#discussion_r1478509127 ## pyiceberg/manifest.py: ## @@ -308,6 +308,7 @@ def data_file_with_partition(partition_type: StructType, format_version: Literal field_id=field.field

Re: [PR] Flink: backport #9547 to 1.17 and 1.16 for Adds the ability to read from a branch on the Flink Iceberg Source [iceberg]

2024-02-05 Thread via GitHub
pvary merged PR #9627: URL: https://github.com/apache/iceberg/pull/9627 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] AWS: Add S3 Access Grants Documentation [iceberg]

2024-02-05 Thread via GitHub
jackye1995 commented on PR #9590: URL: https://github.com/apache/iceberg/pull/9590#issuecomment-1927400479 Thanks for the work! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] AWS: Add S3 Access Grants Documentation [iceberg]

2024-02-05 Thread via GitHub
jackye1995 merged PR #9590: URL: https://github.com/apache/iceberg/pull/9590 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-02-05 Thread via GitHub
jbonofre commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1478540570 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcUtil.java: ## @@ -18,14 +18,116 @@ */ package org.apache.iceberg.jdbc; +import static org.assertj.core.api.A

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-02-05 Thread via GitHub
jbonofre commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1478542258 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -81,85 +90,103 @@ final class JdbcUtil { + TABLE_NAME + ")" + ")";

Re: [I] Consolidate FileIO [iceberg-python]

2024-02-05 Thread via GitHub
kevinjqliu commented on issue #310: URL: https://github.com/apache/iceberg-python/issues/310#issuecomment-1927505635 I see. I was under the assumption that PyArrow could completely replace fsspec. But it seems like there are a few use cases where we would prefer fsspec. > fsspec is

Re: [PR] Core: only trim slash when warehouse location is not root path [iceberg]

2024-02-05 Thread via GitHub
abmo-x commented on code in PR #9619: URL: https://github.com/apache/iceberg/pull/9619#discussion_r1478622304 ## core/src/test/java/org/apache/iceberg/util/TestLocationUtil.java: ## @@ -46,6 +46,21 @@ public void testStripTrailingSlash() { assertThat(LocationUtil.stripTrail

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-02-05 Thread via GitHub
jbonofre commented on PR #9008: URL: https://github.com/apache/iceberg/pull/9008#issuecomment-1927543271 I wonder why not using something similar to what we have for `decimal` with `(P,S)` for `timestamp` ? If we want to have "open precision" for `timestamp` we could imagine to have seco

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-02-05 Thread via GitHub
jbonofre commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1478650850 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -25,31 +25,36 @@ import java.util.Map; import java.util.Properties; import java.util.Set; +import

Re: [PR] Add Daft examples and code into PyIceberg docs and Table [iceberg-python]

2024-02-05 Thread via GitHub
jaychia commented on code in PR #355: URL: https://github.com/apache/iceberg-python/pull/355#discussion_r1478653195 ## pyproject.toml: ## @@ -105,6 +105,7 @@ pyarrow = ["pyarrow"] pandas = ["pandas", "pyarrow"] duckdb = ["duckdb", "pyarrow"] ray = ["ray", "pyarrow", "pandas"]

Re: [PR] Iceberg site fixes [iceberg]

2024-02-05 Thread via GitHub
amogh-jahagirdar merged PR #9642: URL: https://github.com/apache/iceberg/pull/9642 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Iceberg site fixes [iceberg]

2024-02-05 Thread via GitHub
amogh-jahagirdar commented on PR #9642: URL: https://github.com/apache/iceberg/pull/9642#issuecomment-1927652917 Thanks @bitsondatadev for fixing these, and @Fokko @nastra for the reviews! -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] Spark 3.5: Support executor cache locality [iceberg]

2024-02-05 Thread via GitHub
aokolnychyi commented on PR #9563: URL: https://github.com/apache/iceberg/pull/9563#issuecomment-1927759989 Thanks, @advancedxy @rdblue! I am going to test this with our RC on a cluster. I can't cover everything locally. I tested the initial prototype on a cluster and it worked well. --

Re: [PR] Spark 3.5: Support executor cache locality [iceberg]

2024-02-05 Thread via GitHub
aokolnychyi merged PR #9563: URL: https://github.com/apache/iceberg/pull/9563 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Core: Fix retry behavior for Jdbc Client [iceberg]

2024-02-05 Thread via GitHub
cccs-br commented on PR #7561: URL: https://github.com/apache/iceberg/pull/7561#issuecomment-1927761923 Since the JdbcCatalog provides the means to specify your own JdbcClientPool by providing a [client pool builder](https://github.com/apache/iceberg/blob/c4cb0fb9993d6743d81a232def6801ea7db

Re: [PR] partitioned write support [iceberg-python]

2024-02-05 Thread via GitHub
jqin61 commented on code in PR #353: URL: https://github.com/apache/iceberg-python/pull/353#discussion_r1478509127 ## pyiceberg/manifest.py: ## @@ -308,6 +308,7 @@ def data_file_with_partition(partition_type: StructType, format_version: Literal field_id=field.field

Re: [PR] Spark 3.4: Support executor cache locality [iceberg]

2024-02-05 Thread via GitHub
aokolnychyi commented on PR #9658: URL: https://github.com/apache/iceberg/pull/9658#issuecomment-1927844378 @nastra @amogh-jahagirdar @szehon-ho @ajantha-bhat, could you take a look at this cherry-pick? -- This is an automated message from the Apache Git Service. To respond to the message

[I] Use self. default_spec_id in else clause [iceberg-rust]

2024-02-05 Thread via GitHub
odysa opened a new issue, #189: URL: https://github.com/apache/iceberg-rust/issues/189 Should it be `self.partition_spec_by_id(self.default_spec_id)` in the else clause? https://github.com/apache/iceberg-rust/blob/09765db611a65a21b88e839d781780c75924e560/crates/iceberg/src/spec/table_met

Re: [PR] Retry with new Access Token on 419 response [iceberg-python]

2024-02-05 Thread via GitHub
Fokko commented on code in PR #340: URL: https://github.com/apache/iceberg-python/pull/340#discussion_r1478822071 ## pyiceberg/catalog/rest.py: ## @@ -118,6 +119,19 @@ class Endpoints: NAMESPACE_SEPARATOR = b"\x1F".decode(UTF8) +def _retry_hook(retry_state: RetryCallState)

Re: [PR] Improve error message in case of a mismatch [iceberg-python]

2024-02-05 Thread via GitHub
Fokko commented on PR #352: URL: https://github.com/apache/iceberg-python/pull/352#issuecomment-1927962033 @HonahX Thanks 🙏 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-05 Thread via GitHub
rahil-c opened a new pull request, #9660: URL: https://github.com/apache/iceberg/pull/9660 Dev List discussion thread around adding support for pagination in list namespaces, tables, and views: https://lists.apache.org/thread/lql05h02qtp8mgq74ovhb0ndd76ck4f3 Credit to @emkornfield f

Re: [PR] Support merge manifests on writes [iceberg-python]

2024-02-05 Thread via GitHub
Fokko commented on code in PR #363: URL: https://github.com/apache/iceberg-python/pull/363#discussion_r1478849196 ## pyiceberg/table/__init__.py: ## @@ -2411,11 +2428,29 @@ def _fetch_existing_manifests() -> List[ManifestFile]: executor = ExecutorFactory.get_or_create

Re: [I] Support setting a snapshot property in same commit as spark.sql [iceberg-python]

2024-02-05 Thread via GitHub
Fokko commented on issue #368: URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-1928020308 Thanks for raising this @brianfromoregon! I think it would be a great addition. We need to extend the `.append` and `.overwrite` API and allow passing in a map. And then it

Re: [I] Implement Centralized Management of Table Properties [iceberg-python]

2024-02-05 Thread via GitHub
Fokko commented on issue #365: URL: https://github.com/apache/iceberg-python/issues/365#issuecomment-1928022429 I think this is a great idea @HonahX. I try to avoid creating a lot of new files, since imports in Python are slow. What do you think of adding a class `TableProperties` to `table

Re: [I] Support reading and writing snapshot properties [iceberg-python]

2024-02-05 Thread via GitHub
Fokko commented on issue #367: URL: https://github.com/apache/iceberg-python/issues/367#issuecomment-1928025807 Hey @brianfromoregon, I agree. For writing, I left a comment in https://github.com/apache/iceberg-python/issues/368 For reading, I think we're missing some documentation since it

Re: [I] Use latest Parquet version for writing [iceberg-python]

2024-02-05 Thread via GitHub
Fokko commented on issue #359: URL: https://github.com/apache/iceberg-python/issues/359#issuecomment-1928038141 Hey @jonashaag Thanks for raising this. I think I made a mistake here. I thought it was referring to the data-page-version, but that one is set to `1.0`. I think we should bump th

Re: [I] Distributed writes in the same iceberg transaction [iceberg-python]

2024-02-05 Thread via GitHub
Fokko commented on issue #357: URL: https://github.com/apache/iceberg-python/issues/357#issuecomment-1928041180 Hey @rahij This is something that we're planning on supporting. I know that the folks at Daft are already working on this. Out of curiosity, how much data are we talking about, an

Re: [I] Use latest Parquet version for writing [iceberg-python]

2024-02-05 Thread via GitHub
jonashaag commented on issue #359: URL: https://github.com/apache/iceberg-python/issues/359#issuecomment-1928060787 Shall we then just stick to the PyArrow default? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] feat(catalog): add initial rest catalog impl [iceberg-go]

2024-02-05 Thread via GitHub
zeroshade commented on code in PR #58: URL: https://github.com/apache/iceberg-go/pull/58#discussion_r1478892511 ## catalog/catalog.go: ## @@ -47,19 +52,136 @@ func WithAwsConfig(cfg aws.Config) Option { } } +func WithCredential(cred string) Option { Review Comment:

Re: [I] `pyiceberg.io.pyarrow.write_file` does not take into account compression settings [iceberg-python]

2024-02-05 Thread via GitHub
Fokko closed issue #345: `pyiceberg.io.pyarrow.write_file` does not take into account compression settings URL: https://github.com/apache/iceberg-python/issues/345 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [I] `pyiceberg.io.pyarrow.write_file` does not take into account compression settings [iceberg-python]

2024-02-05 Thread via GitHub
Fokko commented on issue #345: URL: https://github.com/apache/iceberg-python/issues/345#issuecomment-1928067044 Fixed in https://github.com/apache/iceberg-python/pull/358 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [I] Cannot load a binary column of many rows via the `to_arrow` method. [iceberg-python]

2024-02-05 Thread via GitHub
Fokko commented on issue #344: URL: https://github.com/apache/iceberg-python/issues/344#issuecomment-1928067916 @castedice Also feel free to open up a draft if you want some early feedback. -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] Add Daft examples and code into PyIceberg docs and Table [iceberg-python]

2024-02-05 Thread via GitHub
jaychia commented on PR #355: URL: https://github.com/apache/iceberg-python/pull/355#issuecomment-1928068519 > Should we also have some sanity checks, for example: > > https://github.com/apache/iceberg-python/blob/a4856bc2eadf90ac85dec96d4502ca3517bb1bb5/tests/integration/test_reads.p

Re: [PR] Spark: Move the Writer to a visitor [iceberg]

2024-02-05 Thread via GitHub
amogh-jahagirdar commented on code in PR #9440: URL: https://github.com/apache/iceberg/pull/9440#discussion_r1478907380 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/SparkParquetWriters.java: ## @@ -136,46 +135,126 @@ private ParquetValueWriter newOption(Type f

[PR] Build: Bump mkdocs-material from 9.5.6 to 9.5.7 [iceberg-python]

2024-02-05 Thread via GitHub
dependabot[bot] opened a new pull request, #369: URL: https://github.com/apache/iceberg-python/pull/369 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.6 to 9.5.7. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mkdo

[PR] Build: Bump moto from 4.2.13 to 5.0.1 [iceberg-python]

2024-02-05 Thread via GitHub
dependabot[bot] opened a new pull request, #370: URL: https://github.com/apache/iceberg-python/pull/370 Bumps [moto](https://github.com/getmoto/moto) from 4.2.13 to 5.0.1. Changelog Sourced from https://github.com/getmoto/moto/blob/master/CHANGELOG.md";>moto's changelog. 5.0

Re: [PR] Build: Bump moto from 4.2.13 to 5.0.0 [iceberg-python]

2024-02-05 Thread via GitHub
dependabot[bot] closed pull request #321: Build: Bump moto from 4.2.13 to 5.0.0 URL: https://github.com/apache/iceberg-python/pull/321 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] Build: Bump moto from 4.2.13 to 5.0.0 [iceberg-python]

2024-02-05 Thread via GitHub
dependabot[bot] commented on PR #321: URL: https://github.com/apache/iceberg-python/pull/321#issuecomment-1928203293 Superseded by #370. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[PR] Build: Bump mypy-boto3-glue from 1.34.32 to 1.34.35 [iceberg-python]

2024-02-05 Thread via GitHub
dependabot[bot] opened a new pull request, #371: URL: https://github.com/apache/iceberg-python/pull/371 Bumps [mypy-boto3-glue](https://github.com/youtype/mypy_boto3_builder) from 1.34.32 to 1.34.35. Commits See full diff in https://github.com/youtype/mypy_boto3_builder/commits

[PR] Build: Bump adlfs from 2024.1.0 to 2024.2.0 [iceberg-python]

2024-02-05 Thread via GitHub
dependabot[bot] opened a new pull request, #372: URL: https://github.com/apache/iceberg-python/pull/372 Bumps [adlfs](https://github.com/fsspec/adlfs) from 2024.1.0 to 2024.2.0. Release notes Sourced from https://github.com/fsspec/adlfs/releases";>adlfs's releases. 2024.2.0

Re: [PR] Update README.md tp reflect it is archived [iceberg-docs]

2024-02-05 Thread via GitHub
danielcweeks commented on code in PR #310: URL: https://github.com/apache/iceberg-docs/pull/310#discussion_r1479000645 ## README.md: ## @@ -17,170 +17,11 @@ - under the License. --> -# Apache Iceberg Documentation Site +# Apache Iceberg Documentation Site (Archived) -T

Re: [PR] Update README.md tp reflect it is archived [iceberg-docs]

2024-02-05 Thread via GitHub
danielcweeks merged PR #310: URL: https://github.com/apache/iceberg-docs/pull/310 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ice

Re: [PR] Reorganize identifier field requirement [iceberg-docs]

2024-02-05 Thread via GitHub
bitsondatadev commented on PR #306: URL: https://github.com/apache/iceberg-docs/pull/306#issuecomment-1928354012 @manuzhang, would you mind moving this over to https://github.com/apache/iceberg/pulls please? -- This is an automated message from the Apache Git Service. To respond to the me

Re: [PR] Add a Chinese version Documentation & Fix bug while building website. [iceberg-docs]

2024-02-05 Thread via GitHub
bitsondatadev commented on PR #307: URL: https://github.com/apache/iceberg-docs/pull/307#issuecomment-1928369140 @Waterkin, would you mind moving this over to https://github.com/apache/iceberg/pulls please? This repository is being moved to the main repository. -- This is an automated me

Re: [PR] Add release schedule on the releases page [iceberg-docs]

2024-02-05 Thread via GitHub
bitsondatadev commented on PR #298: URL: https://github.com/apache/iceberg-docs/pull/298#issuecomment-1928373285 Hey @jbonofre, would you mind moving this over to https://github.com/apache/iceberg/pulls please? -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] add Wayang / DataBloom to vendors supporting Iceberg [iceberg-docs]

2024-02-05 Thread via GitHub
bitsondatadev commented on PR #283: URL: https://github.com/apache/iceberg-docs/pull/283#issuecomment-1928378162 Hey @2pk03, would you mind moving this over to https://github.com/apache/iceberg/pulls please? -- This is an automated message from the Apache Git Service. To respond to the me

Re: [PR] Fix branching and tagging images on 1.2.1 branch [iceberg-docs]

2024-02-05 Thread via GitHub
bitsondatadev commented on PR #231: URL: https://github.com/apache/iceberg-docs/pull/231#issuecomment-1928389000 @amogh-jahagirdar, would you mind moving this over to https://github.com/apache/iceberg/pulls please? Unless this should just be closed. -- This is an automated message

  1   2   >