Re: [PR] Docs: Remove outdated python autoformating command from doc [iceberg]

2023-11-02 Thread via GitHub
ajantha-bhat commented on code in PR #8961: URL: https://github.com/apache/iceberg/pull/8961#discussion_r1379668687 ## site/docs/contribute.md: ## @@ -150,8 +150,6 @@ Example: ### Style -For Python, please use the tox command `tox -e format` to apply autoformatting to the

Re: [PR] Core: Add View support for REST catalog [iceberg]

2023-11-02 Thread via GitHub
nastra commented on code in PR #7913: URL: https://github.com/apache/iceberg/pull/7913#discussion_r1379704585 ## core/src/main/java/org/apache/iceberg/UpdateRequirement.java: ## @@ -20,11 +20,17 @@ import org.apache.iceberg.exceptions.CommitFailedException; import org.apache

Re: [PR] Core: Use ParallelIterable in Deletes::toPositionIndex (6387) [iceberg]

2023-11-02 Thread via GitHub
nastra commented on PR #8805: URL: https://github.com/apache/iceberg/pull/8805#issuecomment-1790230032 LGTM, thanks @wypoon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Core: Use ParallelIterable in Deletes::toPositionIndex (6387) [iceberg]

2023-11-02 Thread via GitHub
nastra merged PR #8805: URL: https://github.com/apache/iceberg/pull/8805 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[I] Support adding an additional `opType` column when creating a table [iceberg]

2023-11-02 Thread via GitHub
klion26 opened a new issue, #8973: URL: https://github.com/apache/iceberg/issues/8973 ### Feature Request / Improvement Hi community, I want to discuss a feature described as the title here, please let me know what do you think about this, thanks. One of our customers had reach

Re: [PR] feat: Implement load table api. [iceberg-rust]

2023-11-02 Thread via GitHub
liurenjie1024 commented on code in PR #89: URL: https://github.com/apache/iceberg-rust/pull/89#discussion_r1379743215 ## crates/iceberg/src/table.rs: ## @@ -17,10 +17,33 @@ //! Table API for Apache Iceberg +use crate::io::FileIO; use crate::spec::TableMetadata; +use crate:

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-02 Thread via GitHub
nk1506 commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1379792699 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveViewOperations.java: ## @@ -0,0 +1,346 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-11-02 Thread via GitHub
nk1506 commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1379834008 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -82,12 +83,20 @@ import org.apache.iceberg.types.Types; import org.apache.iceberg.u

Re: [PR] Docs: Remove outdated python autoformating command from doc [iceberg]

2023-11-02 Thread via GitHub
Fokko commented on code in PR #8961: URL: https://github.com/apache/iceberg/pull/8961#discussion_r1379836984 ## site/docs/contribute.md: ## @@ -150,8 +150,6 @@ Example: ### Style -For Python, please use the tox command `tox -e format` to apply autoformatting to the project

Re: [PR] Docs: Remove outdated python autoformating command from doc [iceberg]

2023-11-02 Thread via GitHub
ajantha-bhat commented on code in PR #8961: URL: https://github.com/apache/iceberg/pull/8961#discussion_r1379853692 ## site/docs/contribute.md: ## @@ -150,8 +150,6 @@ Example: ### Style -For Python, please use the tox command `tox -e format` to apply autoformatting to the

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-11-02 Thread via GitHub
nastra commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1379868549 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java: ## @@ -82,12 +83,20 @@ import org.apache.iceberg.types.Types; import org.apache.iceberg.u

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-02 Thread via GitHub
nastra commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1379887487 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -718,8 +743,16 @@ public void renameTableTargetAlreadyExistsAsView() { assertThat(catalog

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-02 Thread via GitHub
nastra commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1379889859 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -368,8 +368,19 @@ public void createTableViaTransactionThatAlreadyExistsAsView() { asser

Re: [PR] Parquet: Remove duplicate test code [iceberg]

2023-11-02 Thread via GitHub
nastra commented on code in PR #8098: URL: https://github.com/apache/iceberg/pull/8098#discussion_r1379909688 ## parquet/src/test/java/org/apache/iceberg/avro/TestParquetReadProjection.java: ## @@ -45,4 +45,20 @@ protected GenericData.Record writeAndRead( return Iterables

[PR] Fix doc versions [iceberg]

2023-11-02 Thread via GitHub
bitsondatadev opened a new pull request, #8974: URL: https://github.com/apache/iceberg/pull/8974 I curated this set manually and forgot to update the doc versions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Fix doc versions [iceberg]

2023-11-02 Thread via GitHub
Fokko commented on code in PR #8974: URL: https://github.com/apache/iceberg/pull/8974#discussion_r1379913981 ## latest/mkdocs.yml: ## @@ -0,0 +1,70 @@ +# Licensed to the Apache Software Foundation (ASF) under one Review Comment: Do we want to maintain this copy? Can't we do

Re: [PR] Fix doc versions [iceberg]

2023-11-02 Thread via GitHub
bitsondatadev commented on code in PR #8974: URL: https://github.com/apache/iceberg/pull/8974#discussion_r1379916273 ## latest/mkdocs.yml: ## @@ -0,0 +1,70 @@ +# Licensed to the Apache Software Foundation (ASF) under one Review Comment: good point, I was going to have it in

Re: [PR] Fix doc versions [iceberg]

2023-11-02 Thread via GitHub
Fokko merged PR #8974: URL: https://github.com/apache/iceberg/pull/8974 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Hive: Refactor TestHiveCatalog tests to use the core CatalogTests [iceberg]

2023-11-02 Thread via GitHub
nastra commented on code in PR #8918: URL: https://github.com/apache/iceberg/pull/8918#discussion_r1379954233 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -261,6 +261,16 @@ public void renameTable(TableIdentifier from, TableIdentifier original

Re: [PR] Core: Fix retry behavior for Jdbc Client [iceberg]

2023-11-02 Thread via GitHub
cccs-jc commented on PR #7561: URL: https://github.com/apache/iceberg/pull/7561#issuecomment-1790547504 @amogh-jahagirdar, Is there any progress on making the error codes configurable. This would avoid forks just to add code specific to for example posgresql. -- This is an automated mes

Re: [I] Does JDBC connector uses any retry mechanism? [iceberg]

2023-11-02 Thread via GitHub
cccs-eric commented on issue #7173: URL: https://github.com/apache/iceberg/issues/7173#issuecomment-1790553419 > I see the PR didn't make it to a release. Do you need any help with it? As I [stated ](https://github.com/apache/iceberg/pull/7561#issuecomment-1591804604) in the linked P

Re: [PR] Docs: Remove outdated python autoformating command from doc [iceberg]

2023-11-02 Thread via GitHub
Fokko merged PR #8961: URL: https://github.com/apache/iceberg/pull/8961 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Docs: Remove outdated python autoformating command from doc [iceberg]

2023-11-02 Thread via GitHub
Fokko commented on PR #8961: URL: https://github.com/apache/iceberg/pull/8961#issuecomment-1790588060 Thanks @hussein-awala for cleaning this up, and @ajantha-bhat for the prompt review 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log o

[PR] Fd fix column selection [iceberg-python]

2023-11-02 Thread via GitHub
Fokko opened a new pull request, #116: URL: https://github.com/apache/iceberg-python/pull/116 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

[PR] Refactor schema conversion [iceberg-python]

2023-11-02 Thread via GitHub
Fokko opened a new pull request, #117: URL: https://github.com/apache/iceberg-python/pull/117 We wrapped a schema in a schema. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] patch: Parquet Column Names with "Special Characters" fix [iceberg-python]

2023-11-02 Thread via GitHub
Fokko closed pull request #109: patch: Parquet Column Names with "Special Characters" fix URL: https://github.com/apache/iceberg-python/pull/109 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] patch: Parquet Column Names with "Special Characters" fix [iceberg-python]

2023-11-02 Thread via GitHub
Fokko commented on PR #109: URL: https://github.com/apache/iceberg-python/pull/109#issuecomment-1790626176 Thanks you both @MarquisC and @mchamberlain-mdsol for checking this! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] fix: partition evaluator thread safety [iceberg-python]

2023-11-02 Thread via GitHub
Fokko merged PR #115: URL: https://github.com/apache/iceberg-python/pull/115 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Parquet: Remove duplicate test code [iceberg]

2023-11-02 Thread via GitHub
ajantha-bhat commented on code in PR #8098: URL: https://github.com/apache/iceberg/pull/8098#discussion_r1380068898 ## parquet/src/test/java/org/apache/iceberg/avro/TestParquetReadProjection.java: ## @@ -45,4 +45,20 @@ protected GenericData.Record writeAndRead( return Ite

Re: [PR] Parquet: Remove duplicate test code [iceberg]

2023-11-02 Thread via GitHub
nastra merged PR #8098: URL: https://github.com/apache/iceberg/pull/8098 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Core: Move `iceberg-parquet` files to `iceberg-core` [iceberg]

2023-11-02 Thread via GitHub
ajantha-bhat commented on code in PR #8500: URL: https://github.com/apache/iceberg/pull/8500#discussion_r1380176383 ## api/src/test/java/org/apache/iceberg/TestHelpers.java: ## @@ -265,6 +266,18 @@ public static void serialize(final Serializable obj, final OutputStream outputSt

Re: [PR] Core: Move `iceberg-parquet` files to `iceberg-core` [iceberg]

2023-11-02 Thread via GitHub
ajantha-bhat commented on code in PR #8500: URL: https://github.com/apache/iceberg/pull/8500#discussion_r1380176383 ## api/src/test/java/org/apache/iceberg/TestHelpers.java: ## @@ -265,6 +266,18 @@ public static void serialize(final Serializable obj, final OutputStream outputSt

Re: [PR] Run dependabot daily [iceberg-python]

2023-11-02 Thread via GitHub
Fokko merged PR #66: URL: https://github.com/apache/iceberg-python/pull/66 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

[PR] Build: Bump griffe from 0.36.7 to 0.36.9 [iceberg-python]

2023-11-02 Thread via GitHub
dependabot[bot] opened a new pull request, #118: URL: https://github.com/apache/iceberg-python/pull/118 Bumps [griffe](https://github.com/mkdocstrings/griffe) from 0.36.7 to 0.36.9. Release notes Sourced from https://github.com/mkdocstrings/griffe/releases";>griffe's releases.

[PR] Build: Bump fastavro from 1.8.4 to 1.9.0 [iceberg-python]

2023-11-02 Thread via GitHub
dependabot[bot] opened a new pull request, #119: URL: https://github.com/apache/iceberg-python/pull/119 Bumps [fastavro](https://github.com/fastavro/fastavro) from 1.8.4 to 1.9.0. Changelog Sourced from https://github.com/fastavro/fastavro/blob/master/ChangeLog";>fastavro's changel

[PR] Build: Bump pyarrow from 13.0.0 to 14.0.0 [iceberg-python]

2023-11-02 Thread via GitHub
dependabot[bot] opened a new pull request, #121: URL: https://github.com/apache/iceberg-python/pull/121 Bumps [pyarrow](https://github.com/apache/arrow) from 13.0.0 to 14.0.0. Commits https://github.com/apache/arrow/commit/2dcee3f82c6cf54b53a64729fd81840efa583244";>2dcee3f MINO

[PR] Build: Bump zstandard from 0.21.0 to 0.22.0 [iceberg-python]

2023-11-02 Thread via GitHub
dependabot[bot] opened a new pull request, #120: URL: https://github.com/apache/iceberg-python/pull/120 Bumps [zstandard](https://github.com/indygreg/python-zstandard) from 0.21.0 to 0.22.0. Release notes Sourced from https://github.com/indygreg/python-zstandard/releases";>zstandar

Re: [PR] Build: Bump griffe from 0.36.7 to 0.36.9 [iceberg-python]

2023-11-02 Thread via GitHub
Fokko merged PR #118: URL: https://github.com/apache/iceberg-python/pull/118 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

[PR] Build: Bump cython from 3.0.3 to 3.0.5 [iceberg-python]

2023-11-02 Thread via GitHub
dependabot[bot] opened a new pull request, #122: URL: https://github.com/apache/iceberg-python/pull/122 Bumps [cython](https://github.com/cython/cython) from 3.0.3 to 3.0.5. Changelog Sourced from https://github.com/cython/cython/blob/master/CHANGES.rst";>cython's changelog.

Re: [PR] Nessie: Support views for NessieCatalog [iceberg]

2023-11-02 Thread via GitHub
adutra commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1380041190 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieTableOperations.java: ## @@ -132,74 +131,36 @@ protected void doRefresh() { @Override protected void doCo

Re: [PR] Spark 3.5: Honor Spark conf spark.sql.files.maxPartitionBytes in read split [iceberg]

2023-11-02 Thread via GitHub
holdenk commented on PR #8922: URL: https://github.com/apache/iceberg/pull/8922#issuecomment-1791080603 What about if we made it an explicit Iceberg property rather than re-using a Spark property? -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] Spark 3.5: Use DataFile constants in SparkDataFile [iceberg]

2023-11-02 Thread via GitHub
aokolnychyi commented on PR #8936: URL: https://github.com/apache/iceberg/pull/8936#issuecomment-1791094061 Thanks, @singhpk234 @amogh-jahagirdar! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Spark 3.5: Use DataFile constants in SparkDataFile [iceberg]

2023-11-02 Thread via GitHub
aokolnychyi merged PR #8936: URL: https://github.com/apache/iceberg/pull/8936 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Add Spark UI metrics from Iceberg scan metrics [iceberg]

2023-11-02 Thread via GitHub
karuppayya commented on PR #8717: URL: https://github.com/apache/iceberg/pull/8717#issuecomment-1791205092 Thanks @aokolnychyi for the review, i have addressed the latest comments, ready for another round -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2023-11-02 Thread via GitHub
nk1506 commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1380574090 ## core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java: ## @@ -718,8 +743,16 @@ public void renameTableTargetAlreadyExistsAsView() { assertThat(catalog

Re: [PR] Refactor Arrow schema conversion [iceberg-python]

2023-11-02 Thread via GitHub
bitsondatadev commented on code in PR #117: URL: https://github.com/apache/iceberg-python/pull/117#discussion_r1380591348 ## tests/io/test_pyarrow.py: ## @@ -708,15 +709,17 @@ def _write_table_to_file(filepath: str, schema: pa.Schema, table: pa.Table) -> s @pytest.fixture d

[I] Support MOR CDC view [iceberg]

2023-11-02 Thread via GitHub
puchengy opened a new issue, #8975: URL: https://github.com/apache/iceberg/issues/8975 ### Feature Request / Improvement Currently Flink upserts to Iceberg table can not produce CDC view. ``` Error: Delete files are currently not supported in changelog scans ``` This is due

[PR] Release notes for 1.4.2 patch release [iceberg-docs]

2023-11-02 Thread via GitHub
amogh-jahagirdar opened a new pull request, #292: URL: https://github.com/apache/iceberg-docs/pull/292 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

[PR] Support usage of Separate OIDC Authorization Server URI [iceberg]

2023-11-02 Thread via GitHub
syun64 opened a new pull request, #8976: URL: https://github.com/apache/iceberg/pull/8976 Related: https://github.com/apache/iceberg/issues/8869 Introduce `OAuth2Properties.TOKEN_URI = "token-uri"` that can be used to override the default behavior that assumes that the Rest Catalog Se

[I] Catalog fails to load table using the table's identifier [iceberg-python]

2023-11-02 Thread via GitHub
pdames opened a new issue, #123: URL: https://github.com/apache/iceberg-python/issues/123 ### Apache Iceberg version main (development) ### Please describe the bug 🐞 **Reproduction Steps** 1. Load a catalog named `test_catalog`. 2. Create a table named `test_namespa

Re: [PR] Spark 3.5: Use rolling manifest writers when optimizing metadata [iceberg]

2023-11-02 Thread via GitHub
RussellSpitzer commented on code in PR #8972: URL: https://github.com/apache/iceberg/pull/8972#discussion_r1380659942 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -307,8 +268,16 @@ private int targetNumManifests(long

Re: [PR] Release notes for 1.4.2 patch release [iceberg-docs]

2023-11-02 Thread via GitHub
amogh-jahagirdar merged PR #292: URL: https://github.com/apache/iceberg-docs/pull/292 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...

Re: [PR] Spark 3.5: Use rolling manifest writers when optimizing metadata [iceberg]

2023-11-02 Thread via GitHub
RussellSpitzer commented on code in PR #8972: URL: https://github.com/apache/iceberg/pull/8972#discussion_r1380662798 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -406,12 +406,13 @@ public void testRewriteLargeManifest

Re: [PR] Spark 3.5: Use rolling manifest writers when optimizing metadata [iceberg]

2023-11-02 Thread via GitHub
RussellSpitzer commented on code in PR #8972: URL: https://github.com/apache/iceberg/pull/8972#discussion_r1380671597 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -221,41 +206,24 @@ private Dataset buildManifestEntry

Re: [I] Catalog fails to load table using the table's identifier [iceberg-python]

2023-11-02 Thread via GitHub
danielcweeks commented on issue #123: URL: https://github.com/apache/iceberg-python/issues/123#issuecomment-1791411464 Other than the obvious edge case: loading a table from a catalog that supports 4-part namespaces and happens to have a namespace that matchings the first part. I wou

Re: [PR] Spark 3.5: Use rolling manifest writers when optimizing metadata [iceberg]

2023-11-02 Thread via GitHub
aokolnychyi commented on code in PR #8972: URL: https://github.com/apache/iceberg/pull/8972#discussion_r1380712826 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -406,12 +406,13 @@ public void testRewriteLargeManifestsPa

Re: [PR] Spark 3.5: Use rolling manifest writers when optimizing metadata [iceberg]

2023-11-02 Thread via GitHub
aokolnychyi commented on code in PR #8972: URL: https://github.com/apache/iceberg/pull/8972#discussion_r1380713525 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -221,41 +206,24 @@ private Dataset buildManifestEntryDF(

Re: [PR] Spark 3.5: Use rolling manifest writers when optimizing metadata [iceberg]

2023-11-02 Thread via GitHub
aokolnychyi commented on code in PR #8972: URL: https://github.com/apache/iceberg/pull/8972#discussion_r1380713525 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -221,41 +206,24 @@ private Dataset buildManifestEntryDF(

Re: [PR] Implement pre-existing session support for dynamodb catalog [iceberg-python]

2023-11-02 Thread via GitHub
waifairer commented on code in PR #104: URL: https://github.com/apache/iceberg-python/pull/104#discussion_r1380715053 ## pyiceberg/catalog/dynamodb.py: ## @@ -80,7 +80,15 @@ class DynamoDbCatalog(Catalog): def __init__(self, name: str, **properties: str): super().

Re: [PR] Spark 3.5: Use rolling manifest writers when optimizing metadata [iceberg]

2023-11-02 Thread via GitHub
aokolnychyi commented on code in PR #8972: URL: https://github.com/apache/iceberg/pull/8972#discussion_r1380715154 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -307,8 +268,16 @@ private int targetNumManifests(long tot

Re: [PR] Implement pre-existing session support for dynamodb catalog [iceberg-python]

2023-11-02 Thread via GitHub
waifairer commented on code in PR #104: URL: https://github.com/apache/iceberg-python/pull/104#discussion_r1380715916 ## mkdocs/docs/configuration.md: ## @@ -195,6 +195,19 @@ catalog: table-name: iceberg ``` +If you prefer to pass the credentials explicitly to the client

Re: [PR] Spark 3.5: Use rolling manifest writers when optimizing metadata [iceberg]

2023-11-02 Thread via GitHub
aokolnychyi commented on code in PR #8972: URL: https://github.com/apache/iceberg/pull/8972#discussion_r1380712826 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteManifestsAction.java: ## @@ -406,12 +406,13 @@ public void testRewriteLargeManifestsPa

Re: [PR] Spark 3.5: Use rolling manifest writers when optimizing metadata [iceberg]

2023-11-02 Thread via GitHub
aokolnychyi commented on code in PR #8972: URL: https://github.com/apache/iceberg/pull/8972#discussion_r1380715154 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -307,8 +268,16 @@ private int targetNumManifests(long tot

[I] View is no longer in sync with table after catalog cache entry expires [iceberg]

2023-11-02 Thread via GitHub
sethwhite-sf opened a new issue, #8977: URL: https://github.com/apache/iceberg/issues/8977 ### Apache Iceberg version 1.4.1 (latest release) ### Query engine Spark ### Please describe the bug 🐞 We have found that temporary views that reference an iceberg tab

Re: [PR] Spark: Fix usage of staging location when optimizing metadata [iceberg]

2023-11-02 Thread via GitHub
flyrain commented on code in PR #8959: URL: https://github.com/apache/iceberg/pull/8959#discussion_r1380821842 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -73,8 +73,10 @@ * * By default, this action rewrites all

[PR] Build: Bump sqlalchemy from 2.0.22 to 2.0.23 [iceberg-python]

2023-11-02 Thread via GitHub
dependabot[bot] opened a new pull request, #125: URL: https://github.com/apache/iceberg-python/pull/125 Bumps [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) from 2.0.22 to 2.0.23. Release notes Sourced from https://github.com/sqlalchemy/sqlalchemy/releases";>sqlalchemy's r

Re: [I] Facing warning when starting spark-sql in EMR using Glue Catalog [iceberg]

2023-11-02 Thread via GitHub
wood-dunamu commented on issue #8544: URL: https://github.com/apache/iceberg/issues/8544#issuecomment-1791671365 Resolved Issue. EMR: 6.14.0 Iceberg: 1.3.1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Spark 3.5: Use rolling manifest writers when optimizing metadata [iceberg]

2023-11-02 Thread via GitHub
singhpk234 commented on code in PR #8972: URL: https://github.com/apache/iceberg/pull/8972#discussion_r1380655906 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -183,9 +170,7 @@ private RewriteManifests.Result doExecute

Re: [PR] Spark 3.5: Fix Migrate procedure renaming issue for custom catalog [iceberg]

2023-11-02 Thread via GitHub
singhpk234 commented on code in PR #8931: URL: https://github.com/apache/iceberg/pull/8931#discussion_r1380889545 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/MigrateTableSparkAction.java: ## @@ -108,6 +109,23 @@ public MigrateTableSparkAction backupTableNa

Re: [PR] Spark 3.5: Fix Migrate procedure renaming issue for custom catalog [iceberg]

2023-11-02 Thread via GitHub
singhpk234 commented on code in PR #8931: URL: https://github.com/apache/iceberg/pull/8931#discussion_r1380889545 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/MigrateTableSparkAction.java: ## @@ -108,6 +109,23 @@ public MigrateTableSparkAction backupTableNa

Re: [PR] Spark 3.5: Use rolling manifest writers when optimizing metadata [iceberg]

2023-11-02 Thread via GitHub
aokolnychyi commented on code in PR #8972: URL: https://github.com/apache/iceberg/pull/8972#discussion_r1380909387 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -183,9 +170,7 @@ private RewriteManifests.Result doExecut

Re: [I] Improve retry algorithm for Tasks [iceberg]

2023-11-02 Thread via GitHub
github-actions[bot] commented on issue #7528: URL: https://github.com/apache/iceberg/issues/7528#issuecomment-1791727528 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Implement pre-existing session support for dynamodb catalog [iceberg-python]

2023-11-02 Thread via GitHub
HonahX commented on code in PR #104: URL: https://github.com/apache/iceberg-python/pull/104#discussion_r1380909825 ## pyiceberg/catalog/dynamodb.py: ## @@ -80,7 +80,15 @@ class DynamoDbCatalog(Catalog): def __init__(self, name: str, **properties: str): super().__i

Re: [PR] Implement pre-existing session support for dynamodb catalog [iceberg-python]

2023-11-02 Thread via GitHub
HonahX commented on code in PR #104: URL: https://github.com/apache/iceberg-python/pull/104#discussion_r1380907420 ## pyiceberg/catalog/dynamodb.py: ## @@ -110,7 +118,7 @@ def _dynamodb_table_exists(self) -> bool: return False except self.dynamodb.exception

Re: [PR] Core: Optimize the process of find start delete file index [iceberg]

2023-11-02 Thread via GitHub
aokolnychyi commented on PR #: URL: https://github.com/apache/iceberg/pull/#issuecomment-1791762628 @bknbkn, we don't anticipate a big number of delete files within a partition with the same sequence number. Do you have a benchmark comparing the proposed solution to what we already

Re: [PR] Add Spark UI metrics from Iceberg scan metrics [iceberg]

2023-11-02 Thread via GitHub
aokolnychyi merged PR #8717: URL: https://github.com/apache/iceberg/pull/8717 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Add Spark UI metrics from Iceberg scan metrics [iceberg]

2023-11-02 Thread via GitHub
aokolnychyi commented on PR #8717: URL: https://github.com/apache/iceberg/pull/8717#issuecomment-1791771494 Thanks, @karuppayya! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Implement pre-existing session support for dynamodb catalog [iceberg-python]

2023-11-02 Thread via GitHub
waifairer commented on code in PR #104: URL: https://github.com/apache/iceberg-python/pull/104#discussion_r1380963042 ## pyiceberg/catalog/dynamodb.py: ## @@ -110,7 +118,7 @@ def _dynamodb_table_exists(self) -> bool: return False except self.dynamodb.except

Re: [PR] Spark: Fix usage of staging location when optimizing metadata [iceberg]

2023-11-02 Thread via GitHub
aokolnychyi commented on code in PR #8959: URL: https://github.com/apache/iceberg/pull/8959#discussion_r1380975737 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -73,8 +73,10 @@ * * By default, this action rewrites

Re: [PR] Spark: Fix usage of staging location when optimizing metadata [iceberg]

2023-11-02 Thread via GitHub
aokolnychyi commented on code in PR #8959: URL: https://github.com/apache/iceberg/pull/8959#discussion_r1380984561 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -104,13 +107,20 @@ public class RewriteManifestsSparkActi

Re: [PR] Spark: Fix usage of staging location when optimizing metadata [iceberg]

2023-11-02 Thread via GitHub
aokolnychyi commented on code in PR #8959: URL: https://github.com/apache/iceberg/pull/8959#discussion_r1380984561 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java: ## @@ -104,13 +107,20 @@ public class RewriteManifestsSparkActi

[I] org.apache.iceberg.spark.source.SerializableTableWithSize cannot be cast to org.apache.iceberg.Table [iceberg]

2023-11-02 Thread via GitHub
HilbertGodel opened a new issue, #8978: URL: https://github.com/apache/iceberg/issues/8978 ### Apache Iceberg version 1.4.1 (latest release) ### Query engine Spark ### Please describe the bug 🐞 I connected to spark connect server with python, showing the exc

Re: [PR] feat: Implement load table api. [iceberg-rust]

2023-11-02 Thread via GitHub
liurenjie1024 commented on PR #89: URL: https://github.com/apache/iceberg-rust/pull/89#issuecomment-1791793211 CC @Xuanwo @Fokko Any other comments? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [I] View is no longer in sync with table after catalog cache entry expires [iceberg]

2023-11-02 Thread via GitHub
singhpk234 commented on issue #8977: URL: https://github.com/apache/iceberg/issues/8977#issuecomment-1791799131 This is interesting, sounds more of spark's view handling issue than iceberg spark integration to the best of my understanding the spark temp views (resolved plan) get cached as w

Re: [I] Partitioning by Year/Month/Day [iceberg]

2023-11-02 Thread via GitHub
wfxxh commented on issue #4129: URL: https://github.com/apache/iceberg/issues/4129#issuecomment-1791842127 I use pa > If anyone is confused about this error on AWS Athena, unlike with hive-style file partitioning in S3 like `{table-s3-location}/year={}/month={}/day={}/` > > Th

Re: [I] Catalog fails to load table using the table's identifier [iceberg-python]

2023-11-02 Thread via GitHub
Fokko commented on issue #123: URL: https://github.com/apache/iceberg-python/issues/123#issuecomment-1791855246 Thanks for raising this @pdames In hindsight, I think adding the catalog name to the identifier was a bad choice. We tried to mimic the behavior of Java, but I don't see any

Re: [PR] Spark: Fix usage of staging location when optimizing metadata [iceberg]

2023-11-02 Thread via GitHub
aokolnychyi merged PR #8959: URL: https://github.com/apache/iceberg/pull/8959 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Spark: Fix usage of staging location when optimizing metadata [iceberg]

2023-11-02 Thread via GitHub
aokolnychyi commented on PR #8959: URL: https://github.com/apache/iceberg/pull/8959#issuecomment-1791856684 Thanks, @flyrain! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] API, Core: implement types timestamp_ns and timestamptz_ns [iceberg]

2023-11-02 Thread via GitHub
jacobmarble commented on PR #8971: URL: https://github.com/apache/iceberg/pull/8971#issuecomment-1791880462 Fixing four failing tests caused me to do most of the new type work for Parquet and ORC as well. Please note that the ORC type attributes need an update in the spec, else it isn't pos

Re: [I] api: Catalog API design [iceberg-rust]

2023-11-02 Thread via GitHub
Xuanwo closed issue #23: api: Catalog API design URL: https://github.com/apache/iceberg-rust/issues/23 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: i

Re: [I] api: Catalog API design [iceberg-rust]

2023-11-02 Thread via GitHub
Xuanwo commented on issue #23: URL: https://github.com/apache/iceberg-rust/issues/23#issuecomment-1791880681 API has been added, let's track tasks on other issues. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Build: Bump cython from 3.0.3 to 3.0.5 [iceberg-python]

2023-11-02 Thread via GitHub
Fokko merged PR #122: URL: https://github.com/apache/iceberg-python/pull/122 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Build: Bump sqlalchemy from 2.0.22 to 2.0.23 [iceberg-python]

2023-11-02 Thread via GitHub
Fokko merged PR #125: URL: https://github.com/apache/iceberg-python/pull/125 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Build: Bump zstandard from 0.21.0 to 0.22.0 [iceberg-python]

2023-11-02 Thread via GitHub
Fokko merged PR #120: URL: https://github.com/apache/iceberg-python/pull/120 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Spark 3.5: Fix Migrate procedure renaming issue for custom catalog [iceberg]

2023-11-02 Thread via GitHub
tomtongue commented on code in PR #8931: URL: https://github.com/apache/iceberg/pull/8931#discussion_r1381164050 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/MigrateTableSparkAction.java: ## @@ -108,6 +109,23 @@ public MigrateTableSparkAction backupTableNam

Re: [PR] Spark 3.5: Fix Migrate procedure renaming issue for custom catalog [iceberg]

2023-11-02 Thread via GitHub
tomtongue commented on code in PR #8931: URL: https://github.com/apache/iceberg/pull/8931#discussion_r1381164050 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/MigrateTableSparkAction.java: ## @@ -108,6 +109,23 @@ public MigrateTableSparkAction backupTableNam

Re: [PR] API, Core: implement types timestamp_ns and timestamptz_ns [iceberg]

2023-11-02 Thread via GitHub
Fokko commented on code in PR #8971: URL: https://github.com/apache/iceberg/pull/8971#discussion_r1381166396 ## api/src/main/java/org/apache/iceberg/expressions/ExpressionUtil.java: ## @@ -589,17 +603,17 @@ private static String sanitizeNumber(Number value, String type) {

Re: [PR] API, Core: implement types timestamp_ns and timestamptz_ns [iceberg]

2023-11-02 Thread via GitHub
Fokko commented on code in PR #8971: URL: https://github.com/apache/iceberg/pull/8971#discussion_r1381171214 ## api/src/main/java/org/apache/iceberg/transforms/Days.java: ## @@ -55,14 +56,14 @@ public boolean satisfiesOrderOf(Transform other) { } if (other instanceof

Re: [I] Improve `All` Metadata Tables with Snapshot Information [iceberg]

2023-11-02 Thread via GitHub
nk1506 commented on issue #8856: URL: https://github.com/apache/iceberg/issues/8856#issuecomment-1791915604 @RussellSpitzer , Is it fine, If I will continue on this ? If so Please assign this to me. -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [PR] Build: Bump fastavro from 1.8.4 to 1.9.0 [iceberg-python]

2023-11-02 Thread via GitHub
Fokko merged PR #119: URL: https://github.com/apache/iceberg-python/pull/119 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

  1   2   >