Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-16 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1527421375 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveViewOperations.java: ## @@ -0,0 +1,216 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-16 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1527420981 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCatalog.java: ## @@ -0,0 +1,165 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-16 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1527420651 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCommits.java: ## @@ -0,0 +1,474 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-16 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1527420076 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCommits.java: ## @@ -0,0 +1,474 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-16 Thread via GitHub
nk1506 commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1527419265 ## hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCommits.java: ## @@ -0,0 +1,474 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[PR] Build: Bump datamodel-code-generator from 0.25.4 to 0.25.5 [iceberg]

2024-03-16 Thread via GitHub
dependabot[bot] opened a new pull request, #9979: URL: https://github.com/apache/iceberg/pull/9979 Bumps [datamodel-code-generator](https://github.com/koxudaxi/datamodel-code-generator) from 0.25.4 to 0.25.5. Release notes Sourced from https://github.com/koxudaxi/datamodel-code-ge

[PR] Build: Bump com.palantir.baseline:gradle-baseline-java from 4.42.0 to 5.44.0 [iceberg]

2024-03-16 Thread via GitHub
dependabot[bot] opened a new pull request, #9978: URL: https://github.com/apache/iceberg/pull/9978 Bumps [com.palantir.baseline:gradle-baseline-java](https://github.com/palantir/gradle-baseline) from 4.42.0 to 5.44.0. Release notes Sourced from https://github.com/palantir/gradle-b

[PR] Build: Bump guava from 33.0.0-jre to 33.1.0-jre [iceberg]

2024-03-16 Thread via GitHub
dependabot[bot] opened a new pull request, #9977: URL: https://github.com/apache/iceberg/pull/9977 Bumps `guava` from 33.0.0-jre to 33.1.0-jre. Updates `com.google.guava:guava` from 33.0.0-jre to 33.1.0-jre Release notes Sourced from https://github.com/google/guava/releases";>com

[PR] Build: Bump nessie from 0.77.1 to 0.79.0 [iceberg]

2024-03-16 Thread via GitHub
dependabot[bot] opened a new pull request, #9976: URL: https://github.com/apache/iceberg/pull/9976 Bumps `nessie` from 0.77.1 to 0.79.0. Updates `org.projectnessie.nessie:nessie-client` from 0.77.1 to 0.79.0 Updates `org.projectnessie.nessie:nessie-jaxrs-testextension` from 0.77.1 t

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.24.5 to 2.25.6 [iceberg]

2024-03-16 Thread via GitHub
dependabot[bot] closed pull request #9907: Build: Bump software.amazon.awssdk:bom from 2.24.5 to 2.25.6 URL: https://github.com/apache/iceberg/pull/9907 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.24.5 to 2.25.6 [iceberg]

2024-03-16 Thread via GitHub
dependabot[bot] commented on PR #9907: URL: https://github.com/apache/iceberg/pull/9907#issuecomment-2002306140 Superseded by #9975. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[PR] Build: Bump software.amazon.awssdk:bom from 2.24.5 to 2.25.11 [iceberg]

2024-03-16 Thread via GitHub
dependabot[bot] opened a new pull request, #9975: URL: https://github.com/apache/iceberg/pull/9975 Bumps software.amazon.awssdk:bom from 2.24.5 to 2.25.11. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=softwar

[PR] Build: Bump org.xerial:sqlite-jdbc from 3.45.1.0 to 3.45.2.0 [iceberg]

2024-03-16 Thread via GitHub
dependabot[bot] opened a new pull request, #9974: URL: https://github.com/apache/iceberg/pull/9974 Bumps [org.xerial:sqlite-jdbc](https://github.com/xerial/sqlite-jdbc) from 3.45.1.0 to 3.45.2.0. Release notes Sourced from https://github.com/xerial/sqlite-jdbc/releases";>org.xerial

[PR] Build: Bump org.openapitools:openapi-generator-gradle-plugin from 6.6.0 to 7.4.0 [iceberg]

2024-03-16 Thread via GitHub
dependabot[bot] opened a new pull request, #9973: URL: https://github.com/apache/iceberg/pull/9973 Bumps [org.openapitools:openapi-generator-gradle-plugin](https://github.com/OpenAPITools/openapi-generator) from 6.6.0 to 7.4.0. Release notes Sourced from https://github.com/OpenAPI

[PR] Build: Bump com.google.errorprone:error_prone_annotations from 2.24.1 to 2.26.1 [iceberg]

2024-03-16 Thread via GitHub
dependabot[bot] opened a new pull request, #9972: URL: https://github.com/apache/iceberg/pull/9972 Bumps [com.google.errorprone:error_prone_annotations](https://github.com/google/error-prone) from 2.24.1 to 2.26.1. Release notes Sourced from https://github.com/google/error-prone/r

[PR] Build: Bump com.adobe.testing:s3mock-junit5 from 2.11.0 to 3.5.2 [iceberg]

2024-03-16 Thread via GitHub
dependabot[bot] opened a new pull request, #9971: URL: https://github.com/apache/iceberg/pull/9971 Bumps com.adobe.testing:s3mock-junit5 from 2.11.0 to 3.5.2. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=com.

[PR] Build: Bump org.awaitility:awaitility from 4.2.0 to 4.2.1 [iceberg]

2024-03-16 Thread via GitHub
dependabot[bot] opened a new pull request, #9970: URL: https://github.com/apache/iceberg/pull/9970 Bumps [org.awaitility:awaitility](https://github.com/awaitility/awaitility) from 4.2.0 to 4.2.1. Changelog Sourced from https://github.com/awaitility/awaitility/blob/master/changelog.

[PR] Build: Bump org.springframework:spring-web from 5.3.30 to 6.1.5 [iceberg]

2024-03-16 Thread via GitHub
dependabot[bot] opened a new pull request, #9969: URL: https://github.com/apache/iceberg/pull/9969 Bumps [org.springframework:spring-web](https://github.com/spring-projects/spring-framework) from 5.3.30 to 6.1.5. Release notes Sourced from https://github.com/spring-projects/spring

Re: [I] ValueError: Unrecognized filesystem type in URI: abfss [iceberg-python]

2024-03-16 Thread via GitHub
djouallah commented on issue #527: URL: https://github.com/apache/iceberg-python/issues/527#issuecomment-2002280228 getting this error now ``` 319 return file_io 320 else: --> 321 raise ValueError(f"Could not initialize FileIO: {io_impl}")

Re: [I] ExpireSnapshots deletes active files [iceberg]

2024-03-16 Thread via GitHub
github-actions[bot] commented on issue #2131: URL: https://github.com/apache/iceberg/issues/2131#issuecomment-2002198546 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Improve HadoopCatalog performance/scalability [iceberg]

2024-03-16 Thread via GitHub
github-actions[bot] commented on issue #2124: URL: https://github.com/apache/iceberg/issues/2124#issuecomment-2002198530 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] MySQL binlog is synchronized to iceberg through CDC [iceberg]

2024-03-16 Thread via GitHub
github-actions[bot] commented on issue #2128: URL: https://github.com/apache/iceberg/issues/2128#issuecomment-2002198535 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Spark can't read Iceberg table created from Presto [iceberg]

2024-03-16 Thread via GitHub
github-actions[bot] commented on issue #2122: URL: https://github.com/apache/iceberg/issues/2122#issuecomment-2002198514 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Hive: Vectorization is not working [iceberg]

2024-03-16 Thread via GitHub
github-actions[bot] commented on issue #2120: URL: https://github.com/apache/iceberg/issues/2120#issuecomment-2002198506 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Flink: store watermark as iceberg table's property. [iceberg]

2024-03-16 Thread via GitHub
github-actions[bot] commented on issue #2108: URL: https://github.com/apache/iceberg/issues/2108#issuecomment-2002198497 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Add "Help" command for CALL methods [iceberg]

2024-03-16 Thread via GitHub
github-actions[bot] commented on issue #1856: URL: https://github.com/apache/iceberg/issues/1856#issuecomment-2002198443 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Add "Help" command for CALL methods [iceberg]

2024-03-16 Thread via GitHub
github-actions[bot] closed issue #1856: Add "Help" command for CALL methods URL: https://github.com/apache/iceberg/issues/1856 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [I] Why are there few methods for hive/flinkcatalog? [iceberg]

2024-03-16 Thread via GitHub
github-actions[bot] closed issue #1834: Why are there few methods for hive/flinkcatalog? URL: https://github.com/apache/iceberg/issues/1834 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [I] Why are there few methods for hive/flinkcatalog? [iceberg]

2024-03-16 Thread via GitHub
github-actions[bot] commented on issue #1834: URL: https://github.com/apache/iceberg/issues/1834#issuecomment-2002198433 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Make changes to write a Partition stats [iceberg]

2024-03-16 Thread via GitHub
github-actions[bot] closed issue #1833: Make changes to write a Partition stats URL: https://github.com/apache/iceberg/issues/1833 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [I] Make changes to write a Partition stats [iceberg]

2024-03-16 Thread via GitHub
github-actions[bot] commented on issue #1833: URL: https://github.com/apache/iceberg/issues/1833#issuecomment-2002198424 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

[PR] Adding a new dev dep, `deptry` [iceberg-python]

2024-03-16 Thread via GitHub
kevinjqliu opened a new pull request, #528: URL: https://github.com/apache/iceberg-python/pull/528 This PR adds a new dev dependency named [`deptry`](https://github.com/fpgmaas/deptry), which is a "command line tool to check for issues with dependencies in a Python project". ``` poet

Re: [PR] Exclude Python 3.9.7 due to import error in catalog module [iceberg-python]

2024-03-16 Thread via GitHub
ndrluis commented on PR #526: URL: https://github.com/apache/iceberg-python/pull/526#issuecomment-2002045213 Related python issue: https://bugs.python.org/issue45081 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Parquet, Arrow: Refactor vectorized reader [iceberg]

2024-03-16 Thread via GitHub
wgtmac commented on code in PR #9772: URL: https://github.com/apache/iceberg/pull/9772#discussion_r1527198459 ## arrow/src/main/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedParquetDefinitionLevelReader.java: ## @@ -47,35 +46,45 @@ public VectorizedParquetDefinition

Re: [PR] Parquet, Arrow: Refactor vectorized reader [iceberg]

2024-03-16 Thread via GitHub
wgtmac commented on code in PR #9772: URL: https://github.com/apache/iceberg/pull/9772#discussion_r1527197363 ## arrow/src/main/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedParquetDefinitionLevelReader.java: ## @@ -47,35 +46,45 @@ public VectorizedParquetDefinition

Re: [PR] Parquet, Arrow: Refactor vectorized reader [iceberg]

2024-03-16 Thread via GitHub
wgtmac commented on code in PR #9772: URL: https://github.com/apache/iceberg/pull/9772#discussion_r1527197363 ## arrow/src/main/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedParquetDefinitionLevelReader.java: ## @@ -47,35 +46,45 @@ public VectorizedParquetDefinition

Re: [I] ValueError: Unrecognized filesystem type in URI: abfss [iceberg-python]

2024-03-16 Thread via GitHub
kevinjqliu commented on issue #527: URL: https://github.com/apache/iceberg-python/issues/527#issuecomment-2002034161 In the meantime, I think you might be able to workaround this by explicitly using fsspec. You'd have to set this in the catalog properties setting ``` catalog = SqlC

Re: [I] ValueError: Unrecognized filesystem type in URI: abfss [iceberg-python]

2024-03-16 Thread via GitHub
kevinjqliu commented on issue #527: URL: https://github.com/apache/iceberg-python/issues/527#issuecomment-2002032924 Looks like pyarrow can support "fsspec-compatible filesystems" like Azure Blob Storage (`abfs`/`abfss`) https://arrow.apache.org/docs/python/filesystems.html#using-fsspec-

Re: [I] ValueError: Unrecognized filesystem type in URI: abfss [iceberg-python]

2024-03-16 Thread via GitHub
kevinjqliu commented on issue #527: URL: https://github.com/apache/iceberg-python/issues/527#issuecomment-2002031396 Pulling the example out of the notebook: ``` table = catalog.create_table( "default.taxi_dataset", schema=df.schema,location="abfss://onelakene.dfs.cor

Re: [PR] Add Snapshots table metadata [iceberg-python]

2024-03-16 Thread via GitHub
kevinjqliu commented on code in PR #524: URL: https://github.com/apache/iceberg-python/pull/524#discussion_r1527191777 ## tests/integration/test_writes.py: ## @@ -664,3 +668,55 @@ def test_table_properties_raise_for_none_value( session_catalog, identifier, {"format-

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-16 Thread via GitHub
jbonofre commented on PR #9884: URL: https://github.com/apache/iceberg/pull/9884#issuecomment-2001889239 @danielcweeks OK, I understood correctly. I'm not sure it's a so special use case (I'm not sure though 😄 ). Maybe worth to experiment our "proposal process" and have a vote from communit