Re: [PR] [doc] Support multilingual doc by google-translator plugin [iceberg-python]

2024-01-20 Thread via GitHub
HonahX commented on PR #277: URL: https://github.com/apache/iceberg-python/pull/277#issuecomment-1902545955 Thanks for raising this @Waterkin. I noticed that Google Translator also translates package names and some terminology into Chinese: https://github.com/apache/iceberg-python/assets/

Re: [PR] Build: Bump pyspark from 3.4.2 to 3.5.0 [iceberg-python]

2024-01-20 Thread via GitHub
HonahX commented on PR #283: URL: https://github.com/apache/iceberg-python/pull/283#issuecomment-1902539269 Hi @Fokko. > Maybe we should pass that in through an environment variable Do you mean putting it in a `.env` file? Also, I am thinking if it is possible to link the iceb

Re: [PR] Set Glue Table Information when creating/updating tables [iceberg-python]

2024-01-20 Thread via GitHub
HonahX commented on PR #288: URL: https://github.com/apache/iceberg-python/pull/288#issuecomment-1902533747 Also, if possible, could you please also add some tests in [integration_test_glue.py](https://github.com/apache/iceberg-python/blob/3085c404c99ba8c5c8856f21a6c8d63a12ca0113/tests/catal

Re: [PR] Set Glue Table Information when creating/updating tables [iceberg-python]

2024-01-20 Thread via GitHub
HonahX commented on code in PR #288: URL: https://github.com/apache/iceberg-python/pull/288#discussion_r1460723261 ## pyiceberg/catalog/glue.py: ## @@ -84,19 +110,105 @@ def _construct_parameters( return new_parameters +def _type_to_glue_type_string(input_type: IcebergT

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-20 Thread via GitHub
jbonofre commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1460724589 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -503,6 +541,78 @@ public boolean namespaceExists(Namespace namespace) { return JdbcUtil.name

Re: [PR] Build: Bump org.xerial:sqlite-jdbc from 3.44.0.0 to 3.44.1.0 [iceberg]

2024-01-20 Thread via GitHub
dependabot[bot] closed pull request #9203: Build: Bump org.xerial:sqlite-jdbc from 3.44.0.0 to 3.44.1.0 URL: https://github.com/apache/iceberg/pull/9203 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[PR] Build: Bump org.xerial:sqlite-jdbc from 3.44.0.0 to 3.45.0.0 [iceberg]

2024-01-20 Thread via GitHub
dependabot[bot] opened a new pull request, #9540: URL: https://github.com/apache/iceberg/pull/9540 Bumps [org.xerial:sqlite-jdbc](https://github.com/xerial/sqlite-jdbc) from 3.44.0.0 to 3.45.0.0. Release notes Sourced from https://github.com/xerial/sqlite-jdbc/releases";>org.xerial

[PR] Build: Bump com.diffplug.spotless:spotless-plugin-gradle from 6.13.0 to 6.24.0 [iceberg]

2024-01-20 Thread via GitHub
dependabot[bot] opened a new pull request, #9538: URL: https://github.com/apache/iceberg/pull/9538 Bumps [com.diffplug.spotless:spotless-plugin-gradle](https://github.com/diffplug/spotless) from 6.13.0 to 6.24.0. Commits https://github.com/diffplug/spotless/commit/1387d1beedfd

Re: [PR] Build: Bump org.xerial:sqlite-jdbc from 3.44.0.0 to 3.44.1.0 [iceberg]

2024-01-20 Thread via GitHub
dependabot[bot] commented on PR #9203: URL: https://github.com/apache/iceberg/pull/9203#issuecomment-1902511767 Superseded by #9540. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[PR] Build: Bump software.amazon.awssdk:bom from 2.23.2 to 2.23.7 [iceberg]

2024-01-20 Thread via GitHub
dependabot[bot] opened a new pull request, #9539: URL: https://github.com/apache/iceberg/pull/9539 Bumps software.amazon.awssdk:bom from 2.23.2 to 2.23.7. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=software

[PR] Build: Bump nessie from 0.76.2 to 0.76.3 [iceberg]

2024-01-20 Thread via GitHub
dependabot[bot] opened a new pull request, #9537: URL: https://github.com/apache/iceberg/pull/9537 Bumps `nessie` from 0.76.2 to 0.76.3. Updates `org.projectnessie.nessie:nessie-client` from 0.76.2 to 0.76.3 Updates `org.projectnessie.nessie:nessie-jaxrs-testextension` from 0.76.2 t

[PR] Build: Bump io.netty:netty-buffer from 4.1.68.Final to 4.1.106.Final [iceberg]

2024-01-20 Thread via GitHub
dependabot[bot] opened a new pull request, #9536: URL: https://github.com/apache/iceberg/pull/9536 Bumps [io.netty:netty-buffer](https://github.com/netty/netty) from 4.1.68.Final to 4.1.106.Final. Commits https://github.com/netty/netty/commit/9d0ec7b9356487f11921340c6b54f9f0c8c

[PR] Build: Bump spring-boot from 2.5.4 to 3.2.2 [iceberg]

2024-01-20 Thread via GitHub
dependabot[bot] opened a new pull request, #9535: URL: https://github.com/apache/iceberg/pull/9535 Bumps `spring-boot` from 2.5.4 to 3.2.2. Updates `org.springframework.boot:spring-boot-starter-jetty` from 2.5.4 to 3.2.2 Release notes Sourced from https://github.com/spring-proje

Re: [PR] Flink: Implement enumerator metrics for pending splits, pending recor… [iceberg]

2024-01-20 Thread via GitHub
stevenzwu commented on code in PR #9524: URL: https://github.com/apache/iceberg/pull/9524#discussion_r1460717138 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/assigner/SplitAssigner.java: ## @@ -115,4 +115,7 @@ default void onCompletedSplits(Collection comp

Re: [PR] Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.29.0 [iceberg]

2024-01-20 Thread via GitHub
dependabot[bot] closed pull request #9312: Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.29.0 URL: https://github.com/apache/iceberg/pull/9312 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.29.0 [iceberg]

2024-01-20 Thread via GitHub
dependabot[bot] commented on PR #9312: URL: https://github.com/apache/iceberg/pull/9312#issuecomment-1902511663 Superseded by #9534. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[PR] Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.30.0 [iceberg]

2024-01-20 Thread via GitHub
dependabot[bot] opened a new pull request, #9534: URL: https://github.com/apache/iceberg/pull/9534 Bumps [com.google.cloud:libraries-bom](https://github.com/googleapis/java-cloud-bom) from 26.28.0 to 26.30.0. Release notes Sourced from https://github.com/googleapis/java-cloud-bom/

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

2024-01-20 Thread via GitHub
bryanck commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1460715863 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/IcebergSinkConfig.java: ## @@ -0,0 +1,478 @@ +/* + * Licensed to the Apache Software Foundation

[PR] Build: Bump mkdocs-material from 9.5.3 to 9.5.4 [iceberg]

2024-01-20 Thread via GitHub
dependabot[bot] opened a new pull request, #9533: URL: https://github.com/apache/iceberg/pull/9533 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.3 to 9.5.4. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mkdocs-ma

[PR] Build: Bump actions/cache from 3 to 4 [iceberg]

2024-01-20 Thread via GitHub
dependabot[bot] opened a new pull request, #9532: URL: https://github.com/apache/iceberg/pull/9532 Bumps [actions/cache](https://github.com/actions/cache) from 3 to 4. Release notes Sourced from https://github.com/actions/cache/releases";>actions/cache's releases. v4.0.0

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

2024-01-20 Thread via GitHub
bryanck commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1460713886 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/PartitionedAppendWriter.java: ## @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software F

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

2024-01-20 Thread via GitHub
bryanck commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1460713684 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/Utilities.java: ## @@ -0,0 +1,254 @@ +/* + * Licensed to the Apache Software Foundation (AS

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

2024-01-20 Thread via GitHub
bryanck commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1460711350 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/Utilities.java: ## @@ -0,0 +1,254 @@ +/* + * Licensed to the Apache Software Foundation (AS

Re: [PR] Build: Fix errorprone warning [iceberg]

2024-01-20 Thread via GitHub
ajantha-bhat commented on code in PR #9531: URL: https://github.com/apache/iceberg/pull/9531#discussion_r1460708203 ## data/src/main/java/org/apache/iceberg/data/BaseDeleteLoader.java: ## @@ -151,6 +151,7 @@ public PositionDeleteIndex loadPositionDeletes( return PositionDel

Re: [PR] Build: Fix errorprone warning [iceberg]

2024-01-20 Thread via GitHub
ajantha-bhat commented on code in PR #9531: URL: https://github.com/apache/iceberg/pull/9531#discussion_r1460708018 ## baseline.gradle: ## @@ -105,6 +105,7 @@ subprojects { '-Xep:IntLongMath:ERROR', '-Xep:MissingSummary:ERROR', '-Xep:AnnotateForm

[PR] Build: Fix errorprone warning [iceberg]

2024-01-20 Thread via GitHub
ajantha-bhat opened a new pull request, #9531: URL: https://github.com/apache/iceberg/pull/9531 https://github.com/apache/iceberg/assets/5889404/a49093cd-4901-432e-90ad-58eb855cdf56";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-20 Thread via GitHub
ajantha-bhat commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1460705621 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -503,6 +541,78 @@ public boolean namespaceExists(Namespace namespace) { return JdbcUtil.

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-01-20 Thread via GitHub
ajantha-bhat commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1460705273 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -303,6 +287,325 @@ public static Properties filterAndRemovePrefix(Map properties, S return

Re: [PR] Arrow, AWS, Core: Remove deprecated code for 1.5.0 release [iceberg]

2024-01-20 Thread via GitHub
ajantha-bhat commented on code in PR #9505: URL: https://github.com/apache/iceberg/pull/9505#discussion_r1460702238 ## aws/src/test/java/org/apache/iceberg/aws/TestAwsProperties.java: ## @@ -1,50 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or m

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

2024-01-20 Thread via GitHub
bryanck commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1460701487 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/Utilities.java: ## @@ -0,0 +1,254 @@ +/* + * Licensed to the Apache Software Foundation (AS

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

2024-01-20 Thread via GitHub
bryanck commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1460700886 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/Utilities.java: ## @@ -0,0 +1,254 @@ +/* + * Licensed to the Apache Software Foundation (AS

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

2024-01-20 Thread via GitHub
bryanck commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1460700772 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/Utilities.java: ## @@ -0,0 +1,254 @@ +/* + * Licensed to the Apache Software Foundation (AS

Re: [I] [doc] Word `Gague ` spelled incorrectly in https://iceberg.apache.org/docs/latest/flink-writes/#metrics [iceberg]

2024-01-20 Thread via GitHub
Aireed commented on issue #9527: URL: https://github.com/apache/iceberg/issues/9527#issuecomment-1902470431 > Thanks @Aireed ! PRs are welcome, feel free to make the change in the docs and we can review OK, i will submit a PR to fix it. -- This is an automated message from the Apac

Re: [I] Add update-statement support in the Flink engine [iceberg]

2024-01-20 Thread via GitHub
yimlin commented on issue #9517: URL: https://github.com/apache/iceberg/issues/9517#issuecomment-1902467499 Thanks very much. as you know, the trino engine also suppport update statement, it's better for iceberg that it has the same behaviors in different engines. -- This is an automat

Re: [I] Support case-sensitivity in all operations with metrics evaluators [iceberg]

2024-01-20 Thread via GitHub
github-actions[bot] commented on issue #413: URL: https://github.com/apache/iceberg/issues/413#issuecomment-1902456535 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-01-20 Thread via GitHub
danielcweeks commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1460690912 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveViewOperations.java: ## @@ -0,0 +1,302 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-01-20 Thread via GitHub
danielcweeks commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1460690425 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveViewOperations.java: ## @@ -0,0 +1,302 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-01-20 Thread via GitHub
danielcweeks commented on code in PR #8907: URL: https://github.com/apache/iceberg/pull/8907#discussion_r1460673422 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -115,40 +126,22 @@ public void initialize(String inputName, Map properties) {

[PR] Kevinliu/test [iceberg-python]

2024-01-20 Thread via GitHub
kevinjqliu opened a new pull request, #289: URL: https://github.com/apache/iceberg-python/pull/289 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscri

Re: [I] iceberg-core: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

2024-01-20 Thread via GitHub
seyeh commented on issue #9085: URL: https://github.com/apache/iceberg/issues/9085#issuecomment-1902197424 @nastra Could I be assigned this issue? Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [I] AWS: Support setting description for Glue table [iceberg]

2024-01-20 Thread via GitHub
lkokhreidze commented on issue #9462: URL: https://github.com/apache/iceberg/issues/9462#issuecomment-1902094162 Thanks @singhpk234. Created the PR https://github.com/apache/iceberg/pull/9530 -- This is an automated message from the Apache Git Service. To respond to the message, please lo

[PR] Increase operations-per-run to 100 [iceberg]

2024-01-20 Thread via GitHub
manuzhang opened a new pull request, #9529: URL: https://github.com/apache/iceberg/pull/9529 With the default value 30 of `operations-per-run`, actually only 15 issues can be checked daily for staleness. ``` Starting the stale action process... Processing the batch of issues

Re: [I] GlueCatalog: Set Glue table input information based on Iceberg table metadata [iceberg-python]

2024-01-20 Thread via GitHub
mgmarino commented on issue #216: URL: https://github.com/apache/iceberg-python/issues/216#issuecomment-1902055468 Ok, ready to go: #288. @nicor88 my changes with Location apparently still didn't allow `DROP TABLE` in Athena to clean up the resources and I didn't have a chance to inv

[PR] Set Glue Table Information when creating/updating tables [iceberg-python]

2024-01-20 Thread via GitHub
mgmarino opened a new pull request, #288: URL: https://github.com/apache/iceberg-python/pull/288 Resolves #216. This PR adds information about the schema (on update/create) and location (create) of the table to Glue, enabling both an improved UI experience as well as querying wit

Re: [I] GlueCatalog: Set Glue table input information based on Iceberg table metadata [iceberg-python]

2024-01-20 Thread via GitHub
nicor88 commented on issue #216: URL: https://github.com/apache/iceberg-python/issues/216#issuecomment-1901966386 > This is, I think, a more general issue with Athena, maybe Trino. This also fails for me with Iceberg tables created by Spark/Flink. I have some tables created via GlueJobs (

Re: [I] GlueCatalog: Set Glue table input information based on Iceberg table metadata [iceberg-python]

2024-01-20 Thread via GitHub
mgmarino commented on issue #216: URL: https://github.com/apache/iceberg-python/issues/216#issuecomment-1901965562 > I noticed another side effect on the above. After such table is created via pyicberg, when I drop it via Athena the s3 location is not cleaned up, only the table reference in

Re: [I] GlueCatalog: Set Glue table input information based on Iceberg table metadata [iceberg-python]

2024-01-20 Thread via GitHub
nicor88 commented on issue #216: URL: https://github.com/apache/iceberg-python/issues/216#issuecomment-1901960922 I noticed another side effect on the above. After such table is created via pyicberg, when I drop it via Athena the s3 location is not cleaned up, only the table reference in th

Re: [I] GlueCatalog: Set Glue table input information based on Iceberg table metadata [iceberg-python]

2024-01-20 Thread via GitHub
mgmarino commented on issue #216: URL: https://github.com/apache/iceberg-python/issues/216#issuecomment-1901956392 > > Also any query of this type `SHOW CREATE TABLE `test_iceberg_writing`;` in athena fails - even when the columns are added manually to glue. > This is, I think

Re: [I] GlueCatalog: Set Glue table input information based on Iceberg table metadata [iceberg-python]

2024-01-20 Thread via GitHub
mgmarino commented on issue #216: URL: https://github.com/apache/iceberg-python/issues/216#issuecomment-1901953276 FYI, I am preparing something here, I've already mostly translated the code from the java library. I hope to have something latest early next week, earlier if I can get some ti

Re: [I] GlueCatalog: Set Glue table input information based on Iceberg table metadata [iceberg-python]

2024-01-20 Thread via GitHub
nicor88 commented on issue #216: URL: https://github.com/apache/iceberg-python/issues/216#issuecomment-1901946836 I confirm the behvior that @mgmarino raised. After the table is created in glue catalog, the Schema section for the table is empty - as you can see in my screenshoot: https