Re: [PR] Address Intellij inspection findings [iceberg]

2024-07-03 Thread via GitHub
nastra commented on code in PR #10583: URL: https://github.com/apache/iceberg/pull/10583#discussion_r1663651011 ## api/src/main/java/org/apache/iceberg/PartitionSpec.java: ## @@ -410,7 +410,7 @@ private void checkAndAddPartitionName(String name, Integer sourceColumnId) {

[I] idea(catalog/rest): Separate static and dynamic props [iceberg-rust]

2024-07-03 Thread via GitHub
Xuanwo opened a new issue, #430: URL: https://github.com/apache/iceberg-rust/issues/430 Current reset catalog's init logic doesn't look good: ```rust pub async fn new(config: RestCatalogConfig) -> Result { let mut catalog = Self { client: config.try_create_rest_cli

[PR] core:Refactor the code of HadoopTableOptions [iceberg]

2024-07-03 Thread via GitHub
BsoBird opened a new pull request, #10623: URL: https://github.com/apache/iceberg/pull/10623 # Refactor the code of HadoopTableOptions ## 1.Current problems ### 1.1.Enabling `write.metadata.delete-after-commit.enabled` may result in dirty commits. Since this option is tu

Re: [PR] Address Intellij inspection findings [iceberg]

2024-07-03 Thread via GitHub
snazy commented on PR #10583: URL: https://github.com/apache/iceberg/pull/10583#issuecomment-2205375422 Repeating my note from above: IMHO no _functional_ or potentially _user facing_ changes should go into this PR. Going to rebase this once more due to the conflict. -- This is an auto

Re: [PR] View Spec implementation [iceberg-rust]

2024-07-03 Thread via GitHub
twuebi commented on PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#issuecomment-2205383749 Hi @nastra, this script: ```python3 import pyspark from pyspark.conf import SparkConf from pyspark.sql import SparkSession import pandas as pd CATALOG_URL

Re: [I] idea(catalog/rest): Separate static and dynamic props [iceberg-rust]

2024-07-03 Thread via GitHub
liurenjie1024 commented on issue #430: URL: https://github.com/apache/iceberg-rust/issues/430#issuecomment-2205398177 The problem is that the first `update_config` may return things that may change behavior of rest client, for example headers, or relocated url, IIRC. -- This is an automat

Re: [PR] View Spec implementation [iceberg-rust]

2024-07-03 Thread via GitHub
twuebi commented on PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#issuecomment-2205402299 Setting the namespace via: ```python3 spark.sql("USE spark_demo") spark.sql("CREATE view vv3 as select * from spark_demo.mytable") ``` ends up setting `default-name

Re: [PR] REST: assume issued token type is access token [iceberg]

2024-07-03 Thread via GitHub
adutra commented on PR #10314: URL: https://github.com/apache/iceberg/pull/10314#issuecomment-2205417586 @nastra @amogh-jahagirdar This PR has been rebased. Could I get a review please? 🙏 -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [I] idea(catalog/rest): Separate static and dynamic props [iceberg-rust]

2024-07-03 Thread via GitHub
Xuanwo commented on issue #430: URL: https://github.com/apache/iceberg-rust/issues/430#issuecomment-2205440671 Got it. I have a better idea, let's discuss on the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] REST: assume issued token type is access token [iceberg]

2024-07-03 Thread via GitHub
nastra commented on PR #10314: URL: https://github.com/apache/iceberg/pull/10314#issuecomment-2205480722 @adutra IMO https://github.com/apache/iceberg/pull/10314/files#r1601220154 still needs to be reverted. It's ok to add a workaround for Keycloak but I don't see a good enough reason to ad

Re: [PR] Address Intellij inspection findings [iceberg]

2024-07-03 Thread via GitHub
nastra commented on PR #10583: URL: https://github.com/apache/iceberg/pull/10583#issuecomment-2205486386 should be good to go once https://github.com/apache/iceberg/pull/10583/files#r1663651011 has been addressed -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Address Intellij inspection findings [iceberg]

2024-07-03 Thread via GitHub
snazy commented on PR #10583: URL: https://github.com/apache/iceberg/pull/10583#issuecomment-2205497949 > should be good to go once https://github.com/apache/iceberg/pull/10583/files#r1663651011 has been addressed Sorry, it's really out of scope of this PR ("Adress IntelliJ Inspectio

Re: [PR] View Spec implementation [iceberg-rust]

2024-07-03 Thread via GitHub
nastra commented on PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#issuecomment-2205517652 `"default-namespace": []` indicates an empty namespace, not a null one, which is a valid case and there are tests for that here: https://github.com/apache/iceberg/blob/42a2c19cec31c626c

Re: [PR] Address Intellij inspection findings [iceberg]

2024-07-03 Thread via GitHub
nastra commented on PR #10583: URL: https://github.com/apache/iceberg/pull/10583#issuecomment-2205538061 > > should be good to go once https://github.com/apache/iceberg/pull/10583/files#r1663651011 has been addressed > > Sorry, it's really out of scope of this PR ("Adress IntelliJ In

Re: [I] idea(catalog/rest): Separate static and dynamic props [iceberg-rust]

2024-07-03 Thread via GitHub
TennyZhuang commented on issue #430: URL: https://github.com/apache/iceberg-rust/issues/430#issuecomment-2205613572 BTW should we use a strong-typed Config/Props instead of a Hashmap? Do we have discussions before? -- This is an automated message from the Apache Git Service. To respon

Re: [PR] Address Intellij inspection findings [iceberg]

2024-07-03 Thread via GitHub
snazy commented on PR #10583: URL: https://github.com/apache/iceberg/pull/10583#issuecomment-2205653868 > We're improving the code and the null check is being removed "improving code" is very broad - and the null-check was never hit. Really - this PR is meant to address the intellij f

Re: [PR] View Spec implementation [iceberg-rust]

2024-07-03 Thread via GitHub
nastra commented on PR #331: URL: https://github.com/apache/iceberg-rust/pull/331#issuecomment-2205682556 Ok I checked the surrounding code and the handling is correct. In https://github.com/apache/iceberg/blob/2a39af894f4f00aa37922ef765cc2583517fa1d1/spark/v3.5/spark-extensions/src/main/sca

Re: [I] idea(catalog/rest): Separate static and dynamic props [iceberg-rust]

2024-07-03 Thread via GitHub
Xuanwo commented on issue #430: URL: https://github.com/apache/iceberg-rust/issues/430#issuecomment-2205737253 > BTW should we use a strong-typed Config/Props instead of a Hashmap? Do we have discussions before? I like this idea. Please feel free to start a discussion about it. Actual

Re: [PR] REST: assume issued token type is access token [iceberg]

2024-07-03 Thread via GitHub
adutra commented on PR #10314: URL: https://github.com/apache/iceberg/pull/10314#issuecomment-2205747465 > @adutra IMO https://github.com/apache/iceberg/pull/10314/files#r1601220154 still needs to be reverted. It's ok to add a workaround for Keycloak but I don't see a good enough reason to

Re: [PR] REST: refactor OAuth logic into AuthManager Interface [iceberg]

2024-07-03 Thread via GitHub
adutra commented on PR #10621: URL: https://github.com/apache/iceberg/pull/10621#issuecomment-2205792107 @jackye1995 Thank you so much for initiating this. I have been working on something similar as well, so it's great to see that there is a solid community engagement around this! A

Re: [PR] REST: assume issued token type is access token [iceberg]

2024-07-03 Thread via GitHub
nastra commented on PR #10314: URL: https://github.com/apache/iceberg/pull/10314#issuecomment-2205796889 @adutra I've read RFC 6749 but as I mentioned in https://github.com/apache/iceberg/pull/10314#discussion_r1601201951 I'm assuming that the token exchange is following RFC 8693 (and my as

Re: [PR] Address Intellij inspection findings [iceberg]

2024-07-03 Thread via GitHub
nastra commented on code in PR #10583: URL: https://github.com/apache/iceberg/pull/10583#discussion_r1664024915 ## api/src/main/java/org/apache/iceberg/PartitionSpec.java: ## @@ -410,7 +410,7 @@ private void checkAndAddPartitionName(String name, Integer sourceColumnId) {

Re: [PR] REST: refactor OAuth logic into AuthManager Interface [iceberg]

2024-07-03 Thread via GitHub
adutra commented on code in PR #10621: URL: https://github.com/apache/iceberg/pull/10621#discussion_r1664022154 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Manager.java: ## @@ -0,0 +1,241 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] REST: assume issued token type is access token [iceberg]

2024-07-03 Thread via GitHub
adutra commented on PR #10314: URL: https://github.com/apache/iceberg/pull/10314#issuecomment-2205848299 > @adutra I've read RFC 6749 but as I mentioned in [#10314 (comment)](https://github.com/apache/iceberg/pull/10314#discussion_r1601201951) I'm assuming that the token exchange is followi

Re: [PR] REST: assume issued token type is access token [iceberg]

2024-07-03 Thread via GitHub
adutra commented on PR #10314: URL: https://github.com/apache/iceberg/pull/10314#issuecomment-2205866224 @rdblue since you originally authored this part (https://github.com/apache/iceberg/pull/4771#discussion_r874114263) would you mind having a look as well? Thank you! -- This is an auto

Re: [PR] REST: assume issued token type is access token [iceberg]

2024-07-03 Thread via GitHub
snazy commented on PR #10314: URL: https://github.com/apache/iceberg/pull/10314#issuecomment-2205870528 @nastra Alex is 100% right. I do not understand why it takes so long to review this fix for an OAuth spec-compliance. -- This is an automated message from the Apache Git Service. To res

Re: [PR] chore: Don't enable reqwest default features [iceberg-rust]

2024-07-03 Thread via GitHub
liurenjie1024 merged PR #432: URL: https://github.com/apache/iceberg-rust/pull/432 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Flink: Pre-create fieldGetters to avoid constructing them for each row [iceberg]

2024-07-03 Thread via GitHub
fengjiajie commented on PR #10565: URL: https://github.com/apache/iceberg/pull/10565#issuecomment-2205930321 Hi @nastra , the failing test cases seem unrelated to this change. Could you please re-trigger the tests? Thanks! -- This is an automated message from the Apache Git Service. To re

Re: [PR] Kafka Connect: Commit coordination [iceberg]

2024-07-03 Thread via GitHub
fqaiser94 commented on code in PR #10351: URL: https://github.com/apache/iceberg/pull/10351#discussion_r1663401716 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/KafkaUtils.java: ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundat

Re: [PR] Kafka Connect: Commit coordination [iceberg]

2024-07-03 Thread via GitHub
fqaiser94 commented on code in PR #10351: URL: https://github.com/apache/iceberg/pull/10351#discussion_r1663401716 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/KafkaUtils.java: ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundat

Re: [PR] Address Intellij inspection findings [iceberg]

2024-07-03 Thread via GitHub
nastra merged PR #10583: URL: https://github.com/apache/iceberg/pull/10583 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Enable the Gradle build cache [iceberg]

2024-07-03 Thread via GitHub
nastra merged PR #10602: URL: https://github.com/apache/iceberg/pull/10602 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Core: Refactor ZOrderByteUtils [iceberg]

2024-07-03 Thread via GitHub
ajantha-bhat commented on PR #10624: URL: https://github.com/apache/iceberg/pull/10624#issuecomment-2206021634 cc: @snazy, @nastra -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Flink: Pre-create fieldGetters to avoid constructing them for each row [iceberg]

2024-07-03 Thread via GitHub
Fokko commented on PR #10565: URL: https://github.com/apache/iceberg/pull/10565#issuecomment-2206068288 @fengjiajie Rerunning the failed tests 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] Bump pypa/cibuildwheel from 2.19.1 to 2.19.2 [iceberg-python]

2024-07-03 Thread via GitHub
Fokko merged PR #889: URL: https://github.com/apache/iceberg-python/pull/889 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Kafka Connect: Commit coordination [iceberg]

2024-07-03 Thread via GitHub
fqaiser94 commented on code in PR #10351: URL: https://github.com/apache/iceberg/pull/10351#discussion_r1664211281 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/IcebergSinkTask.java: ## @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundatio

Re: [PR] Kafka Connect: Commit coordination [iceberg]

2024-07-03 Thread via GitHub
fqaiser94 commented on code in PR #10351: URL: https://github.com/apache/iceberg/pull/10351#discussion_r1663397086 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/Worker.java: ## @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation

Re: [I] Spark: Support alter partition in V2 Catalog SparkCatalog Class [iceberg]

2024-07-03 Thread via GitHub
psilos commented on issue #3558: URL: https://github.com/apache/iceberg/issues/3558#issuecomment-2206288118 hi @huaxingao I was probably referring to the later one, adding it on the iceberg extended syntax. -- This is an automated message from the Apache Git Service. To respond to the me

Re: [PR] REST: refactor OAuth logic into AuthManager Interface [iceberg]

2024-07-03 Thread via GitHub
jackye1995 commented on PR #10621: URL: https://github.com/apache/iceberg/pull/10621#issuecomment-2206480833 > How are you going to migrate RESTSigV4Signer? Yes that's the thing I am planning to prototype today. My position is that, if it fits then it fits, if it does not let's not tr

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-07-03 Thread via GitHub
nastra commented on PR #9008: URL: https://github.com/apache/iceberg/pull/9008#issuecomment-2206491863 Hey @jacobmarble thanks for being patient here. There's no particular concern but I've been busy with other things and it was difficult to find a big block of time to re-review this PR. I'

Re: [PR] REST: refactor OAuth logic into AuthManager Interface [iceberg]

2024-07-03 Thread via GitHub
flyrain commented on PR #10621: URL: https://github.com/apache/iceberg/pull/10621#issuecomment-2206504324 Thanks @jackye1995 for pinging me. Will take a look today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] OpenAPI: Express server capabilities via /config endpoint [iceberg]

2024-07-03 Thread via GitHub
jackye1995 commented on code in PR #9940: URL: https://github.com/apache/iceberg/pull/9940#discussion_r1664416515 ## open-api/rest-catalog-open-api.yaml: ## @@ -28,6 +28,10 @@ info: description: Defines the specification for the first version of the REST Catalog API.

Re: [PR] OpenAPI: Express server capabilities via /config endpoint [iceberg]

2024-07-03 Thread via GitHub
jackye1995 commented on code in PR #9940: URL: https://github.com/apache/iceberg/pull/9940#discussion_r1664416515 ## open-api/rest-catalog-open-api.yaml: ## @@ -28,6 +28,10 @@ info: description: Defines the specification for the first version of the REST Catalog API.

Re: [PR] OpenAPI: Express server capabilities via /config endpoint [iceberg]

2024-07-03 Thread via GitHub
jackye1995 commented on code in PR #9940: URL: https://github.com/apache/iceberg/pull/9940#discussion_r1664421615 ## open-api/rest-catalog-open-api.yaml: ## @@ -191,7 +220,8 @@ paths: get: tags: -- Catalog API +- tables +- views Review Comm

Re: [PR] OpenAPI: Express server capabilities via /config endpoint [iceberg]

2024-07-03 Thread via GitHub
jackye1995 commented on code in PR #9940: URL: https://github.com/apache/iceberg/pull/9940#discussion_r1664416515 ## open-api/rest-catalog-open-api.yaml: ## @@ -28,6 +28,10 @@ info: description: Defines the specification for the first version of the REST Catalog API.

Re: [PR] OpenAPI: Express server capabilities via /config endpoint [iceberg]

2024-07-03 Thread via GitHub
jackye1995 commented on code in PR #9940: URL: https://github.com/apache/iceberg/pull/9940#discussion_r1664441868 ## open-api/rest-catalog-open-api.yaml: ## @@ -191,7 +220,8 @@ paths: get: tags: -- Catalog API +- tables +- views Review Comm

Re: [PR] REST: assume issued token type is access token [iceberg]

2024-07-03 Thread via GitHub
danielcweeks commented on PR #10314: URL: https://github.com/apache/iceberg/pull/10314#issuecomment-2206759806 I think the application of extensions referenced in RFC 8693 are a little ambiguous due to the following: `RFC 6749` section 4.1 references the response described in [section

Re: [PR] REST: assume issued token type is access token [iceberg]

2024-07-03 Thread via GitHub
snazy commented on PR #10314: URL: https://github.com/apache/iceberg/pull/10314#issuecomment-2206790685 [@nastra requested to revert](https://github.com/apache/iceberg/pull/10314#discussion_r1601220154) @adutra 's fix for the client-credentials flow, only RFC 6749 applies here. This part o

Re: [PR] Add pool_pre_ping param to SQLCatalog and fix echo parsing logic [iceberg-python]

2024-07-03 Thread via GitHub
kevinjqliu commented on code in PR #886: URL: https://github.com/apache/iceberg-python/pull/886#discussion_r1664510351 ## pyiceberg/catalog/sql.py: ## @@ -110,8 +114,14 @@ def __init__(self, name: str, **properties: str): if not (uri_prop := self.properties.get("uri")

Re: [PR] Add pool_pre_ping param to SQLCatalog and fix echo parsing logic [iceberg-python]

2024-07-03 Thread via GitHub
cccs-eric commented on code in PR #886: URL: https://github.com/apache/iceberg-python/pull/886#discussion_r1664515537 ## pyiceberg/catalog/sql.py: ## @@ -110,8 +114,14 @@ def __init__(self, name: str, **properties: str): if not (uri_prop := self.properties.get("uri"))

Re: [PR] REST: assume issued token type is access token [iceberg]

2024-07-03 Thread via GitHub
adutra commented on PR #10314: URL: https://github.com/apache/iceberg/pull/10314#issuecomment-2206868094 > I believe that the intent includes that a client credential exchange could return any of the enumerated token types defined in [section 3](https://www.rfc-editor.org/rfc/rfc8693.html#n

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-07-03 Thread via GitHub
amogh-jahagirdar commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1664436827 ## api/src/main/java/org/apache/iceberg/transforms/Timestamps.java: ## @@ -31,54 +33,160 @@ import org.apache.iceberg.util.DateTimeUtil; import org.apache.ic

Re: [PR] Add pool_pre_ping param to SQLCatalog and fix echo parsing logic [iceberg-python]

2024-07-03 Thread via GitHub
kevinjqliu commented on code in PR #886: URL: https://github.com/apache/iceberg-python/pull/886#discussion_r1664532945 ## pyiceberg/catalog/sql.py: ## @@ -110,8 +114,12 @@ def __init__(self, name: str, **properties: str): if not (uri_prop := self.properties.get("uri")

Re: [PR] Add pool_pre_ping param to SQLCatalog and fix echo parsing logic [iceberg-python]

2024-07-03 Thread via GitHub
cccs-eric commented on code in PR #886: URL: https://github.com/apache/iceberg-python/pull/886#discussion_r1664545780 ## pyiceberg/catalog/sql.py: ## @@ -110,8 +114,12 @@ def __init__(self, name: str, **properties: str): if not (uri_prop := self.properties.get("uri"))

Re: [PR] feat(visitors): Implement basic boolean expression visitors [iceberg-go]

2024-07-03 Thread via GitHub
zeroshade commented on PR #108: URL: https://github.com/apache/iceberg-go/pull/108#issuecomment-2206884654 CC @Fokko @nastra @wolfeidau -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] Docs: Update defaults for distribution mode [iceberg]

2024-07-03 Thread via GitHub
dramaticlly commented on code in PR #10575: URL: https://github.com/apache/iceberg/pull/10575#discussion_r1664551015 ## docs/docs/configuration.md: ## @@ -67,7 +67,7 @@ Iceberg tables support table properties to configure table behavior, like the de | write.metadata.metrics.co

Re: [PR] Spark Action to Analyze table [iceberg]

2024-07-03 Thread via GitHub
karuppayya commented on code in PR #10288: URL: https://github.com/apache/iceberg/pull/10288#discussion_r1664540073 ## spark/v3.5/build.gradle: ## @@ -59,6 +59,7 @@ project(":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}") { implementation project(':iceb

Re: [PR] REST: refactor OAuth logic into AuthManager Interface [iceberg]

2024-07-03 Thread via GitHub
adutra commented on PR #10621: URL: https://github.com/apache/iceberg/pull/10621#issuecomment-2206891162 > Yes that's the thing I am planning to prototype today. My position is that, if it fits then it fits, if it does not let's not try to fit it. I explored a bit deeper today, includ

Re: [PR] Add pool_pre_ping param to SQLCatalog and fix echo parsing logic [iceberg-python]

2024-07-03 Thread via GitHub
cccs-eric commented on code in PR #886: URL: https://github.com/apache/iceberg-python/pull/886#discussion_r1664555741 ## tests/catalog/test_sql.py: ## @@ -168,6 +168,39 @@ def test_creation_with_unsupported_uri(catalog_name: str) -> None: SqlCatalog(catalog_name, uri="

Re: [PR] REST: assume issued token type is access token [iceberg]

2024-07-03 Thread via GitHub
danielcweeks commented on PR #10314: URL: https://github.com/apache/iceberg/pull/10314#issuecomment-2206897363 @adutra > An RFC cannot modify another one's structs without officially superseding it. > . . . RFC 8693 as a general rewrite of RFC 6749 section 5.1 I'm not sugg

Re: [PR] REST: refactor OAuth logic into AuthManager Interface [iceberg]

2024-07-03 Thread via GitHub
adutra commented on code in PR #10621: URL: https://github.com/apache/iceberg/pull/10621#discussion_r1664568478 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Manager.java: ## @@ -0,0 +1,241 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or m

Re: [PR] Add pool_pre_ping param to SQLCatalog and fix echo parsing logic [iceberg-python]

2024-07-03 Thread via GitHub
kevinjqliu commented on code in PR #886: URL: https://github.com/apache/iceberg-python/pull/886#discussion_r1664577534 ## tests/catalog/test_sql.py: ## @@ -168,6 +168,39 @@ def test_creation_with_unsupported_uri(catalog_name: str) -> None: SqlCatalog(catalog_name, uri=

Re: [PR] WIP: Initial Support for Spark 4.0 [iceberg]

2024-07-03 Thread via GitHub
amogh-jahagirdar commented on PR #10622: URL: https://github.com/apache/iceberg/pull/10622#issuecomment-2206939528 Updated the description to link #10497, this is great to see @huaxingao -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] Spark: support read of partition metadata column when table is over 1k [iceberg]

2024-07-03 Thread via GitHub
szehon-ho commented on code in PR #10547: URL: https://github.com/apache/iceberg/pull/10547#discussion_r1664595395 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -366,15 +371,56 @@ public void pruneColumns(StructType requestedSchem

Re: [PR] Spark: support read of partition metadata column when table is over 1k [iceberg]

2024-07-03 Thread via GitHub
szehon-ho commented on code in PR #10547: URL: https://github.com/apache/iceberg/pull/10547#discussion_r1664595395 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -366,15 +371,56 @@ public void pruneColumns(StructType requestedSchem

Re: [PR] Apply IntelliJ inspection findings to older Spark + Flink versions [iceberg]

2024-07-03 Thread via GitHub
snazy commented on PR #10625: URL: https://github.com/apache/iceberg/pull/10625#issuecomment-2206956499 /cc @nastra -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [PR] Core: Support appending files with different specs [iceberg]

2024-07-03 Thread via GitHub
fqaiser94 commented on PR #9860: URL: https://github.com/apache/iceberg/pull/9860#issuecomment-2206964353 Simplified the PR to just multi-partition-append use case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Kafka Connect: Commit coordination [iceberg]

2024-07-03 Thread via GitHub
fqaiser94 commented on code in PR #10351: URL: https://github.com/apache/iceberg/pull/10351#discussion_r1664602970 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/Coordinator.java: ## @@ -0,0 +1,294 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Spark: support read of partition metadata column when table is over 1k [iceberg]

2024-07-03 Thread via GitHub
dramaticlly commented on code in PR #10547: URL: https://github.com/apache/iceberg/pull/10547#discussion_r1664606896 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -366,15 +371,56 @@ public void pruneColumns(StructType requestedSch

Re: [PR] DO-NOT-MERGE: Jackson access issue [iceberg]

2024-07-03 Thread via GitHub
pvary commented on PR #10460: URL: https://github.com/apache/iceberg/pull/10460#issuecomment-2206994528 > @pvary can we disable CBO in Iceberg Hive test? otherwise, we can not upgrade Hive 2.3.10. Disabling the test would just hide the issue that Hive 2.3.10 is not supported with CBO

Re: [PR] Bump mkdocs-material from 9.5.27 to 9.5.28 [iceberg-python]

2024-07-03 Thread via GitHub
Fokko merged PR #888: URL: https://github.com/apache/iceberg-python/pull/888 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Bump PyIceberg in CI to 0.6.1 [iceberg-python]

2024-07-03 Thread via GitHub
Fokko commented on PR #879: URL: https://github.com/apache/iceberg-python/pull/879#issuecomment-2207011586 @kevinjqliu That's an excellent suggestion, just added it 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] Flink: Pre-create fieldGetters to avoid constructing them for each row [iceberg]

2024-07-03 Thread via GitHub
pvary commented on code in PR #10565: URL: https://github.com/apache/iceberg/pull/10565#discussion_r1664639363 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/data/RowDataUtil.java: ## @@ -79,7 +79,11 @@ public static Object convertConstant(Type type, Object value)

Re: [PR] Flink: Pre-create fieldGetters to avoid constructing them for each row [iceberg]

2024-07-03 Thread via GitHub
pvary commented on code in PR #10565: URL: https://github.com/apache/iceberg/pull/10565#discussion_r1664640913 ## flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/reader/RowDataRecordFactory.java: ## @@ -40,6 +42,14 @@ static TypeSerializer[] createFieldSerializer

Re: [PR] Flink: Pre-create fieldGetters to avoid constructing them for each row [iceberg]

2024-07-03 Thread via GitHub
pvary commented on code in PR #10565: URL: https://github.com/apache/iceberg/pull/10565#discussion_r1664641504 ## flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/TestHelpers.java: ## @@ -87,7 +87,11 @@ public static RowData copyRowData(RowData from, RowType rowType) {

Re: [PR] Flink: Pre-create fieldGetters to avoid constructing them for each row [iceberg]

2024-07-03 Thread via GitHub
pvary commented on PR #10565: URL: https://github.com/apache/iceberg/pull/10565#issuecomment-2207028521 Thanks for the detailed explanation and the PR @fengjiajie! Left a few small comments. As a general rule we create a PR for a single version. Usually the latest. So if there

Re: [PR] Forward Compatible large_* type support: read as large, write as small [iceberg-python]

2024-07-03 Thread via GitHub
syun64 commented on PR #890: URL: https://github.com/apache/iceberg-python/pull/890#issuecomment-2207040653 This is almost ready - I'm going to introduce the flag to all read API as well, so that users can control which schema they are using to read their pyarrow tables, as they may be usin

Re: [PR] Support for Flink's SpeculativeExecution in batch execution mode [iceberg]

2024-07-03 Thread via GitHub
pvary commented on PR #10548: URL: https://github.com/apache/iceberg/pull/10548#issuecomment-2207052299 @venkata91: How can we be sure that the tests are exercising the speculative execution code path? Does any of the tests reads some splits multiple times, and use the result of the

[I] NullPointerException after deleting old partition column [iceberg]

2024-07-03 Thread via GitHub
mgmarino opened a new issue, #10626: URL: https://github.com/apache/iceberg/issues/10626 ### Apache Iceberg version 1.5.2 (latest release) ### Query engine Spark ### Please describe the bug 🐞 We have an iceberg table where we have changed the partitioning, g

Re: [PR] Add pool_pre_ping param to SQLCatalog and fix echo parsing logic [iceberg-python]

2024-07-03 Thread via GitHub
Fokko commented on code in PR #886: URL: https://github.com/apache/iceberg-python/pull/886#discussion_r1664804723 ## mkdocs/docs/configuration.md: ## @@ -222,6 +222,12 @@ catalog: uri: sqlite:tmp/pyiceberg.db ``` +| Key | Example

Re: [PR] Use SupportsPrefixOperations for Remove OrphanFile Procedure [iceberg]

2024-07-03 Thread via GitHub
schobe commented on PR #7914: URL: https://github.com/apache/iceberg/pull/7914#issuecomment-2207301578 Hi , I am also facing the same issue while running orphan file clean up via Nessie REST. Auto-compaction and snapshot expiry works, but orphan file clean up procecure gives the same error.

Re: [PR] Add pool_pre_ping param to SQLCatalog and fix echo parsing logic [iceberg-python]

2024-07-03 Thread via GitHub
cccs-eric commented on code in PR #886: URL: https://github.com/apache/iceberg-python/pull/886#discussion_r1664857704 ## mkdocs/docs/configuration.md: ## @@ -222,6 +222,12 @@ catalog: uri: sqlite:tmp/pyiceberg.db ``` +| Key | Example

Re: [PR] REST: refactor OAuth logic into AuthManager Interface [iceberg]

2024-07-03 Thread via GitHub
amogh-jahagirdar commented on code in PR #10621: URL: https://github.com/apache/iceberg/pull/10621#discussion_r1664893504 ## core/src/main/java/org/apache/iceberg/rest/auth/AuthSession.java: ## @@ -0,0 +1,28 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

Re: [PR] Fix some `long` casting issues [iceberg]

2024-07-03 Thread via GitHub
amogh-jahagirdar commented on code in PR #10580: URL: https://github.com/apache/iceberg/pull/10580#discussion_r1664908939 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/data/FlinkParquetReaders.java: ## @@ -422,7 +422,7 @@ public TimestampData read(TimestampData ign

Re: [PR] Fix some `long` casting issues [iceberg]

2024-07-03 Thread via GitHub
amogh-jahagirdar merged PR #10580: URL: https://github.com/apache/iceberg/pull/10580 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [I] Allow snapshotting iceberg table (create new table based on certain Iceberg snapshot) [iceberg]

2024-07-03 Thread via GitHub
github-actions[bot] commented on issue #2481: URL: https://github.com/apache/iceberg/issues/2481#issuecomment-2207609767 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Avro inferred mapping issues with nested struct [iceberg]

2024-07-03 Thread via GitHub
github-actions[bot] commented on issue #2774: URL: https://github.com/apache/iceberg/issues/2774#issuecomment-2207610352 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Avro inferred mapping issues with nested struct [iceberg]

2024-07-03 Thread via GitHub
github-actions[bot] closed issue #2774: Avro inferred mapping issues with nested struct URL: https://github.com/apache/iceberg/issues/2774 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [I] Drop table doesn't clean up metadata and data files [iceberg]

2024-07-03 Thread via GitHub
github-actions[bot] commented on issue #2784: URL: https://github.com/apache/iceberg/issues/2784#issuecomment-2207610460 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Spark3 DSv2: Handle Snapshots of Type OVERWRITE - while reading Stream from Iceberg table [iceberg]

2024-07-03 Thread via GitHub
github-actions[bot] commented on issue #2788: URL: https://github.com/apache/iceberg/issues/2788#issuecomment-2207610565 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] NullPointerException after deleting old partition column [iceberg]

2024-07-03 Thread via GitHub
dramaticlly commented on issue #10626: URL: https://github.com/apache/iceberg/issues/10626#issuecomment-2207648800 Based on the stacktrace, looks like partitions table need to evaluate all historical partition specs to build the partition column per #7551 , but since the column is alread

Re: [I] NullPointerException after deleting old partition column [iceberg]

2024-07-03 Thread via GitHub
lurnagao-dahua commented on issue #10626: URL: https://github.com/apache/iceberg/issues/10626#issuecomment-2207685799 Hi, same issues in [10234](https://github.com/apache/iceberg/issues/10234) -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] core:Refactor the code of HadoopTableOptions [iceberg]

2024-07-03 Thread via GitHub
BsoBird commented on PR #10623: URL: https://github.com/apache/iceberg/pull/10623#issuecomment-2207862734 @nastra @Fokko @RussellSpitzer @rdblue @pvary hello. If someone could help me review this PR, that would be great. Tks. -- This is an automated message from the Apache Git Service.

Re: [PR] refactor(catalogs/rest): Split user config and runtime config [iceberg-rust]

2024-07-03 Thread via GitHub
liurenjie1024 commented on code in PR #431: URL: https://github.com/apache/iceberg-rust/pull/431#discussion_r1664977067 ## crates/catalog/rest/Cargo.toml: ## @@ -32,13 +32,15 @@ keywords = ["iceberg", "rest", "catalog"] # async-trait = { workspace = true } async-trait = { work

Re: [I] idea(catalog/rest): Separate static and dynamic props [iceberg-rust]

2024-07-03 Thread via GitHub
liurenjie1024 commented on issue #430: URL: https://github.com/apache/iceberg-rust/issues/430#issuecomment-2207930775 > BTW should we use a strong-typed Config/Props instead of a Hashmap? Do we have discussions before? It seems that there is no clear definition of all props. -- Thi

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-07-03 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1665080053 ## api/src/main/java/org/apache/iceberg/types/Types.java: ## @@ -259,6 +261,60 @@ public int hashCode() { } } + public static class TimestampNanoType extends P

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-07-03 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1665080227 ## api/src/test/java/org/apache/iceberg/TestPartitionPaths.java: ## @@ -54,6 +54,25 @@ public void testPartitionPath() { .isEqualTo("ts_hour=2017-12-01-10/id_buc

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-07-03 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1665079821 ## api/src/main/java/org/apache/iceberg/transforms/Timestamps.java: ## @@ -31,54 +33,160 @@ import org.apache.iceberg.util.DateTimeUtil; import org.apache.iceberg.util.

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-07-03 Thread via GitHub
epgif commented on code in PR #9008: URL: https://github.com/apache/iceberg/pull/9008#discussion_r1665079938 ## api/src/main/java/org/apache/iceberg/expressions/ExpressionUtil.java: ## @@ -594,6 +605,12 @@ private static String sanitizeString(CharSequence value, long now, int t

Re: [PR] API: implement types timestamp_ns and timestamptz_ns [iceberg]

2024-07-03 Thread via GitHub
epgif commented on PR #9008: URL: https://github.com/apache/iceberg/pull/9008#issuecomment-2208073422 Thanks @amogh-jahagirdar! I believe I've addressed all the comments. Please note that @jacobmarble and I are both in the US and will be out of the office Thursday and Friday. W

  1   2   >