Re: [PR] Docs: Update features for Hive 4.0 [iceberg]

2024-05-03 Thread via GitHub
SourabhBadhya commented on PR #10162: URL: https://github.com/apache/iceberg/pull/10162#issuecomment-2092426702 I have updated the PR. @pvary and @ajantha-bhat Can you please take a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] feat: add `ExpressionEvaluator` [iceberg-rust]

2024-05-03 Thread via GitHub
sdd commented on code in PR #363: URL: https://github.com/apache/iceberg-rust/pull/363#discussion_r1588925183 ## crates/iceberg/src/expr/visitors/expression_evaluator.rs: ## @@ -0,0 +1,819 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] feat: add `ExpressionEvaluator` [iceberg-rust]

2024-05-03 Thread via GitHub
sdd commented on code in PR #363: URL: https://github.com/apache/iceberg-rust/pull/363#discussion_r1588936688 ## crates/iceberg/src/expr/visitors/expression_evaluator.rs: ## @@ -0,0 +1,819 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] feat: add `ExpressionEvaluator` [iceberg-rust]

2024-05-03 Thread via GitHub
sdd commented on code in PR #363: URL: https://github.com/apache/iceberg-rust/pull/363#discussion_r1588936688 ## crates/iceberg/src/expr/visitors/expression_evaluator.rs: ## @@ -0,0 +1,819 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Implement table_exists method with try-catch for NoSuchTableError [iceberg-python]

2024-05-03 Thread via GitHub
MehulBatra commented on PR #678: URL: https://github.com/apache/iceberg-python/pull/678#issuecomment-2092638295 > Overall LGTM! The lint issue can be resolved by executing `make lint` Thank you @HonahX and @kevinjqliu for all the feedback! -- This is an automated message from the Ap

Re: [PR] Docs: Update features for Hive 4.0 [iceberg]

2024-05-03 Thread via GitHub
Fokko commented on code in PR #10162: URL: https://github.com/apache/iceberg/pull/10162#discussion_r1588974658 ## docs/docs/hive.md: ## @@ -24,6 +24,21 @@ Iceberg supports reading and writing Iceberg tables through [Hive](https://hive. a [StorageHandler](https://cwiki.apache.

Re: [PR] Docs: Update features for Hive 4.0 [iceberg]

2024-05-03 Thread via GitHub
SourabhBadhya commented on code in PR #10162: URL: https://github.com/apache/iceberg/pull/10162#discussion_r1588980182 ## docs/docs/hive.md: ## @@ -24,6 +24,21 @@ Iceberg supports reading and writing Iceberg tables through [Hive](https://hive. a [StorageHandler](https://cwiki

Re: [PR] Docs: Update features for Hive 4.0 [iceberg]

2024-05-03 Thread via GitHub
Fokko commented on code in PR #10162: URL: https://github.com/apache/iceberg/pull/10162#discussion_r1588982126 ## docs/docs/hive.md: ## @@ -431,12 +464,120 @@ ALTER TABLE t SET TBLPROPERTIES ('storage_handler'='org.apache.iceberg.mr.hive.H During the migration the data files a

Re: [PR] Docs: Update features for Hive 4.0 [iceberg]

2024-05-03 Thread via GitHub
SourabhBadhya commented on code in PR #10162: URL: https://github.com/apache/iceberg/pull/10162#discussion_r1588984815 ## docs/docs/hive.md: ## @@ -431,12 +464,120 @@ ALTER TABLE t SET TBLPROPERTIES ('storage_handler'='org.apache.iceberg.mr.hive.H During the migration the data

Re: [PR] Docs: Update features for Hive 4.0 [iceberg]

2024-05-03 Thread via GitHub
Fokko merged PR #10162: URL: https://github.com/apache/iceberg/pull/10162 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Build: Bump org.xerial:sqlite-jdbc from 3.45.2.0 to 3.45.3.0 [iceberg]

2024-05-03 Thread via GitHub
Fokko merged PR #10194: URL: https://github.com/apache/iceberg/pull/10194 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [I] Compatibility issues with `org.apache.iceberg:iceberg-spark-runtime-3.5_2.13:1.5.0` [iceberg-rust]

2024-05-03 Thread via GitHub
a-agmon commented on issue #338: URL: https://github.com/apache/iceberg-rust/issues/338#issuecomment-2092863958 Another way to resolve this, in a less workaround-ish way, is simply to capture the fact that we have a V1 schema, a V2 schema, and a V2 compatibility schema, which is identical t

Re: [I] Compatibility issues with `org.apache.iceberg:iceberg-spark-runtime-3.5_2.13:1.5.0` [iceberg-rust]

2024-05-03 Thread via GitHub
Fokko commented on issue #338: URL: https://github.com/apache/iceberg-rust/issues/338#issuecomment-2092909489 > @Fokko if https://github.com/apache/iceberg-rust/issues/354 is applied, iceberg-rust will no longer be able to read the manifest list files created by pre-1.5.0 Spark and pre-http

Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]

2024-05-03 Thread via GitHub
jbonofre commented on code in PR #10140: URL: https://github.com/apache/iceberg/pull/10140#discussion_r1589245337 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -40,6 +40,8 @@ final class JdbcUtil { // property to control if view support is added to the ex

Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]

2024-05-03 Thread via GitHub
jbonofre commented on code in PR #10140: URL: https://github.com/apache/iceberg/pull/10140#discussion_r1589246926 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java: ## @@ -21,17 +21,39 @@ import java.sql.Connection; import java.sql.DriverManager; import java.s

Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]

2024-05-03 Thread via GitHub
amogh-jahagirdar commented on code in PR #10140: URL: https://github.com/apache/iceberg/pull/10140#discussion_r1588457797 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -40,6 +40,8 @@ final class JdbcUtil { // property to control if view support is added t

Re: [PR] feat: add `ExpressionEvaluator` [iceberg-rust]

2024-05-03 Thread via GitHub
marvinlanhenke commented on code in PR #363: URL: https://github.com/apache/iceberg-rust/pull/363#discussion_r1589299148 ## crates/iceberg/src/expr/visitors/expression_evaluator.rs: ## @@ -0,0 +1,819 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more c

Re: [PR] Add ManifestFile Stats in snapshot summary. [iceberg]

2024-05-03 Thread via GitHub
jbonofre commented on code in PR #10246: URL: https://github.com/apache/iceberg/pull/10246#discussion_r1589368924 ## core/src/main/java/org/apache/iceberg/FastAppend.java: ## @@ -156,6 +156,8 @@ public List apply(TableMetadata base, Snapshot snapshot) { manifests.addAll(

Re: [I] Spark: Dropping partition column from old partition table corrupts entire table [iceberg]

2024-05-03 Thread via GitHub
manuzhang commented on issue #10234: URL: https://github.com/apache/iceberg/issues/10234#issuecomment-2093441393 The issue is we don't persist the partition spec along with its schema in `metadata.json` file and always use current schema when rebuilding partition spec from metadata file.

Re: [PR] Add Pagination To List Apis [iceberg]

2024-05-03 Thread via GitHub
rahil-c commented on PR #9782: URL: https://github.com/apache/iceberg/pull/9782#issuecomment-2093492003 Thanks @nastra and @danielcweeks for the help on this! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] Add ManifestFile Stats in snapshot summary. [iceberg]

2024-05-03 Thread via GitHub
nk1506 commented on code in PR #10246: URL: https://github.com/apache/iceberg/pull/10246#discussion_r1589621767 ## core/src/main/java/org/apache/iceberg/FastAppend.java: ## @@ -156,6 +156,8 @@ public List apply(TableMetadata base, Snapshot snapshot) { manifests.addAll(sn

Re: [PR] Implement table_exists method with try-catch for NoSuchTableError [iceberg-python]

2024-05-03 Thread via GitHub
kevinjqliu commented on PR #678: URL: https://github.com/apache/iceberg-python/pull/678#issuecomment-2093659238 Thank you for the PR! Do you mind changing the PR title and description to match? -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] AWS: Add AWS crt client support [iceberg]

2024-05-03 Thread via GitHub
sullis commented on PR #10217: URL: https://github.com/apache/iceberg/pull/10217#issuecomment-2093673973 I am planning to continue working on this PR next week. I'll add an update when I have time. -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [I] [feature request] Improve integration test reliance on docker [iceberg-python]

2024-05-03 Thread via GitHub
kevinjqliu commented on issue #637: URL: https://github.com/apache/iceberg-python/issues/637#issuecomment-2093680333 @AronsonDan I have not heard of this before. I'll take a look, thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [I] [feature request] Improve integration test reliance on docker [iceberg-python]

2024-05-03 Thread via GitHub
kevinjqliu commented on issue #637: URL: https://github.com/apache/iceberg-python/issues/637#issuecomment-2093713871 @AronsonDan please do! Feel free to tag me or reach out with any questions -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] OpenAPI: Add AppendDataFile models to openapi spec for fine grained metadata commits [iceberg]

2024-05-03 Thread via GitHub
dramaticlly commented on code in PR #10202: URL: https://github.com/apache/iceberg/pull/10202#discussion_r1589712093 ## open-api/rest-catalog-open-api.yaml: ## @@ -2893,6 +3003,37 @@ components: additionalProperties: type: string +AppendDataFileRequ

Re: [PR] Iceberg/Comet integration POC [iceberg]

2024-05-03 Thread via GitHub
aokolnychyi commented on code in PR #9841: URL: https://github.com/apache/iceberg/pull/9841#discussion_r1589703297 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java: ## @@ -27,6 +27,10 @@ private SparkSQLProperties() {} // Controls whether vec

Re: [PR] Iceberg/Comet integration POC [iceberg]

2024-05-03 Thread via GitHub
aokolnychyi commented on code in PR #9841: URL: https://github.com/apache/iceberg/pull/9841#discussion_r1589715889 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/SparkReadConf.java: ## @@ -184,7 +185,7 @@ public boolean orcVectorizationEnabled() { .parse();

Re: [PR] Add ManifestFile Stats in snapshot summary. [iceberg]

2024-05-03 Thread via GitHub
dramaticlly commented on code in PR #10246: URL: https://github.com/apache/iceberg/pull/10246#discussion_r1589716377 ## core/src/main/java/org/apache/iceberg/FastAppend.java: ## @@ -156,6 +156,8 @@ public List apply(TableMetadata base, Snapshot snapshot) { manifests.addA

Re: [PR] Iceberg/Comet integration POC [iceberg]

2024-05-03 Thread via GitHub
huaxingao commented on code in PR #9841: URL: https://github.com/apache/iceberg/pull/9841#discussion_r1589730054 ## spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/CometColumnReader.java: ## @@ -0,0 +1,164 @@ +/* + * Licensed to the Apache Software Founda

[PR] Build: Bump tqdm from 4.66.2 to 4.66.3 [iceberg-python]

2024-05-03 Thread via GitHub
dependabot[bot] opened a new pull request, #699: URL: https://github.com/apache/iceberg-python/pull/699 Bumps [tqdm](https://github.com/tqdm/tqdm) from 4.66.2 to 4.66.3. Release notes Sourced from https://github.com/tqdm/tqdm/releases";>tqdm's releases. tqdm v4.66.3 stable

Re: [PR] Core: Support rewriting manifests on branches [iceberg]

2024-05-03 Thread via GitHub
amogh-jahagirdar commented on code in PR #10265: URL: https://github.com/apache/iceberg/pull/10265#discussion_r1589798517 ## core/src/test/java/org/apache/iceberg/TestRewriteManifests.java: ## @@ -266,31 +280,38 @@ public void testReplaceManifestsWithFilter() throws IOException

Re: [I] Compatibility issues with `org.apache.iceberg:iceberg-spark-runtime-3.5_2.13:1.5.0` [iceberg-rust]

2024-05-03 Thread via GitHub
zeodtr commented on issue #338: URL: https://github.com/apache/iceberg-rust/issues/338#issuecomment-2093913951 @a-agmon Since the problem is in the schema, IMO checking the schema itself before reading the record is more appropriate. And since the error could be the other one (for example,

Re: [PR] table_exists unit/integration test for NoSuchTableError [iceberg-python]

2024-05-03 Thread via GitHub
HonahX merged PR #678: URL: https://github.com/apache/iceberg-python/pull/678 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] table_exists unit/integration test for NoSuchTableError [iceberg-python]

2024-05-03 Thread via GitHub
HonahX commented on PR #678: URL: https://github.com/apache/iceberg-python/pull/678#issuecomment-2093962497 Thanks everyone! Merged! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] define ray datasource for Iceberg tables [iceberg-python]

2024-05-03 Thread via GitHub
prabodh1194 commented on PR #698: URL: https://github.com/apache/iceberg-python/pull/698#issuecomment-2093965773 noticed a similar PR -- https://github.com/ray-project/ray/pull/45007 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi