Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-27 Thread via GitHub
liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1582042879 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lice

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-27 Thread via GitHub
liurenjie1024 commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1582042661 ## crates/integrations/datafusion/src/catalog.rs: ## @@ -0,0 +1,94 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-27 Thread via GitHub
viirya commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1582031879 ## crates/integrations/datafusion/src/schema.rs: ## @@ -0,0 +1,95 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license ag

Re: [PR] Build: Bump nessie from 0.79.0 to 0.80.0 [iceberg]

2024-04-27 Thread via GitHub
Fokko merged PR #10237: URL: https://github.com/apache/iceberg/pull/10237 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-27 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1582017255 ## crates/iceberg/src/arrow/reader.rs: ## @@ -186,4 +216,399 @@ impl ArrowReader { Ok(ProjectionMask::leaves(parquet_schema, indices)) } }

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-27 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1582017255 ## crates/iceberg/src/arrow/reader.rs: ## @@ -186,4 +216,399 @@ impl ArrowReader { Ok(ProjectionMask::leaves(parquet_schema, indices)) } }

Re: [PR] Build: Bump mkdocs-material from 9.5.18 to 9.5.19 [iceberg-python]

2024-04-27 Thread via GitHub
Fokko merged PR #663: URL: https://github.com/apache/iceberg-python/pull/663 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

[PR] Build: Bump datamodel-code-generator from 0.25.5 to 0.25.6 [iceberg]

2024-04-27 Thread via GitHub
dependabot[bot] opened a new pull request, #10242: URL: https://github.com/apache/iceberg/pull/10242 Bumps [datamodel-code-generator](https://github.com/koxudaxi/datamodel-code-generator) from 0.25.5 to 0.25.6. Release notes Sourced from https://github.com/koxudaxi/datamodel-code-

[PR] Build: Bump mkdocs-material from 9.5.18 to 9.5.19 [iceberg]

2024-04-27 Thread via GitHub
dependabot[bot] opened a new pull request, #10241: URL: https://github.com/apache/iceberg/pull/10241 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.18 to 9.5.19. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mkdoc

[PR] Build: Bump software.amazon.awssdk:bom from 2.25.35 to 2.25.40 [iceberg]

2024-04-27 Thread via GitHub
dependabot[bot] opened a new pull request, #10240: URL: https://github.com/apache/iceberg/pull/10240 Bumps software.amazon.awssdk:bom from 2.25.35 to 2.25.40. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=soft

[PR] Build: Bump com.gorylenko.gradle-git-properties:gradle-git-properties from 2.4.1 to 2.4.2 [iceberg]

2024-04-27 Thread via GitHub
dependabot[bot] opened a new pull request, #10239: URL: https://github.com/apache/iceberg/pull/10239 Bumps com.gorylenko.gradle-git-properties:gradle-git-properties from 2.4.1 to 2.4.2. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibilit

[PR] Build: Bump nessie from 0.79.0 to 0.80.0 [iceberg]

2024-04-27 Thread via GitHub
dependabot[bot] opened a new pull request, #10237: URL: https://github.com/apache/iceberg/pull/10237 Bumps `nessie` from 0.79.0 to 0.80.0. Updates `org.projectnessie.nessie:nessie-client` from 0.79.0 to 0.80.0 Updates `org.projectnessie.nessie:nessie-jaxrs-testextension` from 0.79.0

[PR] Build: Bump com.azure:azure-sdk-bom from 1.2.22 to 1.2.23 [iceberg]

2024-04-27 Thread via GitHub
dependabot[bot] opened a new pull request, #10238: URL: https://github.com/apache/iceberg/pull/10238 Bumps [com.azure:azure-sdk-bom](https://github.com/azure/azure-sdk-for-java) from 1.2.22 to 1.2.23. Release notes Sourced from https://github.com/azure/azure-sdk-for-java/releases";

[PR] Build: Bump com.google.errorprone:error_prone_annotations from 2.26.1 to 2.27.0 [iceberg]

2024-04-27 Thread via GitHub
dependabot[bot] opened a new pull request, #10236: URL: https://github.com/apache/iceberg/pull/10236 Bumps [com.google.errorprone:error_prone_annotations](https://github.com/google/error-prone) from 2.26.1 to 2.27.0. Release notes Sourced from https://github.com/google/error-prone

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-27 Thread via GitHub
viirya commented on PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#issuecomment-2081289882 @liurenjie1024 Thanks for review. Sorry for late. I addressed the comments by rewriting the visitors using the new API. I replied with another questions. -- This is an automated messa

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-27 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1581996145 ## crates/iceberg/src/arrow/reader.rs: ## @@ -186,4 +216,399 @@ impl ArrowReader { Ok(ProjectionMask::leaves(parquet_schema, indices)) } }

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-27 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1581995193 ## crates/iceberg/src/arrow/reader.rs: ## @@ -186,4 +216,399 @@ impl ArrowReader { Ok(ProjectionMask::leaves(parquet_schema, indices)) } }

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-27 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1581994781 ## crates/iceberg/src/arrow/reader.rs: ## @@ -186,4 +216,399 @@ impl ArrowReader { Ok(ProjectionMask::leaves(parquet_schema, indices)) } }

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-27 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1581994749 ## crates/iceberg/src/arrow/reader.rs: ## @@ -186,4 +216,399 @@ impl ArrowReader { Ok(ProjectionMask::leaves(parquet_schema, indices)) } }

Re: [PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-04-27 Thread via GitHub
viirya commented on code in PR #295: URL: https://github.com/apache/iceberg-rust/pull/295#discussion_r1581994744 ## crates/iceberg/src/arrow/reader.rs: ## @@ -186,4 +216,399 @@ impl ArrowReader { Ok(ProjectionMask::leaves(parquet_schema, indices)) } }

Re: [I] Spark SQL extension syntax for identifier field update [iceberg]

2024-04-27 Thread via GitHub
github-actions[bot] commented on issue #2724: URL: https://github.com/apache/iceberg/issues/2724#issuecomment-2081259388 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Support for low latency writes to iceberg table [iceberg]

2024-04-27 Thread via GitHub
github-actions[bot] commented on issue #2723: URL: https://github.com/apache/iceberg/issues/2723#issuecomment-2081259380 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Clean backport with:Flink: Backport #10207 to v1.18 and v1.17 [iceberg]

2024-04-27 Thread via GitHub
pvary commented on PR #10235: URL: https://github.com/apache/iceberg/pull/10235#issuecomment-2081230888 Thanks for the review and the merge @stevenzwu! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Build: Bump mkdocstrings from 0.24.0 to 0.24.3 [iceberg-python]

2024-04-27 Thread via GitHub
dependabot[bot] closed pull request #661: Build: Bump mkdocstrings from 0.24.0 to 0.24.3 URL: https://github.com/apache/iceberg-python/pull/661 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] Build: Bump mkdocstrings from 0.24.0 to 0.24.3 [iceberg-python]

2024-04-27 Thread via GitHub
dependabot[bot] commented on PR #661: URL: https://github.com/apache/iceberg-python/pull/661#issuecomment-2081165993 Superseded by #664. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Build: Bump griffe from 0.40.1 to 0.44.0 [iceberg-python]

2024-04-27 Thread via GitHub
Fokko merged PR #660: URL: https://github.com/apache/iceberg-python/pull/660 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

[PR] Build: Bump mkdocstrings from 0.24.0 to 0.25.0 [iceberg-python]

2024-04-27 Thread via GitHub
dependabot[bot] opened a new pull request, #664: URL: https://github.com/apache/iceberg-python/pull/664 Bumps [mkdocstrings](https://github.com/mkdocstrings/mkdocstrings) from 0.24.0 to 0.25.0. Release notes Sourced from https://github.com/mkdocstrings/mkdocstrings/releases";>mkdoc

Re: [PR] Basic Integration with Datafusion [iceberg-rust]

2024-04-27 Thread via GitHub
marvinlanhenke commented on code in PR #324: URL: https://github.com/apache/iceberg-rust/pull/324#discussion_r1581362714 ## crates/integrations/datafusion/src/table.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lic

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-04-27 Thread via GitHub
danielcweeks commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1581866578 ## open-api/rest-catalog-open-api.yaml: ## @@ -2106,6 +2213,32 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile'

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-04-27 Thread via GitHub
danielcweeks commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1581866578 ## open-api/rest-catalog-open-api.yaml: ## @@ -2106,6 +2213,32 @@ components: items: $ref: '#/components/schemas/PartitionStatisticsFile'

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-04-27 Thread via GitHub
danielcweeks commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1581866390 ## open-api/rest-catalog-open-api.yaml: ## @@ -537,6 +537,110 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefi

Re: [PR] Add Pagination To List Apis [iceberg]

2024-04-27 Thread via GitHub
danielcweeks commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1581860457 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -136,6 +137,7 @@ public class RESTSessionCatalog extends BaseViewSessionCatalog pr

Re: [PR] HADOOP-18679. Add API for bulk/paged object deletion: Iceberg PoC [iceberg]

2024-04-27 Thread via GitHub
danielcweeks commented on code in PR #10233: URL: https://github.com/apache/iceberg/pull/10233#discussion_r1581856455 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopFileIO.java: ## @@ -166,23 +178,106 @@ public void deletePrefix(String prefix) { @Override public voi

Re: [PR] HADOOP-18679. Add API for bulk/paged object deletion: Iceberg PoC [iceberg]

2024-04-27 Thread via GitHub
danielcweeks commented on code in PR #10233: URL: https://github.com/apache/iceberg/pull/10233#discussion_r1581856096 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopFileIO.java: ## @@ -166,23 +178,106 @@ public void deletePrefix(String prefix) { @Override public voi

Re: [PR] Add filter pushdown API for contains and endsWith [iceberg]

2024-04-27 Thread via GitHub
yabola commented on PR #9710: URL: https://github.com/apache/iceberg/pull/9710#issuecomment-2081026195 @sujithjay could you also take a look since you implements `startWith`~ Spark on parquet also support `endsWith` and `contains` https://github.com/apache/spark/blob/356830ada6c6ebbf54

Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]

2024-04-27 Thread via GitHub
amogh-jahagirdar commented on code in PR #10140: URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581851587 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java: ## @@ -43,8 +53,16 @@ public JdbcClientPool(String dbUrl, Map props) { } public J

Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]

2024-04-27 Thread via GitHub
amogh-jahagirdar commented on code in PR #10140: URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581851445 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java: ## @@ -167,6 +168,67 @@ public void testInitialize() { jdbcCatalog.initialize("test_

Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]

2024-04-27 Thread via GitHub
amogh-jahagirdar commented on code in PR #10140: URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581851517 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java: ## @@ -43,8 +53,16 @@ public JdbcClientPool(String dbUrl, Map props) { } public J

Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]

2024-04-27 Thread via GitHub
amogh-jahagirdar commented on code in PR #10140: URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581851484 ## core/src/test/java/org/apache/iceberg/TestClientPoolImpl.java: ## @@ -0,0 +1,149 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]

2024-04-27 Thread via GitHub
amogh-jahagirdar commented on code in PR #10140: URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581851465 ## core/src/test/java/org/apache/iceberg/TestClientPoolImpl.java: ## @@ -0,0 +1,149 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]

2024-04-27 Thread via GitHub
amogh-jahagirdar commented on code in PR #10140: URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581695111 ## core/src/main/java/org/apache/iceberg/ClientPoolImpl.java: ## @@ -56,26 +69,38 @@ public R run(Action action, boolean retry) throws E, InterruptedExc

Re: [I] Type error in naming a protected attribute in class PyArrowFile(OutputFile, InputFile), breaking readability from HDFS using PyArrow [iceberg-python]

2024-04-27 Thread via GitHub
kevinjqliu commented on issue #654: URL: https://github.com/apache/iceberg-python/issues/654#issuecomment-2080922162 feel free to tag in the PR for review :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [I] Type error in naming a protected attribute in class PyArrowFile(OutputFile, InputFile), breaking readability from HDFS using PyArrow [iceberg-python]

2024-04-27 Thread via GitHub
SebastianoMeneghin commented on issue #654: URL: https://github.com/apache/iceberg-python/issues/654#issuecomment-2080910925 Sure, I can! If it makes sense for you, I will haha -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [I] Type error in naming a protected attribute in class PyArrowFile(OutputFile, InputFile), breaking readability from HDFS using PyArrow [iceberg-python]

2024-04-27 Thread via GitHub
kevinjqliu commented on issue #654: URL: https://github.com/apache/iceberg-python/issues/654#issuecomment-2080872516 It should be a 1 line change, would you like to contribute? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Kafka Connect: Add kerberos authentication option [iceberg]

2024-04-27 Thread via GitHub
bryanck commented on code in PR #10173: URL: https://github.com/apache/iceberg/pull/10173#discussion_r1581844369 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/Utilities.java: ## @@ -28,6 +28,8 @@ import java.util.Set; import java.util.UUID; impo

Re: [PR] Kafka Connect: Add kerberos authentication option [iceberg]

2024-04-27 Thread via GitHub
bryanck commented on code in PR #10173: URL: https://github.com/apache/iceberg/pull/10173#discussion_r1581844026 ## kafka-connect/build.gradle: ## @@ -43,6 +43,7 @@ project(":iceberg-kafka-connect:iceberg-kafka-connect") { implementation "com.fasterxml.jackson.core:jackson-

Re: [I] Type error in naming a protected attribute in class PyArrowFile(OutputFile, InputFile), breaking readability from HDFS using PyArrow [iceberg-python]

2024-04-27 Thread via GitHub
SebastianoMeneghin commented on issue #654: URL: https://github.com/apache/iceberg-python/issues/654#issuecomment-2080830949 Should I fix the problem or will you? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Clean backport with:Flink: Backport #10207 to v1.18 and v1.17 [iceberg]

2024-04-27 Thread via GitHub
stevenzwu merged PR #10235: URL: https://github.com/apache/iceberg/pull/10235 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [I] Tracking issues of OpenDAL API changes [iceberg-rust]

2024-04-27 Thread via GitHub
Xuanwo commented on issue #356: URL: https://github.com/apache/iceberg-rust/issues/356#issuecomment-2080446957 cc @Fokko @liurenjie1024 @ZENOTME to take a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

[I] Tracking issues of OpenDAL API changes [iceberg-rust]

2024-04-27 Thread via GitHub
Xuanwo opened a new issue, #356: URL: https://github.com/apache/iceberg-rust/issues/356 Hi, iceberger! OpenDAL's coming v0.46 release will have API changes that could affect our project. ## Changes I list the major changes that we need to take care: ### OpenDAL Reader d

Re: [PR] docs: installation of the new `iceberg_catalog_rest` added to the docs [iceberg-rust]

2024-04-27 Thread via GitHub
Xuanwo commented on PR #355: URL: https://github.com/apache/iceberg-rust/pull/355#issuecomment-2080421284 Also cc @Fokko and @liurenjie1024 for a review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to