Re: [PR] Build: Upgrade to gradle 8.6 [iceberg]

2024-03-23 Thread via GitHub
jbonofre commented on PR #8486: URL: https://github.com/apache/iceberg/pull/8486#issuecomment-2016404545 Now upgrading to Gradle 8.7 and proposing alternative approach to revapi plugin. -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [I] flink to iceberg commit fail (Datastream API, hiveCatalog) [iceberg]

2024-03-23 Thread via GitHub
SML0127 commented on issue #10019: URL: https://github.com/apache/iceberg/issues/10019#issuecomment-2016405173 Thank you @pvary ! I found the cause! While setting up HMSClient, I found that the hive-metastore lib version in Flink project is the latest version (v3.x). It works fi

Re: [PR] docs: Add links checker [iceberg]

2024-03-23 Thread via GitHub
Fokko merged PR #9965: URL: https://github.com/apache/iceberg/pull/9965 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] docs: Add links checker [iceberg]

2024-03-23 Thread via GitHub
Fokko commented on PR #9965: URL: https://github.com/apache/iceberg/pull/9965#issuecomment-2016443994 Thanks for the review @nastra and @manuzhang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] A logo for iceberg rust! [iceberg-rust]

2024-03-23 Thread via GitHub
pmcgleenon commented on issue #216: URL: https://github.com/apache/iceberg-rust/issues/216#issuecomment-2016470531 ![image](https://github.com/apache/iceberg-rust/assets/8458078/0e315b79-373f-412c-9117-e4cb6b00f7b1) -- This is an automated message from the Apache Git Service. To respo

Re: [I] Pyarrow type error [iceberg-python]

2024-03-23 Thread via GitHub
Fokko commented on issue #541: URL: https://github.com/apache/iceberg-python/issues/541#issuecomment-2016497750 @dev-goyal Thanks for raising this. It looks like that's a timestamp with nanosecond precision. Support for nanosecond timestamps is currently being added in the latest specificat

Re: [I] [Potential Bug] insertedRowMap too large could cause flink application failure? [iceberg]

2024-03-23 Thread via GitHub
puchengy commented on issue #10030: URL: https://github.com/apache/iceberg/issues/10030#issuecomment-2016507348 But it does not avoid the map getting too large and fail the application right? On Fri, Mar 22, 2024 at 11:49 PM pvary ***@***.***> wrote: > You would like to compa

Re: [PR] Add Pagination To List Apis [iceberg]

2024-03-23 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1536645727 ## core/src/main/java/org/apache/iceberg/rest/responses/ListTablesResponse.java: ## @@ -30,13 +30,15 @@ public class ListTablesResponse implements RESTResponse { Revi

Re: [PR] Add Pagination To List Apis [iceberg]

2024-03-23 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1536646978 ## core/src/main/java/org/apache/iceberg/rest/responses/ListNamespacesResponse.java: ## @@ -59,6 +68,7 @@ public static Builder builder() { public static class Bui

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-23 Thread via GitHub
wanghualei commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2016525287 I meet this problem also. when user "drop table purge", it delete files in data and meta directory,but ,the table driectory still exits, in normal sence , the directory should b

Re: [I] Bloom filter not properly leveraged when using an OR condition [iceberg]

2024-03-23 Thread via GitHub
cccs-jc commented on issue #10029: URL: https://github.com/apache/iceberg/issues/10029#issuecomment-2016525638 @huaxingao You are absolutely correct; the issue arises also when combining the `statsFilter` with the `dictFilter`. It's essentially the same underlying problem. The crux o

Re: [I] Pyarrow type error [iceberg-python]

2024-03-23 Thread via GitHub
dev-goyal commented on issue #541: URL: https://github.com/apache/iceberg-python/issues/541#issuecomment-2016573605 Thanks @Fokko, makes sense! I was able to simply reduce precision on my end so it's not a big deal, but I figured it couldn't hurt to raise this. I wrote these data usin

Re: [I] Bloom filter not properly leveraged when using an OR condition [iceberg]

2024-03-23 Thread via GitHub
huaxingao commented on issue #10029: URL: https://github.com/apache/iceberg/issues/10029#issuecomment-2016577139 @cccs-jc Thanks for your proposal! For filter `col1=1 || col2=1`, the current implementation is: ``` shouldRead = statsFilter(col1=1 || col2=1) && dictFilter(col1=1 ||

[PR] Glue Catalog: Basic Setup + Test Infra (1/3) [iceberg-rust]

2024-03-23 Thread via GitHub
marvinlanhenke opened a new pull request, #294: URL: https://github.com/apache/iceberg-rust/pull/294 ### Which issue does this PR close? Partly #249 (Task 1/3) ### Rationale for this change Add support for Glue catalog, to reach feature parity with other implementations.

Re: [PR] Glue Catalog: Basic Setup + Test Infra (1/3) [iceberg-rust]

2024-03-23 Thread via GitHub
marvinlanhenke commented on PR #294: URL: https://github.com/apache/iceberg-rust/pull/294#issuecomment-2016585197 @liurenjie1024 PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

[I] Race condition on `Table.scan` with `limit` [iceberg-python]

2024-03-23 Thread via GitHub
bigluck opened a new issue, #542: URL: https://github.com/apache/iceberg-python/issues/542 ### Apache Iceberg version 0.6.0 (latest release) ### Please describe the bug 🐞 I'm facing a race condition when doing `table.scan` on my code. For some strange reason, the code ex

Re: [PR] [WIP] Add `ManifestEvaluator` to allow filtering of files in a table scan (Issue #152) [iceberg-rust]

2024-03-23 Thread via GitHub
sdd commented on PR #241: URL: https://github.com/apache/iceberg-rust/pull/241#issuecomment-2016630719 @liurenjie1024 are you able to take a look at this again please when you have time? I've resolved the outstanding issues with binding and added accessors to address the missing functionali

Re: [I] Absolute address can be optimized to relative address in file(*.metadata.json.)? [iceberg]

2024-03-23 Thread via GitHub
github-actions[bot] commented on issue #1938: URL: https://github.com/apache/iceberg/issues/1938#issuecomment-2016639937 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Absolute address can be optimized to relative address in file(*.metadata.json.)? [iceberg]

2024-03-23 Thread via GitHub
github-actions[bot] closed issue #1938: Absolute address can be optimized to relative address in file(*.metadata.json.)? URL: https://github.com/apache/iceberg/issues/1938 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [I] Support for role based access of HadoopCatalog table listing [iceberg]

2024-03-23 Thread via GitHub
github-actions[bot] commented on issue #1941: URL: https://github.com/apache/iceberg/issues/1941#issuecomment-2016639948 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Support for role based access of HadoopCatalog table listing [iceberg]

2024-03-23 Thread via GitHub
github-actions[bot] closed issue #1941: Support for role based access of HadoopCatalog table listing URL: https://github.com/apache/iceberg/issues/1941 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [I] Replace table shouldn't clear table history [iceberg]

2024-03-23 Thread via GitHub
github-actions[bot] commented on issue #2233: URL: https://github.com/apache/iceberg/issues/2233#issuecomment-2016640045 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Flink schema validation always has checkNullability and checkOrdering set true. Any particular reason ? [iceberg]

2024-03-23 Thread via GitHub
github-actions[bot] commented on issue #2235: URL: https://github.com/apache/iceberg/issues/2235#issuecomment-2016640050 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Flink: Don't fail to serialize IcebergSourceSplit when there is too many delete files [iceberg]

2024-03-23 Thread via GitHub
javrasya commented on PR #9464: URL: https://github.com/apache/iceberg/pull/9464#issuecomment-2016643451 Hi @pvary, Good idea. Let me have a look at it and update this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[PR] Build: Bump mkdocs-material from 9.5.14 to 9.5.15 [iceberg]

2024-03-23 Thread via GitHub
dependabot[bot] opened a new pull request, #10031: URL: https://github.com/apache/iceberg/pull/10031 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.14 to 9.5.15. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mkdoc

[PR] feat: Convert predicate to arrow filter and push down to parquet reader [iceberg-rust]

2024-03-23 Thread via GitHub
viirya opened a new pull request, #295: URL: https://github.com/apache/iceberg-rust/pull/295 close #265 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[PR] Build: Bump io.netty:netty-buffer from 4.1.107.Final to 4.1.108.Final [iceberg]

2024-03-23 Thread via GitHub
dependabot[bot] opened a new pull request, #10032: URL: https://github.com/apache/iceberg/pull/10032 Bumps [io.netty:netty-buffer](https://github.com/netty/netty) from 4.1.107.Final to 4.1.108.Final. Commits https://github.com/netty/netty/commit/3a3f9d13b129555802de5652667ca0af

[PR] Build: Bump orc from 1.9.2 to 1.9.3 [iceberg]

2024-03-23 Thread via GitHub
dependabot[bot] opened a new pull request, #10033: URL: https://github.com/apache/iceberg/pull/10033 Bumps `orc` from 1.9.2 to 1.9.3. Updates `org.apache.orc:orc-core` from 1.9.2 to 1.9.3 Updates `org.apache.orc:orc-tools` from 1.9.2 to 1.9.3 Dependabot will resolve any c

[PR] Build: Bump software.amazon.awssdk:bom from 2.24.5 to 2.25.16 [iceberg]

2024-03-23 Thread via GitHub
dependabot[bot] opened a new pull request, #10035: URL: https://github.com/apache/iceberg/pull/10035 Bumps software.amazon.awssdk:bom from 2.24.5 to 2.25.16. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=softw

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.24.5 to 2.25.11 [iceberg]

2024-03-23 Thread via GitHub
dependabot[bot] commented on PR #9975: URL: https://github.com/apache/iceberg/pull/9975#issuecomment-2016691028 Superseded by #10035. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.24.5 to 2.25.11 [iceberg]

2024-03-23 Thread via GitHub
dependabot[bot] closed pull request #9975: Build: Bump software.amazon.awssdk:bom from 2.24.5 to 2.25.11 URL: https://github.com/apache/iceberg/pull/9975 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[PR] Build: Bump arrow from 15.0.1 to 15.0.2 [iceberg]

2024-03-23 Thread via GitHub
dependabot[bot] opened a new pull request, #10034: URL: https://github.com/apache/iceberg/pull/10034 Bumps `arrow` from 15.0.1 to 15.0.2. Updates `org.apache.arrow:arrow-memory-netty` from 15.0.1 to 15.0.2 Updates `org.apache.arrow:arrow-vector` from 15.0.1 to 15.0.2 Commits

Re: [PR] Implement __getstate__ and __setstate__ on PyArrowFileIO and FsSpecFileIO so that they can be pickled [iceberg-python]

2024-03-23 Thread via GitHub
amogh-jahagirdar commented on code in PR #543: URL: https://github.com/apache/iceberg-python/pull/543#discussion_r1536736087 ## tests/io/test_pyarrow.py: ## @@ -256,6 +257,14 @@ def test_raise_on_opening_a_local_file_not_found() -> None: assert "[Errno 2] Failed to open

Re: [PR] Implement __getstate__ and __setstate__ on PyArrowFileIO and FsSpecFileIO so that they can be pickled [iceberg-python]

2024-03-23 Thread via GitHub
amogh-jahagirdar commented on code in PR #543: URL: https://github.com/apache/iceberg-python/pull/543#discussion_r1536736087 ## tests/io/test_pyarrow.py: ## @@ -256,6 +257,14 @@ def test_raise_on_opening_a_local_file_not_found() -> None: assert "[Errno 2] Failed to open

Re: [PR] Build: Bump mkdocs-material from 9.5.14 to 9.5.15 [iceberg]

2024-03-23 Thread via GitHub
Fokko merged PR #10031: URL: https://github.com/apache/iceberg/pull/10031 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [I] flink to iceberg commit fail (Datastream API, hiveCatalog) [iceberg]

2024-03-23 Thread via GitHub
pvary closed issue #10019: flink to iceberg commit fail (Datastream API, hiveCatalog) URL: https://github.com/apache/iceberg/issues/10019 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [I] flink to iceberg commit fail (Datastream API, hiveCatalog) [iceberg]

2024-03-23 Thread via GitHub
pvary commented on issue #10019: URL: https://github.com/apache/iceberg/issues/10019#issuecomment-2016702532 @SML0127: Good to hear that your issue has been resolved! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] [Potential Bug] insertedRowMap too large could cause flink application failure? [iceberg]

2024-03-23 Thread via GitHub
pvary commented on issue #10030: URL: https://github.com/apache/iceberg/issues/10030#issuecomment-2016703799 Oh... sry, this is the writer side. My suggestion helps on reader side only. On writer side you can increase your checkpointing period, migrate to a smaller primary key (introd