Re: [PR] Core: Add LockManager to HadoopTableOperations [iceberg]

2024-02-20 Thread via GitHub
BsoBird commented on PR #3663: URL: https://github.com/apache/iceberg/pull/3663#issuecomment-1953660936 @CodingCat Hello, I am working on a fix related to HadoopTableOperations. I would like to ask you a question. ``` if (!lockManager.acquire(dst.toString(), src.toString()))

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-02-20 Thread via GitHub
BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1495384991 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -355,71 +475,90 @@ int findVersion() { * an attempt will be made to delete the sou

Re: [PR] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-20 Thread via GitHub
nastra merged PR #9670: URL: https://github.com/apache/iceberg/pull/9670 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [I] using pyiceberg with kerberized hive metastore [iceberg-python]

2024-02-20 Thread via GitHub
wuxiaocheng0506 commented on issue #135: URL: https://github.com/apache/iceberg-python/issues/135#issuecomment-1953690548 Hello, any update? We also need kerberos support. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-20 Thread via GitHub
tomtongue commented on PR #9670: URL: https://github.com/apache/iceberg/pull/9670#issuecomment-1953697125 Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [PR] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-20 Thread via GitHub
nastra commented on PR #9670: URL: https://github.com/apache/iceberg/pull/9670#issuecomment-1953701608 thanks for getting this done @tomtongue. Looking forward to other PRs from you for the remaining files in `spark-extensions` -- This is an automated message from the Apache Git Service.

Re: [PR] Migrate Write sub-classes in spark-extensions to JUnit5 and AssertJ style [iceberg]

2024-02-20 Thread via GitHub
tomtongue commented on PR #9670: URL: https://github.com/apache/iceberg/pull/9670#issuecomment-1953703329 Of course, I'm working on migrating procedures and other tests. Will create 2 or 3 PRs laterπŸ‘ -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [I] Tracking issues of Iceberg Rust 0.2.0 Release [iceberg-rust]

2024-02-20 Thread via GitHub
liurenjie1024 commented on issue #180: URL: https://github.com/apache/iceberg-rust/issues/180#issuecomment-1953730600 Voting passed: https://lists.apache.org/thread/1rnk9c1ldjhd160bfxvdyk7mlkzrjj03 -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [PR] feat: Add Sync + Send to Catalog trait [iceberg-rust]

2024-02-20 Thread via GitHub
Fokko merged PR #202: URL: https://github.com/apache/iceberg-rust/pull/202 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [I] Add `Sync + Send` to `Catalog` `trait` [iceberg-rust]

2024-02-20 Thread via GitHub
Fokko closed issue #166: Add `Sync + Send` to `Catalog` `trait` URL: https://github.com/apache/iceberg-rust/issues/166 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] feat: Add Sync + Send to Catalog trait [iceberg-rust]

2024-02-20 Thread via GitHub
Fokko commented on PR #202: URL: https://github.com/apache/iceberg-rust/pull/202#issuecomment-1953735878 Thanks for the PR @ZhengLin-Li, and @liurenjie1024 & @Xuanwo for the review πŸ™Œ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] feat(FileScanTask): partial execute impl for parquet [iceberg-rust]

2024-02-20 Thread via GitHub
sdd opened a new pull request, #207: URL: https://github.com/apache/iceberg-rust/pull/207 A refactor of @ZENOTME 's PR (https://github.com/apache/iceberg-rust/pull/199) based on some of the comments on that PR -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] Tracking issues of Iceberg Rust 0.2.0 Release [iceberg-rust]

2024-02-20 Thread via GitHub
liurenjie1024 commented on issue #180: URL: https://github.com/apache/iceberg-rust/issues/180#issuecomment-1953789982 FYI, here is the doc for publishing on crates.io: https://doc.rust-lang.org/cargo/reference/publishing.html#before-publishing-a-new-crate -- This is an automated message f

Re: [I] Tracking issues of Iceberg Rust 0.2.0 Release [iceberg-rust]

2024-02-20 Thread via GitHub
liurenjie1024 commented on issue #180: URL: https://github.com/apache/iceberg-rust/issues/180#issuecomment-1953779553 Updated the issue to add an extra step: upload to crates.io. cc @Xuanwo @Fokko -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [I] [HadoopCatalog]: [HadoopTableOperations]: Commit flow, renameToFinal does not actually check if lock acquired [iceberg]

2024-02-20 Thread via GitHub
N-o-Z commented on issue #9485: URL: https://github.com/apache/iceberg/issues/9485#issuecomment-1953802367 > @N-o-Z Hello, I am working on a fix related to HadoopTableOperations. I would like to ask you a question. > > ``` > if (!lockManager.acquire(dst.toString(), src.toStri

[I] Run cargo audit in ci. [iceberg-rust]

2024-02-20 Thread via GitHub
liurenjie1024 opened a new issue, #209: URL: https://github.com/apache/iceberg-rust/issues/209 cargo audit is a tool to help us find vulnerable dependencies: https://crates.io/crates/cargo-audit Running this in ci helps to keep `iceberg-rust` secure. I think this is a good firs

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-02-20 Thread via GitHub
BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1495178420 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -149,26 +183,71 @@ public void commit(TableMetadata base, TableMetadata metadata) {

Re: [I] [HadoopCatalog]: [HadoopTableOperations]: Commit flow, renameToFinal does not actually check if lock acquired [iceberg]

2024-02-20 Thread via GitHub
BsoBird commented on issue #9485: URL: https://github.com/apache/iceberg/issues/9485#issuecomment-1953821118 @N-o-Z Tks. I checked the code, I don't have any other problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-02-20 Thread via GitHub
BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1493324502 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -355,71 +475,90 @@ int findVersion() { * an attempt will be made to delete the sou

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-02-20 Thread via GitHub
BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1495384991 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -355,71 +475,90 @@ int findVersion() { * an attempt will be made to delete the sou

Re: [I] Add the git hashes to the RC email [iceberg-rust]

2024-02-20 Thread via GitHub
Fokko closed issue #208: Add the git hashes to the RC email URL: https://github.com/apache/iceberg-rust/issues/208 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-02-20 Thread via GitHub
BsoBird commented on code in PR #9546: URL: https://github.com/apache/iceberg/pull/9546#discussion_r1495518766 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java: ## @@ -355,71 +475,90 @@ int findVersion() { * an attempt will be made to delete the sou

Re: [PR] Infra: Fix issue template labels [iceberg]

2024-02-20 Thread via GitHub
nastra merged PR #9759: URL: https://github.com/apache/iceberg/pull/9759 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[PR] docs: Upload crates [iceberg-rust]

2024-02-20 Thread via GitHub
Fokko opened a new pull request, #211: URL: https://github.com/apache/iceberg-rust/pull/211 I noticed this section was missing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[PR] chore: Bump to version 0.3.0 [iceberg-rust]

2024-02-20 Thread via GitHub
Fokko opened a new pull request, #212: URL: https://github.com/apache/iceberg-rust/pull/212 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

Re: [I] Crates don't have README [iceberg-rust]

2024-02-20 Thread via GitHub
Fokko commented on issue #210: URL: https://github.com/apache/iceberg-rust/issues/210#issuecomment-1953901549 @Xuanwo Thanks for raising this, I also noticed this πŸ‘ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] Add more scripts to make release process smoother. [iceberg-rust]

2024-02-20 Thread via GitHub
Fokko commented on issue #206: URL: https://github.com/apache/iceberg-rust/issues/206#issuecomment-1953903097 One suggestion, In PyIceberg/Java we use a script to generate the RC mail as well. This makes it easy to include git hashes -- This is an automated message from the Apache Git Ser

Re: [PR] chore: Bump to version 0.3.0 [iceberg-rust]

2024-02-20 Thread via GitHub
Xuanwo commented on PR #212: URL: https://github.com/apache/iceberg-rust/pull/212#issuecomment-1953904667 Should we bump the version just after releasing or before? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] docs: Upload crates [iceberg-rust]

2024-02-20 Thread via GitHub
Fokko commented on PR #211: URL: https://github.com/apache/iceberg-rust/pull/211#issuecomment-1953906491 @Xuanwo Great suggestion, I think that would be even easier and less error-prone πŸ‘ -- This is an automated message from the Apache Git Service. To respond to the message, please log o

[I] Add CI for dry-run and publish crates [iceberg-rust]

2024-02-20 Thread via GitHub
Xuanwo opened a new issue, #214: URL: https://github.com/apache/iceberg-rust/issues/214 Thanks! By the way, we can use CI to test and publish the crates while pushing tag like https://github.com/apache/opendal/blob/main/.github/workflows/release_rust.yml. I'm willing to

Re: [PR] Spark 3.5: Add max allowed failed commits to RewriteDataFiles when partial progress is enabled [iceberg]

2024-02-20 Thread via GitHub
manuzhang commented on code in PR #9611: URL: https://github.com/apache/iceberg/pull/9611#discussion_r1495599957 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: ## @@ -358,16 +360,30 @@ private Result doExecuteWithPartialProgr

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-20 Thread via GitHub
nastra commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1495651641 ## open-api/rest-catalog-open-api.yaml: ## @@ -1482,6 +1490,24 @@ components: explode: false example: "vended-credentials,remote-signing" +page-token

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-20 Thread via GitHub
nastra commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1495653871 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1607,17 @@ components: type: string example: [ "accounting", "tax" ] +PageToken: + desc

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-20 Thread via GitHub
nastra commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1495654734 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1607,17 @@ components: type: string example: [ "accounting", "tax" ] +PageToken: + desc

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-20 Thread via GitHub
nastra commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1495655651 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1607,17 @@ components: type: string example: [ "accounting", "tax" ] +PageToken: + desc

Re: [PR] Add pagination to open api spec for listing of namespaces, tables, views [iceberg]

2024-02-20 Thread via GitHub
nastra commented on code in PR #9660: URL: https://github.com/apache/iceberg/pull/9660#discussion_r1495657179 ## open-api/rest-catalog-open-api.yaml: ## @@ -1581,6 +1607,17 @@ components: type: string example: [ "accounting", "tax" ] +PageToken: + desc

[I] A logo for iceberg rust! [iceberg-rust]

2024-02-20 Thread via GitHub
Xuanwo opened a new issue, #216: URL: https://github.com/apache/iceberg-rust/issues/216 It must be cool for iceberg rust to have it's own logo! Maybe we can have a logo that a crab eating iceberg or the iceberg is rusty. ## Notes Unlike [python's logo](https://www.python.org/co

Re: [PR] chore: Bump to version 0.3.0 [iceberg-rust]

2024-02-20 Thread via GitHub
Xuanwo commented on PR #212: URL: https://github.com/apache/iceberg-rust/pull/212#issuecomment-1954046224 > LGTM, I think it's fine to upgrade it now. Ok, so we will always bump to next version during release? -- This is an automated message from the Apache Git Service. To respond t

Re: [PR] chore: Bump to version 0.3.0 [iceberg-rust]

2024-02-20 Thread via GitHub
Xuanwo commented on PR #212: URL: https://github.com/apache/iceberg-rust/pull/212#issuecomment-1954049870 The difference lies in how we determine the version to release. Previously, we would decide on the version just before releasing it. This involved starting a discussion on the mai

Re: [PR] chore: Bump to version 0.3.0 [iceberg-rust]

2024-02-20 Thread via GitHub
liurenjie1024 commented on PR #212: URL: https://github.com/apache/iceberg-rust/pull/212#issuecomment-1954146668 > The difference lies in how we determine the version to release. > > Previously, we would decide on the version just before releasing it. This involved starting a discussi

Re: [PR] docs: Add basic README for all crates [iceberg-rust]

2024-02-20 Thread via GitHub
liurenjie1024 commented on code in PR #215: URL: https://github.com/apache/iceberg-rust/pull/215#discussion_r1495770610 ## crates/iceberg/README.md: ## @@ -0,0 +1,31 @@ + + +# Apache Iceberg Official Native Rust Implementation + +[![crates.io](https://img.shields.io/crates/v/ice

Re: [PR] Core: Support IncrementalChangelogScan with deletes. [iceberg]

2024-02-20 Thread via GitHub
Reo-LEI commented on PR #6182: URL: https://github.com/apache/iceberg/pull/6182#issuecomment-1954152890 > @Reo-LEI and @aokolnychyi, are you still working on this? Sorry for the late reply. I haven't been doing this recently. You can continue to push forward. @manuzhang -- This is

Re: [PR] Spark 3.5: Add Spark application id to summary of RewriteDataFilesSparkAction [iceberg]

2024-02-20 Thread via GitHub
nastra commented on code in PR #9273: URL: https://github.com/apache/iceberg/pull/9273#discussion_r1495773405 ## core/src/main/java/org/apache/iceberg/SnapshotProducer.java: ## @@ -351,6 +351,7 @@ private Map summary(TableMetadata previous) { SnapshotSummary.ADDED_EQ_D

Re: [PR] Flink: Incrementally rewrite data files in streaming. [iceberg]

2024-02-20 Thread via GitHub
Reo-LEI commented on PR #3323: URL: https://github.com/apache/iceberg/pull/3323#issuecomment-1954157081 > Hello, may I ask if the rewrite is asynchronous or synchronous The rewrite is performed asynchronously. @lurnagao -- This is an automated message from the Apache Git Service. T

Re: [PR] feat: Make thrift transport configurable [iceberg-rust]

2024-02-20 Thread via GitHub
Fokko commented on PR #194: URL: https://github.com/apache/iceberg-rust/pull/194#issuecomment-1954190864 Thanks @DeaconDesperado for working on this, and @liurenjie1024 @Xuanwo for the review πŸ‘ -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] feat: Make thrift transport configurable [iceberg-rust]

2024-02-20 Thread via GitHub
Fokko merged PR #194: URL: https://github.com/apache/iceberg-rust/pull/194 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Add UnboundSortOrder [iceberg-rust]

2024-02-20 Thread via GitHub
Fokko merged PR #115: URL: https://github.com/apache/iceberg-rust/pull/115 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [I] feat: Add support for `UnboundSortOrder` [iceberg-rust]

2024-02-20 Thread via GitHub
Fokko closed issue #99: feat: Add support for `UnboundSortOrder` URL: https://github.com/apache/iceberg-rust/issues/99 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] chore: Bump to version 0.3.0 [iceberg-rust]

2024-02-20 Thread via GitHub
Xuanwo commented on PR #212: URL: https://github.com/apache/iceberg-rust/pull/212#issuecomment-1954283514 > Upgrading minor version means add new features without breaking change, e.g. 0.2.0 -> 0.3.0. This is a breaking change in semver definition. -- This is an automated message f

Re: [PR] chore: Bump to version 0.3.0 [iceberg-rust]

2024-02-20 Thread via GitHub
Fokko commented on PR #212: URL: https://github.com/apache/iceberg-rust/pull/212#issuecomment-1954392303 What we do at Python/Java, is create a branch for each major/minor version. If there are any bugs then we can backport them to the branch, and do a quick release. This way you don't have

Re: [PR] docs: Add basic README for all crates [iceberg-rust]

2024-02-20 Thread via GitHub
Fokko commented on code in PR #215: URL: https://github.com/apache/iceberg-rust/pull/215#discussion_r1495654538 ## README.md: ## @@ -42,7 +42,7 @@ Native Rust implementation of [Apache Iceberg](https://iceberg.apache.org/). | GCS | Not Started | | HDFS| Not St

Re: [PR] Parquet: Refactor BasePageIterator to add initRepetitionLevelsReader [iceberg]

2024-02-20 Thread via GitHub
wgtmac commented on PR #9751: URL: https://github.com/apache/iceberg/pull/9751#issuecomment-1954470361 @nastra @Fokko Could you help review this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] chore: Bump to version 0.3.0 [iceberg-rust]

2024-02-20 Thread via GitHub
Xuanwo commented on PR #212: URL: https://github.com/apache/iceberg-rust/pull/212#issuecomment-1954551404 Most rust packages just keep moving on next version and don't do backport and cherrypick. And considering our current state, it's less likely for us to maintaining a v0.2 branch 🫣 --

[I] `JdbcCatalog` `add-view-support` should be evaluated individually from `initialize` flag [iceberg]

2024-02-20 Thread via GitHub
jbonofre opened a new issue, #9764: URL: https://github.com/apache/iceberg/issues/9764 ### Apache Iceberg version None ### Query engine None ### Please describe the bug 🐞 We added `jdbc.add-view-support` flag to the JDBC catalog to support view. If the user

[PR] Core/JDBC Catalog: check SQL schema version and eventually update the schema independently from the initialize flag [iceberg]

2024-02-20 Thread via GitHub
jbonofre opened a new pull request, #9765: URL: https://github.com/apache/iceberg/pull/9765 This closes #9764 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [PR] Core/JDBC Catalog: check SQL schema version and eventually update the schema independently from the initialize flag [iceberg]

2024-02-20 Thread via GitHub
jbonofre commented on PR #9765: URL: https://github.com/apache/iceberg/pull/9765#issuecomment-1954605541 @nastra @ajantha-bhat as discussed together: this PR checks the SQL schema version (and eventually update) independently from the `initialize` flag. -- This is an automated message fro

Re: [PR] Core/JDBC Catalog: check SQL schema version and eventually update the schema independently from the initialize flag [iceberg]

2024-02-20 Thread via GitHub
nastra commented on code in PR #9765: URL: https://github.com/apache/iceberg/pull/9765#discussion_r1496183621 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -199,21 +198,29 @@ private void initializeCatalogTables() throws InterruptedException, SQLExceptio

Re: [PR] Core/JDBC Catalog: check SQL schema version and eventually update the schema independently from the initialize flag [iceberg]

2024-02-20 Thread via GitHub
jbonofre commented on code in PR #9765: URL: https://github.com/apache/iceberg/pull/9765#discussion_r1496187604 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -199,21 +198,29 @@ private void initializeCatalogTables() throws InterruptedException, SQLExcept

Re: [PR] Core/JDBC Catalog: check SQL schema version and eventually update the schema independently from the initialize flag [iceberg]

2024-02-20 Thread via GitHub
nastra commented on code in PR #9765: URL: https://github.com/apache/iceberg/pull/9765#discussion_r1496189025 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -199,21 +198,29 @@ private void initializeCatalogTables() throws InterruptedException, SQLExceptio

Re: [PR] Core/JDBC Catalog: check SQL schema version and eventually update the schema independently from the initialize flag [iceberg]

2024-02-20 Thread via GitHub
jbonofre commented on code in PR #9765: URL: https://github.com/apache/iceberg/pull/9765#discussion_r1496191493 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -199,21 +198,29 @@ private void initializeCatalogTables() throws InterruptedException, SQLExcept

Re: [PR] Core/JDBC Catalog: check SQL schema version and eventually update the schema independently from the initialize flag [iceberg]

2024-02-20 Thread via GitHub
amogh-jahagirdar commented on code in PR #9765: URL: https://github.com/apache/iceberg/pull/9765#discussion_r1496192750 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -199,21 +198,29 @@ private void initializeCatalogTables() throws InterruptedException, S

Re: [PR] Core/JDBC Catalog: check SQL schema version and eventually update the schema independently from the initialize flag [iceberg]

2024-02-20 Thread via GitHub
rdblue commented on code in PR #9765: URL: https://github.com/apache/iceberg/pull/9765#discussion_r1496209568 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -199,21 +198,29 @@ private void initializeCatalogTables() throws InterruptedException, SQLExceptio

Re: [PR] Core/JDBC Catalog: check SQL schema version and eventually update the schema independently from the initialize flag [iceberg]

2024-02-20 Thread via GitHub
rdblue commented on code in PR #9765: URL: https://github.com/apache/iceberg/pull/9765#discussion_r1496214318 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -199,21 +198,29 @@ private void initializeCatalogTables() throws InterruptedException, SQLExceptio

Re: [PR] Core/JDBC Catalog: check SQL schema version and eventually update the schema independently from the initialize flag [iceberg]

2024-02-20 Thread via GitHub
jbonofre commented on code in PR #9765: URL: https://github.com/apache/iceberg/pull/9765#discussion_r1496217801 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -199,21 +198,29 @@ private void initializeCatalogTables() throws InterruptedException, SQLExcept

Re: [PR] Core/JDBC Catalog: check SQL schema version and eventually update the schema independently from the initialize flag [iceberg]

2024-02-20 Thread via GitHub
jbonofre commented on code in PR #9765: URL: https://github.com/apache/iceberg/pull/9765#discussion_r1496218954 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -199,21 +198,29 @@ private void initializeCatalogTables() throws InterruptedException, SQLExcept

Re: [PR] Core/JDBC Catalog: check SQL schema version and eventually update the schema independently from the initialize flag [iceberg]

2024-02-20 Thread via GitHub
rdblue commented on code in PR #9765: URL: https://github.com/apache/iceberg/pull/9765#discussion_r1496220389 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -140,6 +139,7 @@ public void initialize(String name, Map properties) { if (initializeCatalo

Re: [PR] Core: Fix JDBC schema version when initialization is disabled [iceberg]

2024-02-20 Thread via GitHub
rdblue commented on code in PR #9765: URL: https://github.com/apache/iceberg/pull/9765#discussion_r1496224382 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -140,6 +139,7 @@ public void initialize(String name, Map properties) { if (initializeCatalo

Re: [PR] Core: Fix JDBC schema version when initialization is disabled [iceberg]

2024-02-20 Thread via GitHub
jbonofre commented on code in PR #9765: URL: https://github.com/apache/iceberg/pull/9765#discussion_r1496228270 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -140,6 +139,7 @@ public void initialize(String name, Map properties) { if (initializeCata

Re: [PR] Core: Fix JDBC schema version when initialization is disabled [iceberg]

2024-02-20 Thread via GitHub
nastra commented on code in PR #9765: URL: https://github.com/apache/iceberg/pull/9765#discussion_r1496228499 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -199,21 +198,29 @@ private void initializeCatalogTables() throws InterruptedException, SQLExceptio

Re: [PR] Core: Fix JDBC schema version when initialization is disabled [iceberg]

2024-02-20 Thread via GitHub
jbonofre commented on code in PR #9765: URL: https://github.com/apache/iceberg/pull/9765#discussion_r1496232351 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -199,21 +198,29 @@ private void initializeCatalogTables() throws InterruptedException, SQLExcept

Re: [PR] Core: Fix JDBC schema version when initialization is disabled [iceberg]

2024-02-20 Thread via GitHub
jbonofre commented on code in PR #9765: URL: https://github.com/apache/iceberg/pull/9765#discussion_r1496233328 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -140,6 +139,7 @@ public void initialize(String name, Map properties) { if (initializeCata

Re: [PR] Plaintext length in GCM Input Streams [iceberg]

2024-02-20 Thread via GitHub
rdblue commented on code in PR #9752: URL: https://github.com/apache/iceberg/pull/9752#discussion_r1496238941 ## api/src/main/java/org/apache/iceberg/Files.java: ## @@ -95,26 +95,40 @@ public String toString() { } public static InputFile localInput(File file) { -retu

Re: [PR] Plaintext length in GCM Input Streams [iceberg]

2024-02-20 Thread via GitHub
rdblue commented on code in PR #9752: URL: https://github.com/apache/iceberg/pull/9752#discussion_r1496240901 ## core/src/main/java/org/apache/iceberg/encryption/AesGcmInputFile.java: ## @@ -20,39 +20,33 @@ import org.apache.iceberg.io.InputFile; import org.apache.iceberg.io

Re: [PR] Plaintext length in GCM Input Streams [iceberg]

2024-02-20 Thread via GitHub
rdblue commented on code in PR #9752: URL: https://github.com/apache/iceberg/pull/9752#discussion_r1496240901 ## core/src/main/java/org/apache/iceberg/encryption/AesGcmInputFile.java: ## @@ -20,39 +20,33 @@ import org.apache.iceberg.io.InputFile; import org.apache.iceberg.io

Re: [PR] Core: Fix JDBC schema version when initialization is disabled [iceberg]

2024-02-20 Thread via GitHub
rdblue commented on code in PR #9765: URL: https://github.com/apache/iceberg/pull/9765#discussion_r1496261892 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -199,21 +198,29 @@ private void initializeCatalogTables() throws InterruptedException, SQLExceptio

Re: [PR] Core: Fix JDBC schema version when initialization is disabled [iceberg]

2024-02-20 Thread via GitHub
rdblue commented on code in PR #9765: URL: https://github.com/apache/iceberg/pull/9765#discussion_r1496263108 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -140,6 +139,7 @@ public void initialize(String name, Map properties) { if (initializeCatalo

Re: [PR] Core: Fix JDBC schema version when initialization is disabled [iceberg]

2024-02-20 Thread via GitHub
jbonofre commented on code in PR #9765: URL: https://github.com/apache/iceberg/pull/9765#discussion_r1496266194 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -140,6 +139,7 @@ public void initialize(String name, Map properties) { if (initializeCata

Re: [PR] Core: Fix JDBC schema version when initialization is disabled [iceberg]

2024-02-20 Thread via GitHub
jbonofre commented on code in PR #9765: URL: https://github.com/apache/iceberg/pull/9765#discussion_r1496267090 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -199,21 +198,29 @@ private void initializeCatalogTables() throws InterruptedException, SQLExcept

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

2024-02-20 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1496277335 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,184 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [I] com.esotericsoftware.kryo.KryoException: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.nio.ByteBuffer [iceberg]

2024-02-20 Thread via GitHub
adevore3 commented on issue #9738: URL: https://github.com/apache/iceberg/issues/9738#issuecomment-1954798452 Hi, I was trying to create a code sample but got distracted. I've tried to copy all the relevant code pieces here, 1 thing I couldn't do is the protobuf file. It seems to be failing

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

2024-02-20 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1496288449 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3324,277 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

2024-02-20 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1496292798 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3324,277 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

2024-02-20 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1496299094 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3324,277 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-20 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1496301006 ## open-api/rest-catalog-open-api.yaml: ## @@ -2800,6 +3053,59 @@ components: additionalProperties: type: string +PlanContext: + type

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-20 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1496301006 ## open-api/rest-catalog-open-api.yaml: ## @@ -2800,6 +3053,59 @@ components: additionalProperties: type: string +PlanContext: + type

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

2024-02-20 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1496277335 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,184 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] Core, Spark: Remove dangling deletes as part of RewriteDataFilesAction [iceberg]

2024-02-20 Thread via GitHub
szehon-ho commented on code in PR #9724: URL: https://github.com/apache/iceberg/pull/9724#discussion_r1496281509 ## api/src/main/java/org/apache/iceberg/RemoveDanglingDeletesMode.java: ## @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

2024-02-20 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1496312255 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3348,217 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

2024-02-20 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1496312814 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3324,277 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

2024-02-20 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1496313430 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3324,277 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] Core, Spark: Remove dangling deletes as part of RewriteDataFilesAction [iceberg]

2024-02-20 Thread via GitHub
dramaticlly commented on code in PR #9724: URL: https://github.com/apache/iceberg/pull/9724#discussion_r1496319649 ## api/src/main/java/org/apache/iceberg/RemoveDanglingDeletesMode.java: ## @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

[I] Issue with 'writeTo' [iceberg]

2024-02-20 Thread via GitHub
jessiedanwang opened a new issue, #9766: URL: https://github.com/apache/iceberg/issues/9766 ### Apache Iceberg version 1.4.2 ### Query engine Spark ### Please describe the bug 🐞 We use spark structured streaming to consume avro message from kafka and write

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-20 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1496347963 ## open-api/rest-catalog-open-api.yaml: ## @@ -532,6 +532,100 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/na

Re: [I] Issue with 'writeTo' [iceberg]

2024-02-20 Thread via GitHub
jessiedanwang commented on issue #9766: URL: https://github.com/apache/iceberg/issues/9766#issuecomment-1954881197 forgot to mention that if we switch from_protobuf to from_arvro to deserialized avro instead of protobuf messages, then we have no problem with 'writeTo' -- This is an autom

Re: [PR] Make the snapshot creation part of the `Transaction` [iceberg-python]

2024-02-20 Thread via GitHub
Fokko merged PR #446: URL: https://github.com/apache/iceberg-python/pull/446 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-20 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1496375283 ## open-api/rest-catalog-open-api.yaml: ## @@ -532,6 +532,100 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/na

Re: [PR] Add PrePlanTable and PlanTable Endpoints to open api spec [iceberg]

2024-02-20 Thread via GitHub
rahil-c commented on code in PR #9695: URL: https://github.com/apache/iceberg/pull/9695#discussion_r1496375283 ## open-api/rest-catalog-open-api.yaml: ## @@ -532,6 +532,100 @@ paths: 5XX: $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/na

Re: [I] Issue with 'writeTo' [iceberg]

2024-02-20 Thread via GitHub
jessiedanwang commented on issue #9766: URL: https://github.com/apache/iceberg/issues/9766#issuecomment-1954917660 Also got the following except when using MERGE INTO logic, Exception in thread "stream execution thread for iceberg/cdc/xxx [id = b9e11736-cc3c-4260-aa2b-17dd6735f8bf, ru

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

2024-02-20 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1496399005 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3324,279 @@ components: type: integer format: int64 +BooleanTypeValue: + type

Re: [PR] OpenAPI: Add ContentFile types to spec for scan and append api [iceberg]

2024-02-20 Thread via GitHub
jackye1995 commented on code in PR #9717: URL: https://github.com/apache/iceberg/pull/9717#discussion_r1496402434 ## open-api/rest-catalog-open-api.yaml: ## @@ -3324,6 +3324,279 @@ components: type: integer format: int64 +BooleanTypeValue: + type

  1   2   3   >