Re: [PR] feat: Read Parquet data file with projection [iceberg-rust]

2024-03-30 Thread via GitHub
viirya commented on code in PR #245: URL: https://github.com/apache/iceberg-rust/pull/245#discussion_r1545178174 ## crates/iceberg/src/scan.rs: ## @@ -187,6 +190,22 @@ impl TableScan { let mut arrow_reader_builder = ArrowReaderBuilder::new(self.file_io.clon

Re: [I] SparkSessionCatalog with JDBC catalog: SHOW TABLES IN ... returns error but table exists in JDBC catalog [iceberg]

2024-03-30 Thread via GitHub
matepek commented on issue #10003: URL: https://github.com/apache/iceberg/issues/10003#issuecomment-2027948348 I'm working on something like [this](https://github.com/apache/iceberg/compare/main...matepek:iceberg:main). It would fixes [this](https://github.com/apache/iceberg/issues/1

Re: [PR] feat: Read Parquet data file with projection [iceberg-rust]

2024-03-30 Thread via GitHub
viirya commented on code in PR #245: URL: https://github.com/apache/iceberg-rust/pull/245#discussion_r1545181844 ## crates/iceberg/src/arrow.rs: ## @@ -101,8 +114,53 @@ impl ArrowReader { .boxed()) } -fn get_arrow_projection_mask(&self, _task: &FileScanTask)

Re: [PR] Spark: Don't allow branch_ usage with TIMESTAMP AS OF [iceberg]

2024-03-30 Thread via GitHub
nastra commented on code in PR #10059: URL: https://github.com/apache/iceberg/pull/10059#discussion_r1545207874 ## spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/sql/TestSelect.java: ## @@ -448,6 +448,12 @@ public void testInvalidTimeTravelAgainstBranchIdentifierWithAs

Re: [PR] Spark: Clarify schema behavior when working with branches [iceberg]

2024-03-30 Thread via GitHub
nastra commented on PR #10055: URL: https://github.com/apache/iceberg/pull/10055#issuecomment-2027962915 thanks for the reviews @manuzhang and @rdblue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Spark: Clarify schema behavior when working with branches [iceberg]

2024-03-30 Thread via GitHub
nastra merged PR #10055: URL: https://github.com/apache/iceberg/pull/10055 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [I] branch schema affected by main table schema [iceberg]

2024-03-30 Thread via GitHub
nastra commented on issue #9737: URL: https://github.com/apache/iceberg/issues/9737#issuecomment-2027964878 > Looks like we are disabling the workaround from #10059. The reason for #10059 is because we don't support time travel on branches themselves, because there's no history track

Re: [I] SparkSessionCatalog with JDBC catalog: SHOW TABLES IN ... returns error but table exists in JDBC catalog [iceberg]

2024-03-30 Thread via GitHub
matepek commented on issue #10003: URL: https://github.com/apache/iceberg/issues/10003#issuecomment-2027977162 Not strictly related but I'm kinda stuck with this: Using SparkSessionCatalog with NessieCatalog I cannot create iceberg table: ``` create or replace table my_schema.tab

Re: [PR] feat: Project transform [iceberg-rust]

2024-03-30 Thread via GitHub
marvinlanhenke commented on PR #309: URL: https://github.com/apache/iceberg-rust/pull/309#issuecomment-2028045351 @Fokko @liurenjie1024 I'm having trouble understanding/ verifiying the test-case for [timestamps_day_inclusive_epoch](https://github.com/apache/iceberg/blob/d350c9b8c995a2953

Re: [PR] Flink: backport PR #9321 for range partitioner on map statistics [iceberg]

2024-03-30 Thread via GitHub
stevenzwu merged PR #10061: URL: https://github.com/apache/iceberg/pull/10061 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Flink: backport PR #9321 for range partitioner on map statistics [iceberg]

2024-03-30 Thread via GitHub
stevenzwu commented on PR #10061: URL: https://github.com/apache/iceberg/pull/10061#issuecomment-2028456258 thanks @pvary for the review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [I] Spark writes the Iceberg dual partition table to report an error [iceberg]

2024-03-30 Thread via GitHub
github-actions[bot] commented on issue #1894: URL: https://github.com/apache/iceberg/issues/1894#issuecomment-2028505202 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Rewrite metrics during schema transformation [iceberg]

2024-03-30 Thread via GitHub
github-actions[bot] commented on issue #2079: URL: https://github.com/apache/iceberg/issues/2079#issuecomment-2028505214 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Flink: store watermark as iceberg table's property. [iceberg]

2024-03-30 Thread via GitHub
github-actions[bot] commented on issue #2108: URL: https://github.com/apache/iceberg/issues/2108#issuecomment-2028505229 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Spark can't read Iceberg table created from Presto [iceberg]

2024-03-30 Thread via GitHub
github-actions[bot] commented on issue #2122: URL: https://github.com/apache/iceberg/issues/2122#issuecomment-2028505243 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] ExpireSnapshots deletes active files [iceberg]

2024-03-30 Thread via GitHub
github-actions[bot] commented on issue #2131: URL: https://github.com/apache/iceberg/issues/2131#issuecomment-2028505271 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Rewrite metrics during schema transformation [iceberg]

2024-03-30 Thread via GitHub
github-actions[bot] closed issue #2079: Rewrite metrics during schema transformation URL: https://github.com/apache/iceberg/issues/2079 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [I] StreamingOffset Of Structured streaming read for Iceberg [iceberg]

2024-03-30 Thread via GitHub
github-actions[bot] commented on issue #2085: URL: https://github.com/apache/iceberg/issues/2085#issuecomment-2028505222 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] StreamingOffset Of Structured streaming read for Iceberg [iceberg]

2024-03-30 Thread via GitHub
github-actions[bot] closed issue #2085: StreamingOffset Of Structured streaming read for Iceberg URL: https://github.com/apache/iceberg/issues/2085 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [I] Spark : The iceberg-spark3-extensions module CI test failed [iceberg]

2024-03-30 Thread via GitHub
github-actions[bot] commented on issue #2098: URL: https://github.com/apache/iceberg/issues/2098#issuecomment-2028505227 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Flink: store watermark as iceberg table's property. [iceberg]

2024-03-30 Thread via GitHub
github-actions[bot] closed issue #2108: Flink: store watermark as iceberg table's property. URL: https://github.com/apache/iceberg/issues/2108 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [I] Spark : The iceberg-spark3-extensions module CI test failed [iceberg]

2024-03-30 Thread via GitHub
github-actions[bot] closed issue #2098: Spark : The iceberg-spark3-extensions module CI test failed URL: https://github.com/apache/iceberg/issues/2098 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Hive: Vectorization is not working [iceberg]

2024-03-30 Thread via GitHub
github-actions[bot] commented on issue #2120: URL: https://github.com/apache/iceberg/issues/2120#issuecomment-2028505237 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Hive: Vectorization is not working [iceberg]

2024-03-30 Thread via GitHub
github-actions[bot] closed issue #2120: Hive: Vectorization is not working URL: https://github.com/apache/iceberg/issues/2120 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] Spark can't read Iceberg table created from Presto [iceberg]

2024-03-30 Thread via GitHub
github-actions[bot] closed issue #2122: Spark can't read Iceberg table created from Presto URL: https://github.com/apache/iceberg/issues/2122 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [I] MySQL binlog is synchronized to iceberg through CDC [iceberg]

2024-03-30 Thread via GitHub
github-actions[bot] closed issue #2128: MySQL binlog is synchronized to iceberg through CDC URL: https://github.com/apache/iceberg/issues/2128 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [I] MySQL binlog is synchronized to iceberg through CDC [iceberg]

2024-03-30 Thread via GitHub
github-actions[bot] commented on issue #2128: URL: https://github.com/apache/iceberg/issues/2128#issuecomment-2028505263 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] ExpireSnapshots deletes active files [iceberg]

2024-03-30 Thread via GitHub
github-actions[bot] closed issue #2131: ExpireSnapshots deletes active files URL: https://github.com/apache/iceberg/issues/2131 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Python: Add before- and after operations to `PreOrderSchemaVisitor` [iceberg-python]

2024-03-30 Thread via GitHub
github-actions[bot] commented on issue #32: URL: https://github.com/apache/iceberg-python/issues/32#issuecomment-2028505894 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occu

Re: [I] Add benchmark to the CI [iceberg-python]

2024-03-30 Thread via GitHub
github-actions[bot] commented on issue #27: URL: https://github.com/apache/iceberg-python/issues/27#issuecomment-2028505909 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occu

Re: [I] Add `to_file` with Python API [iceberg-python]

2024-03-30 Thread via GitHub
github-actions[bot] commented on issue #25: URL: https://github.com/apache/iceberg-python/issues/25#issuecomment-2028505921 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occu

Re: [I] Support virtual addressing style in PyArrowFileIO [iceberg-python]

2024-03-30 Thread via GitHub
github-actions[bot] commented on issue #21: URL: https://github.com/apache/iceberg-python/issues/21#issuecomment-2028505927 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occu

Re: [I] ORC file format support [iceberg-python]

2024-03-30 Thread via GitHub
github-actions[bot] commented on issue #20: URL: https://github.com/apache/iceberg-python/issues/20#issuecomment-2028505937 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occu

Re: [I] Add support for Python 3.12 [iceberg-python]

2024-03-30 Thread via GitHub
github-actions[bot] commented on issue #28: URL: https://github.com/apache/iceberg-python/issues/28#issuecomment-2028505900 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occu

Re: [I] Preserve `selected_fields` order in output [iceberg-python]

2024-03-30 Thread via GitHub
github-actions[bot] commented on issue #26: URL: https://github.com/apache/iceberg-python/issues/26#issuecomment-2028505913 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occu

Re: [I] BoundType and BoundPredicate should match type [iceberg-python]

2024-03-30 Thread via GitHub
github-actions[bot] commented on issue #18: URL: https://github.com/apache/iceberg-python/issues/18#issuecomment-2028505942 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occu

Re: [PR] [0.6.x] Backport PR #352 to Improve error message in case of a mismatch [iceberg-python]

2024-03-30 Thread via GitHub
HonahX merged PR #560: URL: https://github.com/apache/iceberg-python/pull/560 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] [0.6.x] Backport PR #523 to cast data to iceberg table's pyarrow schema [iceberg-python]

2024-03-30 Thread via GitHub
HonahX merged PR #559: URL: https://github.com/apache/iceberg-python/pull/559 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

[PR] Minor fixes, #523 followup [iceberg-python]

2024-03-30 Thread via GitHub
kevinjqliu opened a new pull request, #563: URL: https://github.com/apache/iceberg-python/pull/563 Follow up to #523 Remove `print` in test (left it in accidentally, saw it was removed in #559) Call `.as_arrow()` once Uncomment test for `sqlite` -- This is an automated messag

Re: [PR] Minor fixes, #523 followup [iceberg-python]

2024-03-30 Thread via GitHub
HonahX commented on PR #563: URL: https://github.com/apache/iceberg-python/pull/563#issuecomment-2028529240 @kevinjqliu Thanks for fixing this. Do you mind cleaning up the other 2 "prints" in `test_reads.py` and `test_writes.py` too? (#562 ) -- This is an automated message from the Apache

Re: [PR] Minor fixes, #523 followup [iceberg-python]

2024-03-30 Thread via GitHub
kevinjqliu commented on PR #563: URL: https://github.com/apache/iceberg-python/pull/563#issuecomment-2028530382 oops looks like you already had a PR. I just removed all the prints -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] Minor fixes, #523 followup [iceberg-python]

2024-03-30 Thread via GitHub
HonahX merged PR #563: URL: https://github.com/apache/iceberg-python/pull/563 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Clean up "print" in tests [iceberg-python]

2024-03-30 Thread via GitHub
HonahX closed pull request #562: Clean up "print" in tests URL: https://github.com/apache/iceberg-python/pull/562 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [PR] Clean up "print" in tests [iceberg-python]

2024-03-30 Thread via GitHub
HonahX commented on PR #562: URL: https://github.com/apache/iceberg-python/pull/562#issuecomment-2028532708 Close since this is included in #563 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] Minor fixes, #523 followup [iceberg-python]

2024-03-30 Thread via GitHub
HonahX commented on PR #563: URL: https://github.com/apache/iceberg-python/pull/563#issuecomment-2028534976 OOPs, I forgot that the same pattern also exists in [`table.overwrite()`](https://github.com/apache/iceberg-python/pull/563/files#diff-23e8153e0fd497a9212215bd2067068f3b56fa071770c7ef3

[PR] Call as_arrow() call in `overwrite` [iceberg-python]

2024-03-30 Thread via GitHub
kevinjqliu opened a new pull request, #565: URL: https://github.com/apache/iceberg-python/pull/565 Missed this in #563 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [PR] Call as_arrow() call in `overwrite` [iceberg-python]

2024-03-30 Thread via GitHub
HonahX merged PR #565: URL: https://github.com/apache/iceberg-python/pull/565 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] [0.6.x] Backport #563 for minor fixes [iceberg-python]

2024-03-30 Thread via GitHub
HonahX merged PR #564: URL: https://github.com/apache/iceberg-python/pull/564 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] 5 dremio blog march 2024 [iceberg]

2024-03-30 Thread via GitHub
AlexMercedCoder commented on PR #10067: URL: https://github.com/apache/iceberg/pull/10067#issuecomment-2028542742 Re: The failed check has to do a with a url on the releases page which I didn't touch, when I try out the link I'm able to download the jar so not sure why it failed. -- This

Re: [PR] feat: Read Parquet data file with projection [iceberg-rust]

2024-03-30 Thread via GitHub
liurenjie1024 commented on code in PR #245: URL: https://github.com/apache/iceberg-rust/pull/245#discussion_r1545534908 ## crates/iceberg/src/scan.rs: ## @@ -187,6 +190,46 @@ impl TableScan { let mut arrow_reader_builder = ArrowReaderBuilder::new(self.file_

Re: [PR] feat: Read Parquet data file with projection [iceberg-rust]

2024-03-30 Thread via GitHub
liurenjie1024 commented on code in PR #245: URL: https://github.com/apache/iceberg-rust/pull/245#discussion_r1545535512 ## crates/iceberg/src/arrow.rs: ## @@ -101,8 +114,53 @@ impl ArrowReader { .boxed()) } -fn get_arrow_projection_mask(&self, _task: &FileSca

[PR] Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.35.0 [iceberg]

2024-03-30 Thread via GitHub
dependabot[bot] opened a new pull request, #10070: URL: https://github.com/apache/iceberg/pull/10070 Bumps [com.google.cloud:libraries-bom](https://github.com/googleapis/java-cloud-bom) from 26.28.0 to 26.35.0. Release notes Sourced from https://github.com/googleapis/java-cloud-bo

[PR] Build: Bump com.azure:azure-sdk-bom from 1.2.21 to 1.2.22 [iceberg]

2024-03-30 Thread via GitHub
dependabot[bot] opened a new pull request, #10071: URL: https://github.com/apache/iceberg/pull/10071 Bumps [com.azure:azure-sdk-bom](https://github.com/azure/azure-sdk-for-java) from 1.2.21 to 1.2.22. Commits https://github.com/Azure/azure-sdk-for-java/commit/28669160e6952e57f2

[PR] Build: Bump software.amazon.awssdk:bom from 2.25.18 to 2.25.21 [iceberg]

2024-03-30 Thread via GitHub
dependabot[bot] opened a new pull request, #10072: URL: https://github.com/apache/iceberg/pull/10072 Bumps software.amazon.awssdk:bom from 2.25.18 to 2.25.21. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=soft

Re: [PR] feat: Read Parquet data file with projection [iceberg-rust]

2024-03-30 Thread via GitHub
viirya commented on code in PR #245: URL: https://github.com/apache/iceberg-rust/pull/245#discussion_r1545542266 ## crates/iceberg/src/scan.rs: ## @@ -187,6 +190,46 @@ impl TableScan { let mut arrow_reader_builder = ArrowReaderBuilder::new(self.file_io.clon

Re: [PR] feat: Read Parquet data file with projection [iceberg-rust]

2024-03-30 Thread via GitHub
viirya commented on code in PR #245: URL: https://github.com/apache/iceberg-rust/pull/245#discussion_r1545542266 ## crates/iceberg/src/scan.rs: ## @@ -187,6 +190,46 @@ impl TableScan { let mut arrow_reader_builder = ArrowReaderBuilder::new(self.file_io.clon