Re: [I] Add OAuth Support for iceberg rest client. [iceberg-rust]

2024-03-09 Thread via GitHub
liurenjie1024 commented on issue #238: URL: https://github.com/apache/iceberg-rust/issues/238#issuecomment-1987115350 > I'm ready to give it a try using some OpenAPI rust generators to create a client. If anyone has tried this before and encountered any issues, feel free to share them with

Re: [I] Add OAuth Support for iceberg rest client. [iceberg-rust]

2024-03-09 Thread via GitHub
liurenjie1024 commented on issue #238: URL: https://github.com/apache/iceberg-rust/issues/238#issuecomment-1987115027 > The python implementation is generated by their spec, do we really need to implement the rest client manually? Hi, are you looking at iceberg-python or generated Pyt

Re: [PR] [Bug fix] update name mapping in Transaction.update_schema [iceberg-python]

2024-03-09 Thread via GitHub
HonahX merged PR #508: URL: https://github.com/apache/iceberg-python/pull/508 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Core: add support to add custom schemes via properties in ResolvingFileIO [iceberg]

2024-03-09 Thread via GitHub
jbonofre commented on PR #9884: URL: https://github.com/apache/iceberg/pull/9884#issuecomment-1987107350 @danielcweeks And how can we tackle if an user wants to override an existing Iceberg `ResolverFileIO` ? Imagine if the user wants to change the authentication for `s3` or add `my-s3` sch

Re: [I] Add OAuth Support for iceberg rest client. [iceberg-rust]

2024-03-09 Thread via GitHub
TennyZhuang commented on issue #238: URL: https://github.com/apache/iceberg-rust/issues/238#issuecomment-1987106241 I'm ready to give it a try using some OpenAPI rust generators to create a client. If anyone has tried this before and encountered any issues, feel free to share them with me.

Re: [I] Add OAuth Support for iceberg rest client. [iceberg-rust]

2024-03-09 Thread via GitHub
TennyZhuang commented on issue #238: URL: https://github.com/apache/iceberg-rust/issues/238#issuecomment-1987098584 The python implementation is generated by their spec, do we really need to implement the rest client manually? -- This is an automated message from the Apache Git Service. T

Re: [I] Discussion: Design of `TableMetadataBuilder`. [iceberg-rust]

2024-03-09 Thread via GitHub
liurenjie1024 commented on issue #232: URL: https://github.com/apache/iceberg-rust/issues/232#issuecomment-1987094899 Cool, seems we reached consensus on this, let's move on to create `TableMetadataBuilder`. -- This is an automated message from the Apache Git Service. To respond to the me

Re: [I] Add hive metastore catalog support [iceberg-rust]

2024-03-09 Thread via GitHub
liurenjie1024 commented on issue #113: URL: https://github.com/apache/iceberg-rust/issues/113#issuecomment-1987093854 Hi, @marvinlanhenke Thanks for the suggestion, I think creating tracking issue and clarifying their dependency would be quite helpful, let me create one for 0.3 release. I'm

[PR] Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.34.0 [iceberg]

2024-03-09 Thread via GitHub
dependabot[bot] opened a new pull request, #9916: URL: https://github.com/apache/iceberg/pull/9916 Bumps [com.google.cloud:libraries-bom](https://github.com/googleapis/java-cloud-bom) from 26.28.0 to 26.34.0. Release notes Sourced from https://github.com/googleapis/java-cloud-bom/

[PR] Build: Bump com.fasterxml.jackson.core:jackson-annotations from 2.16.0 to 2.16.2 [iceberg]

2024-03-09 Thread via GitHub
dependabot[bot] opened a new pull request, #9915: URL: https://github.com/apache/iceberg/pull/9915 Bumps [com.fasterxml.jackson.core:jackson-annotations](https://github.com/FasterXML/jackson) from 2.16.0 to 2.16.2. Commits See full diff in https://github.com/FasterXML/jackson/

[PR] Build: Bump com.fasterxml.jackson.dataformat:jackson-dataformat-xml from 2.16.1 to 2.16.2 [iceberg]

2024-03-09 Thread via GitHub
dependabot[bot] opened a new pull request, #9914: URL: https://github.com/apache/iceberg/pull/9914 Bumps [com.fasterxml.jackson.dataformat:jackson-dataformat-xml](https://github.com/FasterXML/jackson-dataformat-xml) from 2.16.1 to 2.16.2. Commits https://github.com/FasterXML/j

[PR] Build: Bump orc from 1.9.2 to 2.0.0 [iceberg]

2024-03-09 Thread via GitHub
dependabot[bot] opened a new pull request, #9913: URL: https://github.com/apache/iceberg/pull/9913 Bumps `orc` from 1.9.2 to 2.0.0. Updates `org.apache.orc:orc-core` from 1.9.2 to 2.0.0 Updates `org.apache.orc:orc-tools` from 1.9.2 to 2.0.0 Dependabot will resolve any con

Re: [PR] Build: Bump org.testcontainers:testcontainers from 1.19.5 to 1.19.6 [iceberg]

2024-03-09 Thread via GitHub
dependabot[bot] closed pull request #9800: Build: Bump org.testcontainers:testcontainers from 1.19.5 to 1.19.6 URL: https://github.com/apache/iceberg/pull/9800 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Build: Bump org.testcontainers:testcontainers from 1.19.5 to 1.19.6 [iceberg]

2024-03-09 Thread via GitHub
dependabot[bot] commented on PR #9800: URL: https://github.com/apache/iceberg/pull/9800#issuecomment-1987090964 Superseded by #9912. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[PR] Build: Bump org.testcontainers:testcontainers from 1.19.5 to 1.19.7 [iceberg]

2024-03-09 Thread via GitHub
dependabot[bot] opened a new pull request, #9912: URL: https://github.com/apache/iceberg/pull/9912 Bumps [org.testcontainers:testcontainers](https://github.com/testcontainers/testcontainers-java) from 1.19.5 to 1.19.7. Release notes Sourced from https://github.com/testcontainers/t

[PR] Build: Bump org.roaringbitmap:RoaringBitmap from 1.0.1 to 1.0.5 [iceberg]

2024-03-09 Thread via GitHub
dependabot[bot] opened a new pull request, #9911: URL: https://github.com/apache/iceberg/pull/9911 Bumps [org.roaringbitmap:RoaringBitmap](https://github.com/RoaringBitmap/RoaringBitmap) from 1.0.1 to 1.0.5. Release notes Sourced from https://github.com/RoaringBitmap/RoaringBitmap

[PR] Build: Bump arrow from 15.0.0 to 15.0.1 [iceberg]

2024-03-09 Thread via GitHub
dependabot[bot] opened a new pull request, #9910: URL: https://github.com/apache/iceberg/pull/9910 Bumps `arrow` from 15.0.0 to 15.0.1. Updates `org.apache.arrow:arrow-memory-netty` from 15.0.0 to 15.0.1 Updates `org.apache.arrow:arrow-vector` from 15.0.0 to 15.0.1 Commits

[PR] Build: Bump nessie from 0.77.1 to 0.78.0 [iceberg]

2024-03-09 Thread via GitHub
dependabot[bot] opened a new pull request, #9909: URL: https://github.com/apache/iceberg/pull/9909 Bumps `nessie` from 0.77.1 to 0.78.0. Updates `org.projectnessie.nessie:nessie-client` from 0.77.1 to 0.78.0 Updates `org.projectnessie.nessie:nessie-jaxrs-testextension` from 0.77.1 t

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.24.5 to 2.25.1 [iceberg]

2024-03-09 Thread via GitHub
dependabot[bot] commented on PR #9856: URL: https://github.com/apache/iceberg/pull/9856#issuecomment-1987090755 Superseded by #9907. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.24.5 to 2.25.1 [iceberg]

2024-03-09 Thread via GitHub
dependabot[bot] closed pull request #9856: Build: Bump software.amazon.awssdk:bom from 2.24.5 to 2.25.1 URL: https://github.com/apache/iceberg/pull/9856 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[PR] Build: Bump org.glassfish.jaxb:jaxb-runtime from 2.3.3 to 4.0.5 [iceberg]

2024-03-09 Thread via GitHub
dependabot[bot] opened a new pull request, #9908: URL: https://github.com/apache/iceberg/pull/9908 Bumps org.glassfish.jaxb:jaxb-runtime from 2.3.3 to 4.0.5. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=org.g

[PR] Build: Bump software.amazon.awssdk:bom from 2.24.5 to 2.25.6 [iceberg]

2024-03-09 Thread via GitHub
dependabot[bot] opened a new pull request, #9907: URL: https://github.com/apache/iceberg/pull/9907 Bumps software.amazon.awssdk:bom from 2.24.5 to 2.25.6. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=software

Re: [I] Add hive metastore catalog support [iceberg-rust]

2024-03-09 Thread via GitHub
marvinlanhenke commented on issue #113: URL: https://github.com/apache/iceberg-rust/issues/113#issuecomment-1987088164 > [...] since it has some dependencies, such as transaction api and table metadata builder, otherwise it's currently not practical. Makes sense to me. So I'd leave

Re: [PR] feat: add always_true and always_false predicate [iceberg-rust]

2024-03-09 Thread via GitHub
Dysprosium0626 commented on PR #227: URL: https://github.com/apache/iceberg-rust/pull/227#issuecomment-1987082954 Hi @liurenjie1024. Thanks for your response! So, we do not need extra `AlwaysTrue`/`AlwaysFalse` predicate enum because there are already there in `BoundPredicate` (I've noticed

Re: [PR] Build: Bump mkdocs-material from 9.5.9 to 9.5.12 [iceberg]

2024-03-09 Thread via GitHub
dependabot[bot] closed pull request #9859: Build: Bump mkdocs-material from 9.5.9 to 9.5.12 URL: https://github.com/apache/iceberg/pull/9859 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

[PR] Build: Bump mkdocs-material from 9.5.9 to 9.5.13 [iceberg]

2024-03-09 Thread via GitHub
dependabot[bot] opened a new pull request, #9906: URL: https://github.com/apache/iceberg/pull/9906 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.9 to 9.5.13. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>mkdocs-m

Re: [PR] Build: Bump mkdocs-material from 9.5.9 to 9.5.12 [iceberg]

2024-03-09 Thread via GitHub
dependabot[bot] commented on PR #9859: URL: https://github.com/apache/iceberg/pull/9859#issuecomment-1987062676 Superseded by #9906. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] feat: Read Parquet data file with projection [iceberg-rust]

2024-03-09 Thread via GitHub
viirya commented on code in PR #245: URL: https://github.com/apache/iceberg-rust/pull/245#discussion_r1518693422 ## crates/iceberg/src/scan.rs: ## @@ -383,18 +389,29 @@ mod tests { // prepare data let schema = { -let fields = -

Re: [PR] Kafka Connect: Record converters [iceberg]

2024-03-09 Thread via GitHub
bryanck commented on PR #9641: URL: https://github.com/apache/iceberg/pull/9641#issuecomment-1987029928 I believe I've addressed all comments, LMK if there is anything else. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Kafka Connect: Record converters [iceberg]

2024-03-09 Thread via GitHub
bryanck commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1518698166 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java: ## @@ -0,0 +1,508 @@ +/* + * Licensed to the Apache Software Foundati

Re: [PR] Kafka Connect: Record converters [iceberg]

2024-03-09 Thread via GitHub
bryanck commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1518697497 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordProjection.java: ## @@ -0,0 +1,200 @@ +/* + * Licensed to the Apache Software Foundat

Re: [PR] Kafka Connect: Record converters [iceberg]

2024-03-09 Thread via GitHub
bryanck commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1518697202 ## kafka-connect/kafka-connect/src/test/java/org/apache/iceberg/connect/data/RecordConverterTest.java: ## @@ -0,0 +1,907 @@ +/* + * Licensed to the Apache Software Foun

Re: [PR] Kafka Connect: Record converters [iceberg]

2024-03-09 Thread via GitHub
bryanck commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1518696748 ## kafka-connect/kafka-connect/src/test/java/org/apache/iceberg/connect/data/RecordConverterTest.java: ## @@ -0,0 +1,907 @@ +/* + * Licensed to the Apache Software Foun

Re: [I] Add hive metastore catalog support [iceberg-rust]

2024-03-09 Thread via GitHub
liurenjie1024 commented on issue #113: URL: https://github.com/apache/iceberg-rust/issues/113#issuecomment-1987025678 Hi, @marvinlanhenke I think currently we can leave `create_table` unimplemented since it has some dependencies, such as transaction api and table metadata builder, otherwis

[I] Add hive metastore catalog support [iceberg-rust]

2024-03-09 Thread via GitHub
Xuanwo opened a new issue, #113: URL: https://github.com/apache/iceberg-rust/issues/113 ## Tasks - [x] https://github.com/apache/iceberg-rust/pull/112 - [ ] Implement catalog trait for hms -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [I] Add hive metastore catalog support [iceberg-rust]

2024-03-09 Thread via GitHub
liurenjie1024 closed issue #113: Add hive metastore catalog support URL: https://github.com/apache/iceberg-rust/issues/113 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [PR] Add hive metastore catalog support (part 1/2) [iceberg-rust]

2024-03-09 Thread via GitHub
liurenjie1024 merged PR #237: URL: https://github.com/apache/iceberg-rust/pull/237 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Spark: Add a test to check if the bloom filters are added to the parquet files [iceberg]

2024-03-09 Thread via GitHub
hussein-awala commented on PR #9902: URL: https://github.com/apache/iceberg/pull/9902#issuecomment-1987021776 I just fixed it, and I updated the tests for the other versions to make them similar to 3.5. It looks like bloom filters are written by Spark, I will re-test my jobs to under

Re: [PR] Kafka Connect: Record converters [iceberg]

2024-03-09 Thread via GitHub
bryanck commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1518694023 ## kafka-connect/kafka-connect/src/test/java/org/apache/iceberg/connect/data/RecordConverterTest.java: ## @@ -0,0 +1,907 @@ +/* + * Licensed to the Apache Software Foun

Re: [I] Absolute address can be optimized to relative address in file(*.metadata.json.)? [iceberg]

2024-03-09 Thread via GitHub
github-actions[bot] commented on issue #1938: URL: https://github.com/apache/iceberg/issues/1938#issuecomment-1987019564 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Kafka Connect: Record converters [iceberg]

2024-03-09 Thread via GitHub
bryanck commented on code in PR #9641: URL: https://github.com/apache/iceberg/pull/9641#discussion_r1518693577 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java: ## @@ -0,0 +1,508 @@ +/* + * Licensed to the Apache Software Foundati

Re: [I] Support for role based access of HadoopCatalog table listing [iceberg]

2024-03-09 Thread via GitHub
github-actions[bot] commented on issue #1941: URL: https://github.com/apache/iceberg/issues/1941#issuecomment-1987019570 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] feat: Read Parquet data file with projection [iceberg-rust]

2024-03-09 Thread via GitHub
viirya commented on code in PR #245: URL: https://github.com/apache/iceberg-rust/pull/245#discussion_r1518693422 ## crates/iceberg/src/scan.rs: ## @@ -383,18 +389,29 @@ mod tests { // prepare data let schema = { -let fields = -

Re: [I] When deleting data from Iceberg tables in Spark, the current approach is to delete all data and then rewrite the new data, which is very wasteful in terms of storage space and computation. How

2024-03-09 Thread via GitHub
hussein-awala commented on issue #9891: URL: https://github.com/apache/iceberg/issues/9891#issuecomment-1987005017 > When deleting data from Iceberg tables in Spark How do you delete the data? could you provide an example for your code? -- This is an automated message from the Apach

Re: [PR] Spark: Add a test to check if the bloom filters are added to the parquet files [iceberg]

2024-03-09 Thread via GitHub
hussein-awala commented on PR #9902: URL: https://github.com/apache/iceberg/pull/9902#issuecomment-1987003661 @amogh-jahagirdar I tried to use spark to write the data (check https://github.com/apache/iceberg/pull/9902/commits/25815479628015143551c2379be5608e2dd09bd7), but I had an issue wit

[PR] [Bug Fix] Allow Partition data to be nullable in ManifestEntry [iceberg-python]

2024-03-09 Thread via GitHub
syun64 opened a new pull request, #509: URL: https://github.com/apache/iceberg-python/pull/509 Similar to how the [partition fields are already initialized](https://github.com/apache/iceberg-python/blob/pyiceberg-0.6.x/pyiceberg/partitioning.py#L193) with `required=False` in `partition_type

Re: [PR] refactor: Make plan_files as asynchronous stream [iceberg-rust]

2024-03-09 Thread via GitHub
viirya commented on code in PR #243: URL: https://github.com/apache/iceberg-rust/pull/243#discussion_r1518675699 ## crates/iceberg/src/scan.rs: ## @@ -143,37 +144,43 @@ pub type FileScanTaskStream = BoxStream<'static, crate::Result>; impl TableScan { /// Returns a stream

Re: [PR] refactor: Make plan_files as asynchronous stream [iceberg-rust]

2024-03-09 Thread via GitHub
viirya commented on code in PR #243: URL: https://github.com/apache/iceberg-rust/pull/243#discussion_r1518675699 ## crates/iceberg/src/scan.rs: ## @@ -143,37 +144,43 @@ pub type FileScanTaskStream = BoxStream<'static, crate::Result>; impl TableScan { /// Returns a stream

[PR] refactor: Make plan_files as asynchronous stream [iceberg-rust]

2024-03-09 Thread via GitHub
viirya opened a new pull request, #243: URL: https://github.com/apache/iceberg-rust/pull/243 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

[PR] [Bug fix] update name mapping in transaction [iceberg-python]

2024-03-09 Thread via GitHub
syun64 opened a new pull request, #508: URL: https://github.com/apache/iceberg-python/pull/508 Also update name mapping if schema is updated through a Transaction. [Related PR](https://github.com/apache/iceberg-python/pull/441) This PR fixes the current behavior, which only upda

Re: [PR] Spark: Add a test to check if the bloom filters are added to the parquet files [iceberg]

2024-03-09 Thread via GitHub
hussein-awala commented on code in PR #9902: URL: https://github.com/apache/iceberg/pull/9902#discussion_r1518648135 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java: ## @@ -367,11 +374,28 @@ public void testReadWithFilter() {

Re: [I] Parquet bloom filter doesn't work with nested fields [iceberg]

2024-03-09 Thread via GitHub
hussein-awala commented on issue #9898: URL: https://github.com/apache/iceberg/issues/9898#issuecomment-1986954890 I use Spark 3.5, Iceberg 1.4.3, and Glue Catalog. > and bloom filters for nested type appear to be written out Interesting, I will retest it on Monday morning. -- Th

Re: [I] Add hive metastore catalog support [iceberg-rust]

2024-03-09 Thread via GitHub
marvinlanhenke commented on issue #113: URL: https://github.com/apache/iceberg-rust/issues/113#issuecomment-1986952590 @Xuanwo @liurenjie1024 I'm currently looking at the table operations in particular `create_table` and I'm having some trouble understanding the handling of the metadata.

Re: [PR] Improve the InMemory Catalog Implementation [iceberg-python]

2024-03-09 Thread via GitHub
kevinjqliu commented on PR #289: URL: https://github.com/apache/iceberg-python/pull/289#issuecomment-1986928262 @bitsondatadev > I also am curious to know if this catalog still makes sense with inmem sqlite? We agreed to not move this implementation to production. See https:/

Re: [PR] Improve the InMemory Catalog Implementation [iceberg-python]

2024-03-09 Thread via GitHub
bitsondatadev commented on PR #289: URL: https://github.com/apache/iceberg-python/pull/289#issuecomment-1986926827 > Thanks for working on this @kevinjqliu. The issues was created a long time ago, before we had the SqlCatalog with sqlite support. Sqlite can also work [in memory](https://www

Re: [PR] Add hive metastore catalog support (part 1/2) [iceberg-rust]

2024-03-09 Thread via GitHub
marvinlanhenke commented on PR #237: URL: https://github.com/apache/iceberg-rust/pull/237#issuecomment-1986899447 @liurenjie1024 @Xuanwo @Fokko I think I covered all the suggestions and improvements; I think we're ready to merge the first part of the hms_catalog? Thanks again for the

Re: [I] Rename `stat_table` in `Catalog` trait [iceberg-rust]

2024-03-09 Thread via GitHub
marvinlanhenke commented on issue #236: URL: https://github.com/apache/iceberg-rust/issues/236#issuecomment-1986898447 @liurenjie1024 @Xuanwo Perhaps this would be a `good-first-issue` ? marker: https://github.com/apache/iceberg-rust/blob/main/crates/iceberg/src/catalog/mod.rs#

Re: [PR] Add hive metastore catalog support (part 1/2) [iceberg-rust]

2024-03-09 Thread via GitHub
marvinlanhenke commented on code in PR #237: URL: https://github.com/apache/iceberg-rust/pull/237#discussion_r1518608227 ## crates/catalog/hms/testdata/hms_catalog/Dockerfile: ## Review Comment: @Fokko thanks for linking your Dockerfile; I integrated it and I am hap

Re: [PR] Increase Partition Start Id to 10000. [iceberg]

2024-03-09 Thread via GitHub
bk-mz commented on PR #9904: URL: https://github.com/apache/iceberg/pull/9904#issuecomment-1986896298 ![image](https://github.com/apache/iceberg/assets/892781/c1169556-f2cf-4cf4-b581-2ff4a77e6c16) So the fix should be something else. -- This is an automated message from the Apache

Re: [PR] Increase Partition Start Id to 10000. [iceberg]

2024-03-09 Thread via GitHub
bk-mz closed pull request #9904: Increase Partition Start Id to 1. URL: https://github.com/apache/iceberg/pull/9904 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] Clarify pagination description [iceberg]

2024-03-09 Thread via GitHub
danielcweeks commented on PR #9872: URL: https://github.com/apache/iceberg/pull/9872#issuecomment-1986884426 I caught up with @rahil-c and discussed the plan forward here and he offered to consolidate all of the feedback across this and the other PR. Seems like best approach is just to clo

Re: [I] Bug: No associated function values.rs // DateTime::from_timestamp_micros [iceberg-rust]

2024-03-09 Thread via GitHub
marvinlanhenke closed issue #240: Bug: No associated function values.rs // DateTime::from_timestamp_micros URL: https://github.com/apache/iceberg-rust/issues/240 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [I] Bug: No associated function values.rs // DateTime::from_timestamp_micros [iceberg-rust]

2024-03-09 Thread via GitHub
marvinlanhenke commented on issue #240: URL: https://github.com/apache/iceberg-rust/issues/240#issuecomment-1986881252 > > @liurenjie1024 > > I think this was the commit that introduced the change? [f61d475](https://github.com/apache/iceberg-rust/commit/f61d47509e95b692f01fca23c53ff04349e

Re: [I] Bug: No associated function values.rs // DateTime::from_timestamp_micros [iceberg-rust]

2024-03-09 Thread via GitHub
liurenjie1024 commented on issue #240: URL: https://github.com/apache/iceberg-rust/issues/240#issuecomment-1986873611 > @liurenjie1024 > > I think this was the commit that introduced the change? https://github.com/apache/iceberg-rust/commit/f61d47509e95b692f01fca23c53ff04349ed8849

Re: [PR] [WIP] Add `PartitionEvaluator` to allow filtering of files in a table scan (Issue #152) [iceberg-rust]

2024-03-09 Thread via GitHub
sdd commented on PR #241: URL: https://github.com/apache/iceberg-rust/pull/241#issuecomment-1986868266 @liurenjie1024 PTAL: Your thoughts on this very much appreciated. This also depends on your predicate binding PR, https://github.com/apache/iceberg-rust/pull/231, getting merged. @F

[PR] [WIP] Add `PartitionEvaluator` to allow filtering of files in a table scan (Issue #152) [iceberg-rust]

2024-03-09 Thread via GitHub
sdd opened a new pull request, #241: URL: https://github.com/apache/iceberg-rust/pull/241 A very early-stage PR to propose an initial design for a partition evaluator to reject files in the table scan's file planner that don't match the scan's filter. Which issue does this PR close?

Re: [I] Bug: No associated function values.rs // DateTime::from_timestamp_micros [iceberg-rust]

2024-03-09 Thread via GitHub
marvinlanhenke commented on issue #240: URL: https://github.com/apache/iceberg-rust/issues/240#issuecomment-1986866126 @liurenjie1024 I think this was the commit that introduced the change? https://github.com/apache/iceberg-rust/commit/f61d47509e95b692f01fca23c53ff04349ed8849 -- This

Re: [I] feat: Implement data file metrics evaluator to prune data files using filter. [iceberg-rust]

2024-03-09 Thread via GitHub
sdd commented on issue #152: URL: https://github.com/apache/iceberg-rust/issues/152#issuecomment-1986863096 I've made a start on this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[I] Bug: No associated function values.rs // DateTime::from_timestamp_micros [iceberg-rust]

2024-03-09 Thread via GitHub
marvinlanhenke opened a new issue, #240: URL: https://github.com/apache/iceberg-rust/issues/240 I just merged main into my dev_branch - now, I cannot compile - perhaps its on my side, since i am not sure how this could slip through CI? https://github.com/apache/iceberg-rust/blob/main

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-03-09 Thread via GitHub
liurenjie1024 commented on PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#issuecomment-1986860460 > > > It would also be good to assert if they are there. > > > > > > Sorry, I'm not sure what this means. Did you mean assert like following? > > > >

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-03-09 Thread via GitHub
liurenjie1024 merged PR #176: URL: https://github.com/apache/iceberg-rust/pull/176 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-03-09 Thread via GitHub
ZENOTME commented on PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#issuecomment-1986854100 > > It would also be good to assert if they are there. > > Sorry, I'm not sure what this means. Did you mean assert like following? > > ``` > let schema = ...; > asse

Re: [PR] Add hive metastore catalog support (part 1/2) [iceberg-rust]

2024-03-09 Thread via GitHub
marvinlanhenke commented on code in PR #237: URL: https://github.com/apache/iceberg-rust/pull/237#discussion_r1518568918 ## crates/catalog/hms/src/catalog.rs: ## @@ -95,9 +106,156 @@ impl HmsCatalog { client: HmsClient(client), }) } + +/// Create a

Re: [PR] Add hive metastore catalog support (part 1/2) [iceberg-rust]

2024-03-09 Thread via GitHub
marvinlanhenke commented on code in PR #237: URL: https://github.com/apache/iceberg-rust/pull/237#discussion_r1518568918 ## crates/catalog/hms/src/catalog.rs: ## @@ -95,9 +106,156 @@ impl HmsCatalog { client: HmsClient(client), }) } + +/// Create a

Re: [PR] feat: Implement binding expression [iceberg-rust]

2024-03-09 Thread via GitHub
liurenjie1024 commented on PR #231: URL: https://github.com/apache/iceberg-rust/pull/231#issuecomment-1986848432 > Changes from review look good to me. Because we're now doing the NOT-elimination here at bind time, does this mean that this PR effectively completes #150 as well? Seem

Re: [PR] Add hive metastore catalog support (part 1/2) [iceberg-rust]

2024-03-09 Thread via GitHub
liurenjie1024 commented on code in PR #237: URL: https://github.com/apache/iceberg-rust/pull/237#discussion_r1518567904 ## crates/catalog/hms/src/catalog.rs: ## @@ -95,9 +106,156 @@ impl HmsCatalog { client: HmsClient(client), }) } + +/// Create an

Re: [PR] Add hive metastore catalog support (part 1/2) [iceberg-rust]

2024-03-09 Thread via GitHub
liurenjie1024 commented on code in PR #237: URL: https://github.com/apache/iceberg-rust/pull/237#discussion_r1518567904 ## crates/catalog/hms/src/catalog.rs: ## @@ -95,9 +106,156 @@ impl HmsCatalog { client: HmsClient(client), }) } + +/// Create an

Re: [PR] feat: Implement binding expression [iceberg-rust]

2024-03-09 Thread via GitHub
sdd commented on PR #231: URL: https://github.com/apache/iceberg-rust/pull/231#issuecomment-1986847123 Changes from review look good to me. Because we're now doing the NOT-elimination here at bind time, does this mean that this PR effectively completes https://github.com/apache/iceberg-rust

Re: [PR] Add hive metastore catalog support (part 1/2) [iceberg-rust]

2024-03-09 Thread via GitHub
liurenjie1024 commented on code in PR #237: URL: https://github.com/apache/iceberg-rust/pull/237#discussion_r1518567066 ## crates/catalog/hms/src/utils.rs: ## @@ -40,3 +54,258 @@ pub fn from_io_error(error: io::Error) -> Error { ) .with_source(error) } + +/// Create a

Re: [PR] Add hive metastore catalog support (part 1/2) [iceberg-rust]

2024-03-09 Thread via GitHub
marvinlanhenke commented on code in PR #237: URL: https://github.com/apache/iceberg-rust/pull/237#discussion_r1518564967 ## crates/catalog/hms/src/catalog.rs: ## @@ -95,9 +106,156 @@ impl HmsCatalog { client: HmsClient(client), }) } + +/// Create a

Re: [PR] Add hive metastore catalog support (part 1/2) [iceberg-rust]

2024-03-09 Thread via GitHub
liurenjie1024 commented on code in PR #237: URL: https://github.com/apache/iceberg-rust/pull/237#discussion_r1518566810 ## crates/catalog/hms/testdata/hms_catalog/docker-compose.yaml: ## @@ -0,0 +1,48 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more co

Re: [PR] feat: add always_true and always_false predicate [iceberg-rust]

2024-03-09 Thread via GitHub
liurenjie1024 commented on PR #227: URL: https://github.com/apache/iceberg-rust/pull/227#issuecomment-1986844698 Hi, @Dysprosium0626 Thanks for the pr. I think I made a mistake since we already have `AlwaysTrue`/`AlwaysFalse` in `BoundPredicate`. It doesn't make sense to me to add `AlwaysTr

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-03-09 Thread via GitHub
ZENOTME commented on PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#issuecomment-1986843681 > cc @ZENOTME Would you mind to fix the conflicts? Sorry, don't notice it before. I fix it now. -- This is an automated message from the Apache Git Service. To respond to the m

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-03-09 Thread via GitHub
liurenjie1024 commented on PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#issuecomment-1986843324 cc @ZENOTME Would you mind to fix the conflicts? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Add hive metastore catalog support (part 1/2) [iceberg-rust]

2024-03-09 Thread via GitHub
marvinlanhenke commented on code in PR #237: URL: https://github.com/apache/iceberg-rust/pull/237#discussion_r1518565673 ## crates/catalog/hms/testdata/hms_catalog/Dockerfile: ## Review Comment: I also like keeping the Dockerfile as part of the repo - however I am not sati

Re: [PR] Add hive metastore catalog support (part 1/2) [iceberg-rust]

2024-03-09 Thread via GitHub
marvinlanhenke commented on code in PR #237: URL: https://github.com/apache/iceberg-rust/pull/237#discussion_r1518565464 ## crates/catalog/hms/testdata/hms_catalog/docker-compose.yaml: ## @@ -0,0 +1,48 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more c

Re: [PR] Add hive metastore catalog support (part 1/2) [iceberg-rust]

2024-03-09 Thread via GitHub
marvinlanhenke commented on code in PR #237: URL: https://github.com/apache/iceberg-rust/pull/237#discussion_r1518565316 ## crates/catalog/hms/Cargo.toml: ## @@ -33,5 +33,12 @@ anyhow = { workspace = true } async-trait = { workspace = true } hive_metastore = { workspace = true

Re: [PR] Add hive metastore catalog support (part 1/2) [iceberg-rust]

2024-03-09 Thread via GitHub
marvinlanhenke commented on code in PR #237: URL: https://github.com/apache/iceberg-rust/pull/237#discussion_r1518565281 ## crates/catalog/hms/src/catalog.rs: ## @@ -95,9 +106,156 @@ impl HmsCatalog { client: HmsClient(client), }) } + +/// Create a

Re: [PR] Add hive metastore catalog support (part 1/2) [iceberg-rust]

2024-03-09 Thread via GitHub
marvinlanhenke commented on code in PR #237: URL: https://github.com/apache/iceberg-rust/pull/237#discussion_r1518565050 ## crates/catalog/hms/Cargo.toml: ## @@ -33,5 +33,12 @@ anyhow = { workspace = true } async-trait = { workspace = true } hive_metastore = { workspace = true

Re: [PR] Add hive metastore catalog support (part 1/2) [iceberg-rust]

2024-03-09 Thread via GitHub
marvinlanhenke commented on code in PR #237: URL: https://github.com/apache/iceberg-rust/pull/237#discussion_r1518564967 ## crates/catalog/hms/src/catalog.rs: ## @@ -95,9 +106,156 @@ impl HmsCatalog { client: HmsClient(client), }) } + +/// Create a

Re: [PR] feat: Implement binding expression [iceberg-rust]

2024-03-09 Thread via GitHub
liurenjie1024 commented on code in PR #231: URL: https://github.com/apache/iceberg-rust/pull/231#discussion_r1518562233 ## crates/iceberg/src/expr/predicate.rs: ## @@ -55,6 +60,29 @@ impl LogicalExpression { } } +impl Bind for LogicalExpression { +type Bound = Logica

[PR] Support convert orc timestamptz [iceberg]

2024-03-09 Thread via GitHub
ming95 opened a new pull request, #9905: URL: https://github.com/apache/iceberg/pull/9905 ### What changes were proposed in this pull request? A user was attempting to convert an ORC backed external table in hive to a Iceberg table using the migrate command but was immediately met

Re: [PR] Increase Partition Start Id to 10000. [iceberg]

2024-03-09 Thread via GitHub
bk-mz commented on code in PR #9904: URL: https://github.com/apache/iceberg/pull/9904#discussion_r1518539338 ## format/spec.md: ## @@ -1142,6 +1142,7 @@ Supported partition transforms are listed below. |**`month`**|`JSON string: "month"`|`"month"`| |**`day`**|`JSON string: "da