Re: [I] Read Parquet data file with projection [iceberg-rust]

liurenjie1024 commented on issue #244: URL: https://github.com/apache/iceberg-rust/issues/244#issuecomment-2169270213 Close by #245 . Follow up issued tracked in #405 to make things more clear. -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [I] Read Parquet data file with projection [iceberg-rust]

liurenjie1024 closed issue #244: Read Parquet data file with projection URL: https://github.com/apache/iceberg-rust/issues/244 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [I] Read Parquet data file with projection [iceberg-rust]

liurenjie1024 commented on issue #244: URL: https://github.com/apache/iceberg-rust/issues/244#issuecomment-2029081596 > @liurenjie1024 Thanks for reviewing and merging #245. > > > Implement sth like [ArrowProjectionVisitor in python](https://github.com/apache/iceberg-python/blob/afdfa

Re: [I] Read Parquet data file with projection [iceberg-rust]

viirya commented on issue #244: URL: https://github.com/apache/iceberg-rust/issues/244#issuecomment-2029070755 @liurenjie1024 Thanks for reviewing and merging #245. > Implement sth like [ArrowProjectionVisitor in python](https://github.com/apache/iceberg-python/blob/afdfa351119090f09d

Re: [I] Read Parquet data file with projection [iceberg-rust]

liurenjie1024 closed issue #244: Read Parquet data file with projection URL: https://github.com/apache/iceberg-rust/issues/244 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [I] Read Parquet data file with projection [iceberg-rust]

viirya commented on issue #244: URL: https://github.com/apache/iceberg-rust/issues/244#issuecomment-2022062466 > > I think this Implement sth like ArrowProjectionVisitor is for the following two items? > > Yes, but with one extra requirement: reconstructing struct arrays. For example

Re: [I] Read Parquet data file with projection [iceberg-rust]

liurenjie1024 commented on issue #244: URL: https://github.com/apache/iceberg-rust/issues/244#issuecomment-2021856565 > I think this Implement sth like ArrowProjectionVisitor is for the following two items? Yes, but with one extra requirement: reconstructing struct arrays. For exampl

Re: [I] Read Parquet data file with projection [iceberg-rust]

viirya commented on issue #244: URL: https://github.com/apache/iceberg-rust/issues/244#issuecomment-2020967730 Thanks @liurenjie1024. I read through the summary above. I think currently #245 has done the first one `Collect leave column ids after schema pruning, and translate it

Re: [I] Read Parquet data file with projection [iceberg-rust]

liurenjie1024 commented on issue #244: URL: https://github.com/apache/iceberg-rust/issues/244#issuecomment-2019857203 # Problem Statement When converting parquet file to arrow in iceberg, there are several problems to take into consideration: 1. Field id mapping. Iceberg stores

Re: [I] Read Parquet data file with projection [iceberg-rust]

viirya commented on issue #244: URL: https://github.com/apache/iceberg-rust/issues/244#issuecomment-1991782353 Ah, I see. Thanks for the point. I began looking at Java implementation of #251. -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [I] Read Parquet data file with projection [iceberg-rust]

liurenjie1024 commented on issue #244: URL: https://github.com/apache/iceberg-rust/issues/244#issuecomment-1991000253 > But currently the select API cannot select deeply nested fields. In fact, current api is supposed to allow user to select deeply nested fields like `select("person.

Re: [I] Read Parquet data file with projection [iceberg-rust]

viirya commented on issue #244: URL: https://github.com/apache/iceberg-rust/issues/244#issuecomment-1988619183 Thank you @liurenjie1024 . I have looked the doc @sdd mentioned that describes Iceberg column projection. Looks like the projection is specified by field id as @sdd said due

Re: [I] Read Parquet data file with projection [iceberg-rust]

viirya commented on issue #244: URL: https://github.com/apache/iceberg-rust/issues/244#issuecomment-1987147807 Thank you @sdd. I will take a look the doc tomorrow and update the PR accordingly. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [I] Read Parquet data file with projection [iceberg-rust]

sdd commented on issue #244: URL: https://github.com/apache/iceberg-rust/issues/244#issuecomment-1987145982 Firstly, it's great to see someone else helping out on this - getting projection and filtering working on reads will unlock the most important (for me anyway 😅) use cases, so thanks f