liurenjie1024 commented on issue #1036: URL: https://github.com/apache/iceberg-rust/issues/1036#issuecomment-2694070892
Thanks @Xuanwo for raising this, I think in general it's a good idea to have a standalone module for simple data processing, just like what we already have in [java](https://github.com/apache/iceberg/blob/e230f5d79d82a50439029db5c73f8b59497b2e9f/data), e.g. an `iceberg-data` module. That's to say we have following crates: ``` iceberg / \ iceberg-data iceberg-datafusion ``` The `iceberg` crate is similar to the `iceber-core` + `iceberg-api` module in java, which contains api for manipulating metadata, and it's supposed to be compute engine independent. Also it's used for planning scan tasks for compute engines. The `iceberg-data` crate contains necessary implementation for executing table scan in local machine, and doesn't depend on any execution engine. We may add feature of appending data to table, but row level modifications seem somehow challenging at this time. The `iceberg-datafusion` crate contains integration with datafusion, which allows users to execute sql against iceberg, and of course powered by datafusion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
