sdd commented on issue #1036: URL: https://github.com/apache/iceberg-rust/issues/1036#issuecomment-2696442323
Hi @Xuanwo. I find myself agreeing in part with @a-agmon - whilst conceptually I agree that re-architecting into an `iceberg-core` and an `iceberg-engine-lite` does provide the opportunity to structure the API along seams that would allow us to provide better integration hooks to downstream engines, the `engine-lite` itself would be quite small. However I don't necessarily see that `engine-lite` itself being small / lightweight is necessarily a problem. I could see Scan and some of the high-level parts of ArrowReader being subsumed into this mini-engine, and even if that were all that was there, I think this provides value and results in a cleaner and more flexible architecture. I also agree that the current cache implementation is not flexible enough. I'm hoping to touch on this at the Iceberg Summit next month - for my own production service, I extend it further to cache more types of content than what we are already doing so in order to achieve very low latencies for small queries. I'd love a design that made this more pluggable and configurable so that users like me could make heavier use of the cache, but also so that users who are using external query engines can potentially cache less within iceberg-rust itself if they need to. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
