t3hw commented on PR #2342: URL: https://github.com/apache/iceberg-rust/pull/2342#issuecomment-4287325565
> > @blackmwk thanks for the feedback! Your comment sent me down a rabbit hole of past and future changes, and has given me a lot to think about. @vustef, same to you. > > It looks like #2298 will fold parallel scanning directly into the DataFusion integration - but a big part of my use case is lazily streaming entire snapshot deltas, partitions, or whole MOR tables into a downstream sink, and parallelization helps greatly. Pulling DataFusion in just for the parallel decode feels heavy for what's essentially a low-level scan -> stream -> transform -> sink pipeline. > > I'll close this PR. Out of curiosity though, would a separate PR for a thin wrapper crate focused on low-level parallelization utilities be worth opening? If so, where would be the right home for it? > > Perhaps we could start working towards the goals laid out in [this RFC](https://github.com/apache/iceberg-rust/blob/main/docs/rfcs/0001_modularize_iceberg_implementations.md) > > What kind of thin wrapper are you talking about? In rust it's quite trivial if you mean sending different file scan tasks to different async tasks, and I don't think it's worth doing it here. I was going off your "or we could maintain a minimum local parallel engine" comment. Work on #1819 doesn't seem to have started yet, so my idea was to begin moving in that direction. A low-level parallel engine eventually living in a separate crate seems like a reasonable fit, keeps the core iceberg crate focused on just the protocol. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
