alamb commented on PR #23026: URL: https://github.com/apache/datafusion/pull/23026#issuecomment-4771120579
> I must admit our true motivation is Ballista, and halo rows make it easy to distribute. The optimal solution would probably be segment trees within node and hallow rows between nodes. I think one thing that Spark does is to effectively re-optimize the plan after certain stages (where the data was written to disk / shuffle files anyways). I vaguely remember Ballista / @milenkovicm was looking into something similar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
