villebro commented on issue #2299: URL: https://github.com/apache/datafusion-ballista/issues/2299#issuecomment-5298064894
> [@villebro](https://github.com/villebro) maybe related to your interests, I was thinking to create a "datafusion gateway" which could route queries based on pluggable policies to ballista or datafusion (running on k8s) i did not think i through 😀 @milenkovicm love the idea! also related, I've been thinking about adding the option to introduce tiered result storage down the road for perf/redundancy tuning. Some that come to mind: - in worker memory - on worker disk - object storage (large result sets) - Valkey-like distributed cache (small result sets) - etc With a dedicated result proxy it would be easy to introduce these incrementally as needed, as the client facing interface wouldn't change. Probably not high priority now, but maybe later.. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
