Hi Joice, Just confirmed with the team: The code related to the spill-to-disk feature has been merged into the master branch, but after our evaluation, we found that including this work in version 3.0.X could lead to instability, so we plan to release it with our next major release instead, which should be 3.1.0. And 3.1.0 will come with other improvements in addition to spill-to-disk, so it should help with the problems you mentioned.
Thanks for keeping an eye on Apache Doris. Best regards, Shirley Hu <http://www.linkedin.com/in/shirley-hu-apache-doris> Developer Advocate of Apache Doris <https://doris.apache.org/> Joice Jacob <joicejacob1...@gmail.com> 于2025年2月28日周五 19:24写道： > Hi shirley and dev team, > As we continue our evaluation, we would like to check whether the upcoming > *Apache > Doris 3.0.4* version includes any improvements related to query execution > under constrained resources. Specifically, we are interested in: > > - Any configuration options or enhancements that allow large queries > to complete successfully instead of failing due to memory limitations. > - Mechanisms that enable queries to adaptively utilize available > resources without requiring cluster scaling. > > Could you please provide an update on whether these capabilities are > included in the *3.0.4 release*, or if they are planned for future > versions? > > Looking forward to your insights. > > Best regards, > Joice Jacob > > On Thu, Dec 19, 2024 at 5:54 PM Joice Jacob <joicejacob1...@gmail.com> > wrote: > >> Dear Shirley, >> Thank you for the update! I appreciate the clarification regarding the >> release schedule. >> I’ll plan to give it a spin for testing purposes before the official >> release. >> Best regards, >> Joice >> >> On Mon, Dec 16, 2024 at 9:35 AM Shirley Hu <shirleyhuws...@gmail.com> >> wrote: >> >>> Hi Joice, >>> >>> I just confirmed with @GuoLei. The official big release is scheduled for >>> late February. Before that, you can give it a spin for testing purposes. >>> We >>> are preparing the relevant docs lately. I will keep you posted :) >>> >>> Best, >>> >>> Shirley H. >>> Developer Advocate for Apache Doris >>> >>> Joice Jacob <joicejacob1...@gmail.com> 于2024年12月12日周四 14:28写道： >>> >>> > Dear Apache Doris Development Community, >>> > Thank you for the update and the link to the branch! >>> > >>> > Could you kindly let us know when the next release of Apache Doris, >>> > including the disk spill feature, is expected to be available? >>> > >>> > On Tue, Dec 10, 2024 at 10:07 AM Joice Jacob <joicejacob1...@gmail.com >>> > >>> > wrote: >>> > >>> > > Dear Apache Doris Development Community, >>> > > >>> > > Our team is very satisfied with the performance of Apache Doris, but >>> we >>> > > have a few concerns that we would like to clarify. >>> > > >>> > > We are planning to replace Hive with Doris for our use cases. In >>> Hive, >>> > > queries typically complete successfully using the available >>> resources, >>> > even >>> > > under constrained conditions. Most of our queries are small; >>> however, we >>> > > have some offline reporting use cases that require computing data >>> for 3, >>> > 6, >>> > > 9, or 15 months. >>> > > >>> > > In Doris, we have observed that queries fail when resources are >>> > > insufficient. This poses a challenge, as scaling the cluster to >>> > accommodate >>> > > large queries isn't feasible due to their infrequent execution. Even >>> when >>> > > the Doris cluster is entirely free, large queries sometimes fail due >>> to >>> > > resource limitations. >>> > > >>> > > Our question is: >>> > > >>> > > >>> > > 1. Is there any configuration in Doris that can ensure queries >>> > > complete successfully when resources are constrained, rather than >>> > failing? >>> > > 2. Can we expect a feature in future releases that allows large >>> > > queries to gracefully utilize available resources and complete >>> without >>> > > failing, even when memory is limited? >>> > > >>> > > We appreciate your insights and guidance on this matter. >>> > > >>> > > Best regards, >>> > > >>> > >>> >>
