Hi. Thanks for starting this thread.
I agree with Jinsong's point of view, a percentage might be more appropriate. In one case, I have only one pending table, and I have generated tasks that exceed the quota. Can I break the quota limit? Best, Qishang Zhong 陈xx <[email protected]> 于2025年7月22日周二 19:14写道： > Thanks for your suggestion. > > Indeed, if we set the default quota for tables to a fixed value, it is > still possible to lead to a situation where a single table monopolizes all > optimizer resources when there are very few optimizer resources. > Conversely, when optimizer resources are abundant, it may result in > consistently low optimizing efficiency for the table having multiple > self-optimizing tasks, and cause inefficient utilization of optimizer > resources. Therefore, setting the quota as a percentage to all available > optimizer resources is a more appropriate approach. I will proceed with the > corresponding modifications accordingly. > > Best, > XixiChen > > Jinsong Zhou <[email protected]> 于2025年7月22日周二 18:01写道： > > > Hi, > > > > Thanks for bringing up this improvement. > > > > This improvement is indeed valuable. In my production practice, we once > > encountered a situation where a single table suddenly consumed all > > resources, causing all other tables to enter a pending status. We need > the > > capability to limit the maximum resources a single table can use. > > > > However, I'd like to discuss how to set the default quota for tables. I > > believe in most cases, users won't configure individual quotas for each > > table, making default quotas particularly important. Rather than using a > > fixed value, a percentage might be more appropriate - for example, 50%, > > indicating a table can only consume half of the entire group's resources. > > > > Best, > > Jinsong > > > > > > > > On Tue, Jul 22, 2025 at 5:52 PM 陈xx <[email protected]> wrote: > > > > > Hi devs: > > > > > > We would like to start a discussion about AIP-1: Optimizing Allocation > > and > > > Schedule Priority of Optimizer resources for Tables[1]. > > > > > > An optimizer group comprises a collection of optimizers, where each > > > optimizer instance typically contains multiple threads, with each > > optimizer > > > thread responsible for executing a single optimizing task. When > multiple > > > self-optimizing tasks are pending and optimizer resources are limited, > > > tasks originating from the same table may monopolize all available > > > resources in the absence of proper constraints. > > > So, we propose to optimize allocation and schedule priority of > > > optimizer resources for tables. > > > Looking forward to hearing from you. > > > > > > [1] https://cwiki.apache.org/confluence/x/bQ5JFg > > > > > > Best > > > XixiChen > > > > > >
