This is an automated email from the ASF dual-hosted git repository.
zhoujinsong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/amoro.git
The following commit(s) were added to refs/heads/master by this push:
new 3cf72b706 Explain dynamic resource allocation (DRA) for optimizer
groups (#4383)
3cf72b706 is described below
commit 3cf72b7069dbe2d65311a0f4ab8b4f0e22a4c3ba
Author: Jiwon Park <[email protected]>
AuthorDate: Sun Sep 20 16:29:26 2026 +0900
Explain dynamic resource allocation (DRA) for optimizer groups (#4383)
Add a Dynamic resource allocation subsection to the Optimizer group
guide covering how scale-up/scale-down are triggered, constraints,
and a config example. Also drop the AIP-5 reference from the
dynamic-allocation.enabled property row for consistency with the
rest of the page, which doesn't cite proposal numbers.
Signed-off-by: Jiwon Park <[email protected]>
---
docs/admin-guides/managing-optimizers.md | 30 +++++++++++++++++++++++++++++-
1 file changed, 29 insertions(+), 1 deletion(-)
diff --git a/docs/admin-guides/managing-optimizers.md
b/docs/admin-guides/managing-optimizers.md
index 336aec73a..5a86c3f70 100644
--- a/docs/admin-guides/managing-optimizers.md
+++ b/docs/admin-guides/managing-optimizers.md
@@ -290,7 +290,7 @@ The optimizer group supports the following properties:
| cache-timeout | All | No | 10min
| Timeout
in optimizer cache.
[...]
| min-parallelism | All | No | 0
|
Deprecated since 0.9.0 in favor of `dynamic-allocation.min-parallelism`; still
honored as a fallback. The minimum total parallelism (CPU cores) that the
optimizer group should maintain. When the total cores of running optimizers
fall below this value, `OptimizerGroupKeeper` will automatically scale out new
optimizers. Set to `0` to disable [...]
| shutdown-timeout-ms | All | No | 600000(10min)
|
Graceful shutdown timeout in milliseconds. On shutdown the optimizer waits up
to this long for in-progress tasks to complete before force-interrupting them.
For Kubernetes optimizers, the pod's `terminationGracePeriodSeconds` is derived
from this value plus a 30s buffer. For Flink optimizers, the effective wait is
additionally capped below [...]
-| dynamic-allocation.enabled | All | No | false
| Whether
to enable dynamic resource allocation (AIP-5) for this group: the group then
grows its optimizer count automatically in response to optimizing demand,
bounded by `dynamic-allocation.max-parallelism`. Not supported on
externally-registered optimizers. Manual optimizer operations
(scale-out/release via the dashboard) are not recommen [...]
+| dynamic-allocation.enabled | All | No | false
| Whether
to enable dynamic resource allocation for this group: the group then grows its
optimizer count automatically in response to optimizing demand, bounded by
`dynamic-allocation.max-parallelism`. Not supported on externally-registered
optimizers. Manual optimizer operations (scale-out/release via the dashboard)
are not recommended on g [...]
| dynamic-allocation.min-parallelism | All | No | 0
| Lower
bound on the group's total optimizer threads under dynamic allocation.
Supersedes the deprecated flat `min-parallelism`.
[...]
| dynamic-allocation.max-parallelism | All | Yes (when enabled) | N/A
| Upper
bound on the group's total optimizer threads under dynamic allocation; must not
exceed 1024. Also configure Kubernetes `ResourceQuota`/`LimitRange` as the
authoritative cluster-side limit.
[...]
| dynamic-allocation.executor-parallelism | All | No | 1
| Threads
per optimizer instance created by dynamic allocation (the scaling unit, like
Spark's `spark.executor.cores`). The floor and the cap must be reachable in
units of this size. For Kubernetes groups a value of 4–8 is recommended so
per-pod JVM overhead is shared across threads.
[...]
@@ -311,6 +311,34 @@ To better utilize the resources of Flink Optimizer, it is
recommended to add the
* When using `shutdown-timeout-ms` with a Flink Optimizer, also raise
`flink-conf.task.cancellation.timeout` (default 180000) accordingly — the
graceful drain on job cancellation is capped below Flink's cancellation
watchdog, which otherwise fails the whole TaskManager.
{{< /hint >}}
+### Dynamic resource allocation
+
+Dynamic resource allocation (DRA) lets a group grow and shrink its optimizer
count automatically based on optimizing demand, instead of running a fixed
number of optimizers. Enable it by setting `dynamic-allocation.enabled` to
`true` on the group; once enabled, `dynamic-allocation.min-parallelism` and
`dynamic-allocation.max-parallelism` take over from the deprecated flat
`min-parallelism`.
+
+**Scale-up**: when optimizing demand (queued tasks) persists for
`dynamic-allocation.scheduler-backlog-timeout`, the group scales out for the
first time. While demand keeps persisting, it scales out again every
`dynamic-allocation.sustained-backlog-timeout`, up to
`dynamic-allocation.max-parallelism`.
+
+**Scale-down**: an optimizer that has been idle for
`dynamic-allocation.executor-idle-timeout` becomes a scale-down candidate. At
most one optimizer is removed per evaluation round, and only while the group
has no scale-up demand, with at least `dynamic-allocation.scale-down-cooldown`
between removals. A scaled-down optimizer is drained gracefully — it stops
receiving new tasks and is removed once its in-flight tasks finish; if draining
exceeds `dynamic-allocation.drain-timeout`, it is f [...]
+
+{{< hint info >}}
+* DRA is not supported for externally-registered optimizers.
+* Avoid manual scale-out/release from the dashboard on a group with DRA
enabled — it fights with the auto scaler.
+* `dynamic-allocation.min-parallelism` and
`dynamic-allocation.max-parallelism` must both be reachable as multiples of
`dynamic-allocation.executor-parallelism`, the number of threads added or
removed per scaling step.
+{{< /hint >}}
+
+An example enabling DRA on a Kubernetes optimizer group, entered in the
group's `properties`:
+
+```yaml
+dynamic-allocation.enabled: true # Turn on dynamic
resource allocation
+dynamic-allocation.min-parallelism: 8 # Keep at least 1
optimizer (executor-parallelism=8) running
+dynamic-allocation.max-parallelism: 64 # Scale out to at
most 8 optimizers
+dynamic-allocation.executor-parallelism: 8 # Threads per
optimizer pod; 4-8 is recommended on Kubernetes
+dynamic-allocation.scheduler-backlog-timeout: 1min # Wait 1 min of
sustained backlog before the first scale-out
+dynamic-allocation.sustained-backlog-timeout: 30s # Re-evaluate and
scale out every 30s while backlog persists
+dynamic-allocation.executor-idle-timeout: 10min # Consider an
optimizer idle after 10 min without tasks
+dynamic-allocation.scale-down-cooldown: 1min # Wait at least 1
min between scale-down removals
+dynamic-allocation.drain-timeout: 15min # Force-remove a
draining optimizer after 15 min
+```
+
### Edit optimizer group
You can click the `edit` button on the `Optimizer Groups` page to modify the
configuration of the Optimizer group.
