Dennis-Mircea Ciupitu created FLINK-39390:
---------------------------------------------
Summary: Add Extensibility to Autoscaler Components via the Plugin
Mechanism
Key: FLINK-39390
URL: https://issues.apache.org/jira/browse/FLINK-39390
Project: Flink
Issue Type: Improvement
Components: Autoscaler, Kubernetes Operator
Affects Versions: 1.15.4
Reporter: Dennis-Mircea Ciupitu
Fix For: 1.15.5
h1. Summary
Enable plugin-based extensibility for the Flink Autoscaler’s core components
({{{}ScalingRealizer{}}}, {{{}AutoScalerStateStore{}}},
{{{}AutoScalerEventHandler{}}}, {{{}KubernetesJobAutoScalerContext{}}}) using
the existing Flink plugin mechanism, consistent with how
{{{}FlinkResourceListener{}}}, {{{}FlinkResourceMutator{}}}, and
{{FlinkResourceValidator}} are already extensible today.
h1. Motivation
The Flink Autoscaler documentation already describes these interfaces as
extensible, and the Autoscaler Standalone module demonstrates that alternative
implementations can be wired in. However, for the built-in Kubernetes Operator
autoscaler, there is currently no supported way to provide custom
implementations of these components without forking the operator itself.
This is cumbersome and not straightforward for users who need to customize
autoscaler behavior like persisting autoscaler state to an external database,
routing scaling events to a custom monitoring system, or adjusting how scaling
decisions are applied to Flink resources. Today, such customizations require
maintaining a full fork of the operator, which creates a significant
maintenance burden and makes it difficult to stay up to date with upstream
releases.
The operator already has a well-established plugin mechanism for other
extension points (listeners, mutators, validators). Extending this same
mechanism to the autoscaler components would provide a consistent, low-friction
extensibility model and align the actual capabilities with what the
documentation already suggests is possible.
h1. Objective
Add plugin-based extensibility to the built-in Kubernetes Operator autoscaler
so that:
* Custom implementations of {{ScalingRealizer}}, {{AutoScalerStateStore}},
{{AutoScalerEventHandler}}, and {{KubernetesJobAutoScalerContext}} can be
provided by placing plugin JARs in the operator’s plugin directory, with no
forking required.
* The discovery follows the same plugin mechanism already used by other
operator extension points (listeners, mutators, validators), ensuring a
consistent extensibility model.
* When no custom plugin is found, the existing default Kubernetes
implementations are used, preserving full backward compatibility.
* Documentation is updated to guide users through creating and deploying
custom autoscaler plugins.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
