The GitHub Actions job "Flink Kubernetes Operator CI" on 
flink-kubernetes-operator.git/FLINK-40955 has succeeded.
Run started by GitHub user gyfora (triggered by gyfora).

Head commit for run:
4ee70a3a03f18a35bf17295940b4e11379843f60 / Gyula Fora <[email protected]>
[FLINK-40955] Give up and remove finalizer after cleanup keeps failing

If cleanup keeps failing (e.g. the cluster resources can never be deleted
because a pod is stuck Terminating), the failure was retried by JOSDK
forever: the finalizer was never removed, the CR stayed pinned in
Terminating permanently, and every future redeploy under the same name
failed with a 409.

Cleanup failures are now propagated so that JOSDK retries them with its
regular backoff. On a retry, once the resource has been pending deletion
for longer than kubernetes.operator.resource.cleanup.finalizer.timeout
(default 120m, measured from metadata.deletionTimestamp), the operator
gives up: it emits a Warning/CleanupFailed event and removes the finalizer.
The first failed attempt never gives up and a timeout of 0 disables giving
up altogether. If the JOSDK retries are exhausted before the timeout the
cleanup is rescheduled so that it keeps being retried. This accepts that
resources may be orphaned in exchange for keeping the resource name
redeployable. The status changes of a failed attempt are patched before the
error is rethrown so that they are not lost between retries.

Related changes:
 - Resources that were never deployed are cleaned up right away in the
   controllers without observing or reconciling, there is nothing to
   clean up.
 - A job cancellation that does not complete within
   resource.cleanup.timeout is considered stuck: session job deletion
   stops waiting for it (emitting a warning event with the job id, as the
   job may still be running) and applications fall back to an HA metadata
   based last-state upgrade when that is allowed.

Report URL: 
https://github.com/apache/flink-kubernetes-operator/actions/runs/41888665334

With regards,
GitHub Actions via GitBox

Reply via email to