The GitHub Actions job "Flink Kubernetes Operator CI" on 
flink-kubernetes-operator.git/flip-504 has failed.
Run started by GitHub user schongloo (triggered by schongloo).

Head commit for run:
7c15db1d2f2eacc23fae7f5b7a50411737f34da2 / schongloo <[email protected]>
[FLIP-504] Abort an ADVANCED transition whose gate never clears

Once the new deployment is ready, an ADVANCED transition waits for the gate to
report CLEAR_TO_TEARDOWN before tearing the old deployment down. Nothing bounded
that wait: the abort deadline was only consulted while the new deployment was
not ready, so a gate that never cleared (for example a stalled watermark) left
the transition in TRANSITIONING indefinitely with no error.

- New kubernetes.operator.bluegreen.gate.timeout option. When the stage moves to
  TRANSITIONING the abort timestamp is re-armed to now + gate timeout and
  persisted. It defaults to the abort grace period.
- While waiting for CLEAR_TO_TEARDOWN, a passed deadline aborts through the
  existing abort path, with an error naming the gate and the option to tune.

Aborting also has to hand the gate back to the deployment that keeps running,
and the abort path did not do that. It only rolled active-deployment-type back,
so the surviving gate came back ACTIVE with no watermark toggle, a state in 
which
it drops every record while it waits for a toggle that only the now-suspended
deployment would write. This affected every ADVANCED abort, including the
existing "not ready within the abort grace period" one.

- resetGateOnAbort replaces rollbackActiveDeploymentType. It rewrites the
  ConfigMap whole as a first deployment, so the aborted hand-over's strategy
  entries are dropped and the remaining gate passes every record, as it does
  before any transition.
- It runs after markDeploymentFailing's stage write. That write goes through the
  informer's cached copy under optimistic locking, while the reset replaces the
  ConfigMap unconditionally, so the reset is the state that lands.

Tests:
- verifyAdvancedModeGateTimeoutAborts: re-arm on entering TRANSITIONING, no 
abort
  before the deadline, abort and gate reset after it.
- verifyAdvancedModeAbortResetsGate: the not-ready abort now also resets the 
gate.
- BlueGreenUtilsTest: the gate timeout falls back to the abort grace period.
- buildAdvancedSessionCluster pins a 60 s gate timeout, since the builder's 1 ms
  abort grace period would otherwise be inherited.

Each new test fails when its fix is reverted (no reset, no deadline check, no
re-arm).

Report URL: 
https://github.com/apache/flink-kubernetes-operator/actions/runs/36655353570

With regards,
GitHub Actions via GitBox

Reply via email to