The GitHub Actions job "Nightly (beta)" on flink.git/release-1.20 has failed.
Run started by GitHub user github-actions[bot] (triggered by 
github-actions[bot]).

Head commit for run:
a578073b8c1c5d48acc3f2985e6b28fe57fa5fe0 / Martijn Visser 
<[email protected]>
[FLINK-40475][runtime] Fix watermark loss and stall in StatusWatermarkValve 
when subpartitions realign after idleness (#29024)

[FLINK-40475][runtime] Fix watermark loss and stall in StatusWatermarkValve 
when subpartitions realign after idleness

The emission logic of StatusWatermarkValve relies on two invariants over
the set of watermark-aligned subpartitions: it is only empty when no
subpartition is active, and whenever it is non-empty its min watermark
equals the last output watermark. Both were violated once an unaligned
subpartition - one that went idle, resumed, and only partially caught up
to the last output watermark - was involved:

- The FLINK-7728 all-idle flush was skipped unless the last subpartition
  to become idle held the current min watermark. That assumption doesn't
  hold when unaligned subpartitions are involved, since an unaligned
  subpartition's watermark is invisible to the min-derive path. This made
  the final watermark depend on the order in which inputs became idle -
  exactly the defect FLINK-7728 was meant to fix.
- The idle->active branch re-added a caught-up subpartition to the aligned
  set without re-deriving the min. When no other aligned subpartition
  remained, the realigned subpartition's watermark stalled indefinitely,
  emitted only once an even larger watermark arrived on it.

Fix both by re-deriving from the aligned set unconditionally instead of
relying on the broken shortcut:
- Flush the max watermark unconditionally once all subpartitions go idle.
  findAndOutputMaxWatermarkAcrossAllSubpartitions only emits when the max
  advances past the last output watermark, so this cannot regress
  monotonicity, and it keeps the valve consistent with
  CombinedWatermarkStatus, which has flushed unconditionally since
  FLINK-38454.
- Re-derive (and possibly emit) the min watermark after a subpartition
  realigns, after the ACTIVE status is emitted so downstream inputs don't
  drop the watermark while they still consider the input idle.

Documents the invariants on alignedSubpartitionStatuses.

Generated-by: Claude Code (claude-fable-5)
(cherry picked from commit bb773b5fb8a4a87dd4efc3a8ae261b7e08e602a6)
(cherry picked from commit 022b7dce9385b5b40bf0e5169f2171c767232880)

Report URL: https://github.com/apache/flink/actions/runs/42739759214

With regards,
GitHub Actions via GitBox

Reply via email to