The GitHub Actions job "Flink CDC CI" on 
flink-cdc.git/FLINK-39775-release-snapshot-split-metadata has failed.
Run started by GitHub user spoorthibasu (triggered by lvyanquan).

Head commit for run:
78e0b7f72058fcb1bddebf33ad8234a950d4df7d / Spoorthi Basu 
<[email protected]>
[FLINK-39775][mysql] Release snapshot split metadata after entering the stream 
phase

After the snapshot phase finishes and the binlog split is assigned, the source
coordinator kept the full snapshot split metadata (assigned splits, finished
offsets, table schemas) around forever. snapshotState() rebuilt the whole
SnapshotPendingSplitsState on every checkpoint, notifyCheckpointComplete() never
dropped it, and the enumerator held a second copy in its binlog meta cache. For 
a
table with many finished splits (around 300K for a 2.5B-row table at the default
chunk size) this kept JobManager memory high and pushed up direct memory during
checkpoint serialization, eventually OOM-killing the container.

The reader now tells the coordinator (BinlogSplitMetaAssembledEvent) once it
holds the complete binlog split. The coordinator schedules the release in
snapshotState() and performs it in notifyCheckpointComplete() after the covering
checkpoint completes, so the binlog split assignment is always 
checkpoint-covered
and can never be added back into an emptied assigner. Release is skipped when
scan.newly-added-table is enabled, and the state/serializer format is unchanged
(a released state is just empty maps).

The assembled event has to be tied to a specific assignment. Flink does not drop
source events from a failed attempt, and the event carried no identity, so after
a reader failover a leftover event from the dead attempt could reach the
coordinator and re-arm the release while the new reader was still pulling meta
groups, which spans several checkpoints for a large table. The release would 
then
fire and the new reader's next meta request would come back empty. Each
assignment now has a generation that bumps on every add-back; the coordinator
stamps it on the meta groups it serves and the reader echoes it back, so release
only happens when the two match and a stale event is ignored. A split small
enough to ship inline requests no meta groups, so its reader reports a sentinel
the coordinator always accepts, since an inline reader never needs the metadata.
The release state lives only in the singleton coordinator and there is a single
binlog split, so parallel readers need no handling beyond this failover case.

Tests cover the release protocol (assembled event plus a completed checkpoint, 
no
release without the event or before the checkpoint completes, no release with
newly-added-table on, add-back resetting the schedule, restore from a released
light state), the stale-event case where an old-generation event after a 
failover
must not trigger release, the inline-split case where release still happens 
after
a generation bump, a reader-side check that the assembled event carries the 
served
generation or the inline sentinel, a serializer round-trip of the released light
state, and the assembled event being a no-op for a binlog-only assigner.

Report URL: https://github.com/apache/flink-cdc/actions/runs/26727972391

With regards,
GitHub Actions via GitBox

Reply via email to