morningman commented on PR #68302:
URL: https://github.com/apache/doris/pull/68302#issuecomment-5757248161
**Context guide for reviewers: what this PR fixes, in plain terms**
**TL;DR**: this fixes a race in the FE role state machine. When a
FOLLOWER/OBSERVER transition is interrupted, the old code still marked the FE
as "already FOLLOWER", so the real initialization never got a second chance and
the FE stayed half-initialized until a restart.
**Background: how an FE changes its role**
- The role is decided by the BDBJE replication group. On every BDBJE state
change, `BDBStateChangeListener` pushes a new role (FOLLOWER / OBSERVER /
UNKNOWN / MASTER) into `typeTransferQueue`.
- The `stateListener` daemon takes roles from the queue, runs the transition
(`transferToMaster()` / `transferToNonMaster()`), and then sets `feType =
newType`. `feType` is what the FE believes it currently is.
- If the incoming role equals the current `feType`, the listener treats it
as a duplicate notification and ignores it.
**What `transferToNonMaster()` does** (INIT/UNKNOWN -> FOLLOWER/OBSERVER)
1. Start the replayer thread that replays edit logs from master.
2. Block in `postProcessAfterMetadataReplayed(true)` until metadata has
caught up, i.e. until `isReady` becomes true (it is set by the replayer thread).
3. Only then run the actual non-master initialization:
`checkLowerCaseTableNames()`, `startNonMasterDaemonThreads()` (load manager,
label cleaner, tablet stat mgr, DNS cache, admission control, ...),
`MetricRepo.init()`, statistics cache pre-heat, `FollowerColumnSender`.
Step 2 has an escape hatch: if another role notification is already waiting
in the queue, it returns `false` immediately ("the state changed again, don't
wait here forever, go handle the newer state"). That design is fine by itself.
**The bug**
When the escape hatch fired, `transferToNonMaster()` just returned and none
of step 3 ran. But the caller (`runOneCycle`) had no way to know that and
unconditionally did `feType = newType`. The FE now claims to be a FOLLOWER
while none of the FOLLOWER initialization has happened.
Next the listener processes the queued notification that caused the
interruption. If it is another FOLLOWER (BDBJE can deliver the same state more
than once), the listener sees `feType == newType` and drops it as a duplicate.
From then on every further FOLLOWER notification is ignored as well, so step 3
never runs.
**Why it matters**
`isReady` / `canRead` (the "can I serve" flags) are controlled independently
by the replayer thread: as soon as metadata is caught up they flip to true,
regardless of whether step 3 ran. So the FE accepts queries and looks healthy
from the outside, but its metrics are empty, the non-master daemon threads are
not running, and the statistics cache is never pre-heated. This state is hard
to notice and does not heal itself.
**The fix**
- `transferToNonMaster()` now returns a boolean: `true` only when step 3
completed, `false` when interrupted.
- The listener commits `feType` only on `true`. On `false` it keeps the
previous `feType` (e.g. INIT) and continues with the next queued notification.
- When the next FOLLOWER notification arrives, it is a legitimate INIT ->
FOLLOWER transition and the whole initialization runs again. In other words,
`feType` now means "the last state that was fully initialized", not "the latest
state reported by BDB".
**The test**
`EnvStateListenerTest` mocks the wait in step 2 to always return `false`
(simulating an interruption). It sends FOLLOWER once and checks that the
transition was attempted, then sends FOLLOWER again and asserts the transition
is attempted a **second** time instead of being dropped as a duplicate, with
`feType` still INIT. Before this fix the second notification would have been
ignored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
