bosswnx opened a new issue, #67297: URL: https://github.com/apache/doris/issues/67297
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues) and found no similar issues. ### Version Doris 3.0.8 (also verified against master branch code) ### What's Wrong? After a master FE failover (old master lost leadership due to OOM/GC pause but the process stayed alive and kept serving its MySQL/thrift ports), a non-master FE (observer/follower) with a lagging journal replay keeps forwarding statements to the **old master**, and clients continuously receive: ``` ERROR 1105 (HY000): errCode = 2, detailMessage = The statement has been forwarded to master FE(<old master ip>) and failed to execute because Master FE is not ready. You may need to check FE's status ``` The error persists until the observer's journal replay catches up (or the observer is restarted), because: 1. The forward target is `Env.getMasterHost()/getMasterRpcPort()` (`MasterOpExecutor` → `FEOpExecutor`), which is backed by the in-memory `masterInfo` field. 2. `masterInfo` is **only** updated by replaying the `OP_MASTER_INFO_CHANGE` journal entry written by the new master (`EditLog.java` → `Env.setMaster()`), or by loading an image at startup. There is no active master re-discovery on the forward path. 3. Meanwhile the old master process is still listening on its ports, so the forwarded thrift call **succeeds at the transport level** — it reaches `StmtExecutor` on the old master, which (as a proxy-forwarded statement, `isProxy=true`) detects it cannot find a valid master and throws the error above (`StmtExecutor.executeByNereids/executeByLegacy`, the `isProxy` branch) instead of the sender learning "this node is not master, retry elsewhere". 4. LB health checks (TCP-level) keep routing client connections to the degraded old master as well, compounding the issue. Additionally, for `FORWARD_WITH_SYNC` statements (e.g. `CREATE USER`), the old master's error response still carries a `maxJournalId`, so the observer blocks in `JournalObservable.waitOn()` (up to `query_timeout * 1.2`, default ~18 min) before surfacing anything — the client just hangs. ### How to Reproduce? Reproduced deterministically on Doris 3.0.8 with 4 FEs in docker (fe1=master, fe2/fe3=followers, fe4=observer). `nsenter`-based iptables is used to simulate the degraded states (equivalent to what an OOM/GC-pause produces): 1. **Freeze the observer's journal replay**: block fe4 → fe2/fe3 on the bdbje port (9010), so fe4 cannot replay the new master's journals (its `masterInfo` stays = fe1). 2. **Kill fe1** (`docker kill`, simulating the OOM death of the old master). fe2 is elected as the new master. 3. **Restart fe1 with its bdbje isolated** (block 9010 toward fe2/fe3/fe4). fe1 comes back "alive but degraded": MySQL (9030) and thrift (9020) ports still serve, but it is no longer the master and cannot rejoin/catch up — exactly the state of an OOM-recovered old master that lost the election. 4. **Execute a forward-to-master statement via the observer** (fe4): ``` $ mysql -h <fe4> -P 9030 -u root -e 'ADMIN SET FRONTEND CONFIG ("label_keep_max_second" = "259200")' ERROR 1105 (HY000) at line 1: errCode = 2, detailMessage = The statement has been forwarded to master FE(172.20.80.2) and failed to execute because Master FE is not ready. You may need to check FE's status ``` (The IP in the message is the **old master's own IP** — the error is thrown by the old master itself.) Evidence from the logs: - Observer (fe4) keeps forwarding to the old (dead/degraded) master after the switch: ``` [fe4 log] MasterOpExecutor.forward: forward to master FE TNetworkAddress(hostname:172.20.80.2, port:9020) ``` - Old master (fe1) throws the error on the proxy-forwarded statement: ``` [fe1 log] StmtExecutor.executeByLegacy:1042 execute Exception. org.apache.doris.common.UserException: errCode = 2, detailMessage = The statement has been forwarded to master FE(172.20.80.2) and failed to execute because Master FE is not ready. You may need to check FE's status ``` Notes: - With a `FORWARD_WITH_SYNC` statement (`CREATE USER`), the client instead hangs in journal-sync wait on the observer for up to ~18 minutes before failing. - The recovery observed in production (rolling restart of the observer) matches this analysis: a restart forces an image load + full journal replay, refreshing `masterInfo` to the new master. - `meta_delay_toleration_second` (default 300s) eventually marks the observer not-ready, which changes the failure mode but does not fix the stale-forward window (5 minutes of broken/hanging statements even in the best case). ### Anything Else? Related prior work: - #25371 added `master_address` to thrift responses for BE→FE RPCs so the BE can skip an old master, but the FE→FE `forward()` path never got the equivalent treatment. - #62721 fixed a result-packet loss during master switch on this same forward path, confirming this area is fragile during master transitions. - BE-side protection against stale masters exists (e.g. commit f13bd7ad7de "follower fe invalidate dns and be rpc skip old master" probes other FEs after a NOT_MASTER/transport failure), while the FE forward path still relies solely on journal-replay-updated `masterInfo`. Suggested fix directions (for discussion): 1. In `FrontendServiceImpl.forward()`: when the receiving FE is not the master, return a NOT_MASTER-style response immediately (with the real master address as a hint, similar to #25371) instead of executing the statement deep into `StmtExecutor` and throwing "Master FE is not ready". 2. In `FEOpExecutor.forward()`: on receiving NOT_MASTER (or on forward failure), re-discover the current master (e.g. via `Env.getHaProtocol().getLeader()` which asks bdbje directly and does not depend on journal replay) and retry once against the real master. This combination would close the stale-forward window regardless of journal replay lag, and also prevents the journal-wait hang caused by the old master's stale `maxJournalId` in error responses. Full reproduction scripts and timelines available if needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
