Doris-Breakwater commented on issue #67297: URL: https://github.com/apache/doris/issues/67297#issuecomment-5452329817
## Preliminary assessment This is a high-confidence FE high-availability bug, primarily affecting availability rather than showing evidence of metadata corruption. The issue is currently open, unassigned, and has no labels. The supplied logs demonstrate the stale forwarding target and the receiver-side proxy error; I did not independently run the four-FE reproduction. I inspected the nearest published 3.0.8 upstream tag, `3.0.8-rc02` (`96abadcc8fd`), because upstream does not currently publish an exact `3.0.8` tag, and current `master` at `3ba29643fa0`. The relevant mechanism is present in both. ## Code-confirmed facts - [`MasterOpExecutor`](https://github.com/apache/doris/blob/3ba29643fa0aa1ca5c86ae831d64ee888b0b212f/fe/fe-core/src/main/java/org/apache/doris/qe/MasterOpExecutor.java#L43-L50) constructs its thrift target from `Env.getMasterHost()/getMasterRpcPort()`. [`FEOpExecutor.forward()`](https://github.com/apache/doris/blob/3ba29643fa0aa1ca5c86ae831d64ee888b0b212f/fe/fe-core/src/main/java/org/apache/doris/qe/FEOpExecutor.java#L114-L163) retries only the same address after a transport exception; it performs no role check or master rediscovery. - For an already-running non-master FE, the applicable `masterInfo` refresh path is replay of `OP_MASTER_INFO_CHANGE` ([`EditLog`](https://github.com/apache/doris/blob/3ba29643fa0aa1ca5c86ae831d64ee888b0b212f/fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java#L695-L699) -> [`Env.setMaster()`](https://github.com/apache/doris/blob/3ba29643fa0aa1ca5c86ae831d64ee888b0b212f/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java#L5473-L5476)); startup image loading is the other relevant refresh path. There is no active lookup on the SQL forwarding path. - [`FrontendServiceImpl.forward()`](https://github.com/apache/doris/blob/3ba29643fa0aa1ca5c86ae831d64ee888b0b212f/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java#L1152-L1169) has no master-role gate before proxy execution. On a non-master receiver, the `isProxy` guard in [`StmtExecutor`](https://github.com/apache/doris/blob/3ba29643fa0aa1ca5c86ae831d64ee888b0b212f/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java#L844-L850) produces the reported error instead of a structured redirect. - [`ConnectProcessor.proxyExecute()`](https://github.com/apache/doris/blob/3ba29643fa0aa1ca5c86ae831d64ee888b0b212f/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java#L798-L824) sets the receiver's `maxJournalId` for both successful and failed execution. For sync forwarding, [`MasterOpExecutor`](https://github.com/apache/doris/blob/3ba29643fa0aa1ca5c86ae831d64ee888b0b212f/fe/fe-core/src/main/java/org/apache/doris/qe/MasterOpExecutor.java#L60-L75) then waits on that ID without first checking the returned error status. With the default `query_timeout=900s` and the 1.2 coefficient, the maximum wait is 1080 seconds (18 minutes). This confirms the reported hang mechanism when the observer's replayed ID is below the old FE's returned ID. - `TMasterOpResult` currently has neither a structured `NOT_MASTER` status nor a master-address hint, so the sender cannot distinguish this condition from an ordinary SQL error. The code evidence supports a failover availability defect. It does not indicate that the old FE executes this forwarded DDL: the proxy guard rejects it before the command runs. ## Important constraints on the proposed fix 1. `BDBHA.getLeader()` is independent of journal replay, but it still calls BDB JE `ReplicationGroupAdmin` and returns the leader's **edit-log/BDB address**. Under the reproduction's deliberate observer-to-quorum port-9010 isolation, it may be unable to discover a leader at all. It also does not directly return the FE RPC endpoint. This can be one discovery source, but it cannot by itself guarantee recovery for the stated partition. 2. A hint derived from the rejecting FE's cached `masterInfo` is not necessarily authoritative in this failure mode; the degraded old master may still point to itself. The sender must reject same-address/looping hints and have a bounded fallback, such as probing known FE RPC endpoints for a node that explicitly confirms it is master. 3. Rediscovery/retry should be scoped to `MasterOpExecutor` or guarded by an explicit `requireMaster` request flag. Generic `FEOpExecutor` is also used intentionally against specific non-master FEs, including all-FE configuration propagation and cross-FE query kill; globally redirecting it would change those semantics. 4. A mutating/non-idempotent statement may be retried safely after an explicit receiver-side, pre-execution `NOT_MASTER` response. It must not be retried after an ambiguous timeout/transport failure, where execution may already have occurred. Existing `shouldNotRetry` behavior for non-query operations should be preserved for that case. 5. A structured `NOT_MASTER` result must bypass `JournalObservable.waitOn()`; otherwise the protocol addition alone will not fix the sync-forward hang. ## Information still useful from the reporter - The complete FE build version/commit hash for the binary called `3.0.8`, plus the full reproduction scripts, so the scenario can become a repeatable integration test. - For the `CREATE USER` hang, timestamped lines from observer, old master, and new master showing role transitions, `masterInfo`, observer `replayedJournalId`, returned/target `maxJournalId`, `isReady/canRead`, and the session `query_timeout`. The source proves the wait is possible, but those values would verify the exact observed 18-minute timeline. - The production failover window's BDB JE state-change/replay logs and LB health-check configuration. Please redact credentials and unrelated environment details. ## Suggested next steps 1. Triage this as an FE/HA availability bug and reproduce it on the reporter's exact build and current master. 2. Define a backward-compatible master-directed forwarding signal and structured `NOT_MASTER` response, with an optional hint. Reject master-only requests before `proxyExecute()` and return no journal-sync target for that rejection. 3. Add bounded redirect/discovery in the master-directed caller only. Prefer a verified hint or BDB leader mapping when available, with known-FE probing as the partition-tolerant fallback; retry mutations only after the explicit pre-execution rejection. 4. Add tests covering an alive old thrift endpoint with stale `masterInfo`, both `FORWARD_NO_SYNC` and `FORWARD_WITH_SYNC`, unavailable BDB discovery, no journal wait on rejection, redirect-loop prevention, and no duplicate mutating execution. 5. Until a fix is available, remove degraded/lagging FEs from the client LB using FE role/readiness/journal progress rather than TCP-port health alone; restarting the stale observer is a recovery action, not a complete fix. Breakwater-GitHub-Analysis-Slot: slot_ee333f2eee46 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
