bosswnx commented on issue #67297:
URL: https://github.com/apache/doris/issues/67297#issuecomment-5452616028
As requested in the review, here are the full reproduction scripts and the
timestamped evidence timeline.
## Build version
- Image: official `apache/doris:fe-3.0.8` from Docker Hub (manifest digest
below). No internal patches.
```
$ docker inspect apache/doris:fe-3.0.8 --format '{{index .RepoDigests 0}}'
apache/doris@sha256:81c72270254585376d8e46239ed90b4f5c04b0d77ae0a0accafb4ea90f7d54c8
```
- Version reported by the FE: `doris-0.0.0-c7e635238f4` (from `show
frontends`). Note this is the version string baked into the 3.0.8 release
image; the same code path was also confirmed on current master at the source
level in the issue body.
## Reproduction scripts
### docker-compose.yml (4 FE, no BE — only forward-to-master statements are
used)
```yaml
version: '3'
services:
fe1: # initial MASTER (simulates the customer's old master 10.206.172.8)
image: "apache/doris:fe-3.0.8"
container_name: "repro-fe1"
hostname: "fe1"
cap_add: [NET_ADMIN]
environment:
-
FE_SERVERS=fe1:172.20.80.2:9010,fe2:172.20.80.3:9010,fe3:172.20.80.4:9010,fe4:172.20.80.5:9010
- FE_ID=1
ports: ["18030:8030", "19030:9030"]
volumes:
- /data/repro-10907/docker/fe1/meta:/opt/apache-doris/fe/doris-meta
- /data/repro-10907/docker/fe1/log:/opt/apache-doris/fe/log
networks:
doris_net:
ipv4_address: 172.20.80.2
# fe2 = 172.20.80.3 FOLLOWER, fe3 = 172.20.80.4 FOLLOWER, fe4 =
172.20.80.5 OBSERVER
# (identical blocks with FE_ID=2/3/4; fe4 is registered via
# ALTER SYSTEM DROP FOLLOWER "172.20.80.5:9010"; ALTER SYSTEM ADD
OBSERVER "172.20.80.5:9010"
# because the image entrypoint only registers followers)
networks:
doris_net:
ipam:
config:
- subnet: 172.20.80.0/24
```
### Injection script (repro_demo.sh)
Uses `nsenter -t <container-pid> -n iptables` from the host — the image
itself has no iptables binary.
```bash
MYSQL4="mysql -h 127.0.0.1 -P 19033 -u root" # observer fe4 (customer's LB
-> 10.206.172.49)
pid() { docker inspect repro-$1 --format '{{.State.Pid}}'; }
# STEP 1: freeze observer's journal replay toward the potential new masters
(bdbje 9010)
FE4PID=$(pid fe4)
for ip in 172.20.80.3 172.20.80.4; do
nsenter -t $FE4PID -n iptables -A INPUT -p tcp -s $ip --dport 9010 -j
DROP
nsenter -t $FE4PID -n iptables -A OUTPUT -p tcp -d $ip --dport 9010 -j
DROP
done
# STEP 2: old master "OOM" death
docker kill repro-fe1
# STEP 3: wait for the new master election (fe3 won in this run)
# STEP 4: restart fe1, wait for its thrift (9020) to listen, then isolate
its bdbje:
# alive-but-degraded old master (mysql/thrift serving, cannot rejoin)
docker start repro-fe1
# ... wait for /dev/tcp/172.20.80.2/9020 ...
FE1PID=$(pid fe1)
for ip in 172.20.80.3 172.20.80.4 172.20.80.5; do
nsenter -t $FE1PID -n iptables -A INPUT -p tcp -s $ip --dport 9010 -j
DROP
nsenter -t $FE1PID -n iptables -A OUTPUT -p tcp -d $ip --dport 9010 -j
DROP
done
# STEP 5: customer scenario — statement via the observer
mysql -h 127.0.0.1 -P 19033 -u root -e \
'ADMIN SET FRONTEND CONFIG ("label_keep_max_second" = "259200")'
```
## Client-visible result (FORWARD_NO_SYNC statement)
```
$ mysql -h 127.0.0.1 -P 19033 -u root -e 'ADMIN SET FRONTEND CONFIG
("label_keep_max_second" = "259200")'
ERROR 1105 (HY000) at line 1: errCode = 2, detailMessage = The statement has
been forwarded to master FE(172.20.80.2) and failed to execute because Master
FE is not ready. You may need to check FE's status
```
fe4 audit log confirms this exact error was returned to the client:
```
AuditLogger 2026-08-28 11:43:18,657 [query] | ... |Timestamp=2026-08-28
11:43:18.657|Client=172.20.80.1:...|FeIp=172.20.80.5|State=ERR|ErrorCode=1105|ErrorMessage=errCode
= 2, detailMessage = The statement has been forwarded to master
FE(172.20.80.2) and failed to execute because Master FE is not ready. You may
need to check FE's status|...|Stmt=ADMIN SET FRONTEND CONFIG
("label_keep_max_second" = "259200")
```
## Timestamped evidence timeline (all times are container-log times, same
host clock)
| Time | Node | Event (log line) |
|---|---|---|
| 11:37:57 | fe4 (observer) | `setMaster MasterInfo: host=172.20.80.2`
(masterInfo = fe1, normal) |
| 11:38:33 | fe4 | `replayed journal id is 48, replay to journal id is 49`
(replay healthy before injection) |
| 11:38:37 | fe1 | `docker kill` — old master dies |
| 11:38:37 | fe4 | `transfer FE type from OBSERVER to UNKNOWN` (lost its
feeder) |
| 11:39:00–11:39:10 | fe3 | `transfer FE type from UNKNOWN to MASTER` —
**fe3 (172.20.80.4) is the new master** |
| 11:41:41 | fe1 | restarted; `transfer FE type from INIT to FOLLOWER`
(thrift 9020 up; bdbje then isolated by iptables) |
| 11:41:42 | fe1 | `setMaster MasterInfo: host=172.20.80.4` — **briefly
learned the new master from residual replay, then isolated** |
| 11:42:12 | fe4 | `forward to master FE
TNetworkAddress(hostname:172.20.80.2, port:9020), statement id: 12` —
**observer still forwards to the old master fe1** (its masterInfo was never
refreshed: replay frozen since 11:38:37) |
| 11:42:13.114 | fe1 | `StmtExecutor.executeByLegacy():1042 execute
Exception` → `UserException: ... The statement has been forwarded to master
FE(172.20.80.2) and failed to execute because Master FE is not ready` (thrown
by the old master itself, stack: `proxyExecute` →
`FrontendServiceImpl.forward`) |
| 11:42:13.117 | fe4 | `forwarding to master get result max journal id: 79`
→ `JournalObservable.waitOn(): waiting for the observer to replay journal to 79
with timeout: 1080000 ms` — **the sync-wait hang begins even though the result
carried an error** |
| 11:43:18 | fe4 | (with the 25 s client timeout override in this test run)
error surfaces to the client; with the default `query_timeout=900`, this wait
would last up to 1080 s |
Notes on the numbers requested in the review:
- Observer `replayedJournalId` was frozen at ~49 since 11:38:37; the old
master returned `maxJournalId=79` (its own last replayed id), so `waitOn(79)`
can never be satisfied while the replay freeze lasts — this is the exact
WITH_SYNC hang mechanism.
- `meta out of date ... delta:300001ms, hasLog:false, feType:UNKNOWN`
appears at ~300 s after the freeze (default
`meta_delay_toleration_second=300`), after which the observer's failure mode
changes to `Node catalog is not ready` — consistent with the review's note that
the stale-forward window is bounded by but not fixed by this config.
- In the 11:42 attempt above, fe1's own `masterInfo` had already flipped to
fe3 (via its brief residual replay at 11:41:42 before isolation). The
customer's real-world case is harsher: an OOM-recovered old master that never
replays the new master's journals keeps `masterInfo = itself`, which is why the
hint-from-rejecting-FE approach must not be trusted blindly (matches review
constraint #2).
## FORWARD_WITH_SYNC variant (CREATE USER)
Same injection; `CREATE USER fin_1 IDENTIFIED BY 'p'` sent via fe4 while fe1
is dead-but-forwardable produced either:
- `ForwardToMasterException ... cause: EOF, Socket is closed by peer` (fe1
down), or
- when fe1 is in the degraded-alive state: the statement is rejected by fe1
with the same "Master FE is not ready" error, but fe4 blocks in
`JournalObservable.waitOn(79)` for the full timeout (only interrupted by
client-side `SET query_timeout` — with `query_timeout=5` the wait log shows
`timeout: 6000 ms`).
This confirms the review's constraint #5: a structured NOT_MASTER response
alone would not fix the sync-forward hang; the rejection path must bypass
`JournalObservable.waitOn()`.
## What I don't have
- The customer's production LB health-check config and BDB JE state-change
logs from the real failover window (their environment; I'll request them). The
reproduction above is my own environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
