Doris-Breakwater commented on issue #70382: URL: https://github.com/apache/doris/issues/70382#issuecomment-6727840911
Breakwater-GitHub-Analysis-Slot: slot_12ca642457a9 The reported FE → BE stale-IP mechanism is supported by code inspection. A focused FE DNS/query-client fix is reasonable to review separately from the broader rolling-restart work. This is an availability issue when a stable BE hostname acquires a different IP; `UNAVAILABLE` alone does not establish that DNS caused a particular failure. I checked the local `4.1.4` tag (`ad35a140c7fd0b842f18c23300bac581f7d04326`) and fetched the relevant files from current upstream master (`41cd2b473e9d9bb26be089f302cdfdfe6e107eaf`). This was a read-only source investigation; I have not reproduced the Kubernetes failure or tested the proposed patch. The issue currently has no labels. Suggested routing: FE/RPC maintainers, with a 4.1 backport considered after validation. **Verified code path** - `DNSCache.get()` uses `computeIfAbsent`; a populated entry is not re-resolved on a normal read. Refresh runs every 60 seconds, with a **120-second initial delay**. Failed resolution retains a previously cached IP. See [DNSCache in 4.1.4](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/fe/fe-core/src/main/java/org/apache/doris/common/DNSCache.java#L37-L112); the fetched master file is identical. - `BackendServiceProxy.getProxy()` reads that IP before inspecting channel health. Its unhealthy-channel path replaces the client using the same local `realIp`; `removeProxy()` removes/shuts down the client without invalidating DNS. Once DNS actually changes, the IP comparison does replace the client. See [query-client acquisition](https://github.com/apache/doris/blob/41cd2b473e9d9bb26be089f302cdfdfe6e107eaf/fe/fe-core/src/main/java/org/apache/doris/rpc/BackendServiceProxy.java#L106-L169). - Fragment RPC failures commonly arrive asynchronously: both [Coordinator.waitPipelineRpc](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java#L1121-L1174) and [PipelineExecutionTask](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/fe/fe-core/src/main/java/org/apache/doris/qe/runtime/PipelineExecutionTask.java#L187-L240) remove a proxy on `Future.get()` failure without refreshing DNS. Adding handling only around the initial asynchronous RPC submission would miss these failures. - Successful recovery/start-time changes update backend liveness/epoch without query-client or DNS invalidation in the inspected [Backend heartbeat handler](https://github.com/apache/doris/blob/41cd2b473e9d9bb26be089f302cdfdfe6e107eaf/fe/fe-core/src/main/java/org/apache/doris/system/Backend.java#L937-L962). [#69450](https://github.com/apache/doris/pull/69450) is merged, but its heartbeat-pool changes do not add query-client invalidation or force DNS refresh. **A second cache layer needs explicit validation** Calling the resolver again is not necessarily a fresh DNS lookup: [NetUtils.getIpByHost](https://github.com/apache/doris/blob/41cd2b473e9d9bb26be089f302cdfdfe6e107eaf/fe/fe-core/src/main/java/org/apache/doris/common/util/NetUtils.java#L85-L98) calls `InetAddress.getByName()`, and [DorisFE startup](https://github.com/apache/doris/blob/41cd2b473e9d9bb26be089f302cdfdfe6e107eaf/fe/fe-core/src/main/java/org/apache/doris/DorisFE.java#L209-L210) sets `networkaddress.cache.ttl` to 60. Java maintains its own [positive and negative lookup cache](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/net/InetAddress.html#inetaddress.caching). Consequently, invalidating only Doris's map may still return the old JVM-cached IP. The actual contribution depends on the deployed JDK and effective cache policy. A mocked-resolver test alone cannot prove immediate recovery with default startup settings. Likewise, 60 seconds is a refresh interval, not a strict recovery bound: startup delay, resolver caching/failures, and queued resolution work can extend it. **Suggested scope and acceptance checks for the offered PR** 1. Force refresh while retaining the last known IP as a failure fallback. Serialize or fence publication per hostname, including periodic refresh. Test invalidation during an unpublished first lookup, invalidation arriving during refresh, and two warm readers while forced refresh is blocked. An older result must not overwrite a newer IP or consume a newer invalidation. The [first-lookup review](https://github.com/apache/doris/pull/69423#discussion_r4345214798) and [warm-reader review](https://github.com/apache/doris/pull/69423#discussion_r4345947506) on the still-open #69423 are useful regression cases. 2. On transport failure/non-normal state, invalidate before reacquiring the address; do not reuse the already-read stale `realIp`. Cover asynchronous failure completion and ensure a late failure from an old client cannot evict its replacement. Distinguish transport failures from query/application errors. 3. On a successful restart/recovery heartbeat, handle all local proxy instances (`Holder` contains several), including query-serving followers receiving heartbeat replay. Historical checkpoint replay must not shut down serving connections. Test unchanged-IP restarts as well as changed-IP restarts. 4. Run a real-resolver, changing-IP integration test with normal FE startup/JDK settings and the periodic refresh held off. Verify both the connection destination and time to first successful subsequent query. Keep transparent RPC retries, BE exchange caches, and Stream Load behavior outside this narrow fix; cache recovery alone does not guarantee that an already-failed query succeeds. **Evidence still needed to confirm the reported incident and validate the fix** - Exact FE/BE build hashes, JDK version, effective DNS cache policy, and whether #69450 is present in the deployed build. - One timestamped restart timeline: BE hostname/ID, old/new pod IP, DNS answers observed from the affected FE pod, successful restart heartbeat/start-time update, failing query IDs with full nested exception and destination IP/port, and the DNS-change/first-success log entries. Include the serving FE identity/role: an alive indication on one FE does not demonstrate DNS freshness on another. - A minimal query/table setup, repetition rate, FE count, and whether the reproduced failure is FE fragment dispatch or BE-to-BE exchange. A BE exchange failure needs separate sender-side investigation. Please open the proposed focused PR with these tests and a measured before/after timeline. The source evidence supports the cache invalidation gap; the incident timeline and real-resolver test are needed to establish the full cause and the recovery guarantee. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
