Aias00 opened a new issue, #7021: URL: https://github.com/apache/shenyu/issues/7021
## Description `AiProxyExecutorService.executeDirectStream` applies retry to the main provider Flux and then unconditionally switches to the fallback provider through `onErrorResume`. If the main provider emits one or more chunks before failing, those chunks have already been written to the client SSE response. The fallback provider then starts a new completion from the beginning and appends its chunks to the already committed response. This issue is distinct from #6647: - #6647 tracks replaying the main upstream after a mid-stream retry. - This issue tracks switching to a different fallback provider after partial output, producing one SSE response containing content from multiple completions/providers. Related PR: #6341, reviewed at head `8fa93d7446e052f149d3740bbe7ff2052c375e23`. ## Affected code - `shenyu-plugin-ai-proxy/.../AiProxyExecutorService.java`: `retryWhen(...).onErrorResume(...fallback...)` wraps the whole streaming Flux. - `shenyu-plugin-ai-proxy/.../AiProxyPlugin.java`: the Flux is written directly to the committed SSE response. ## Reproduction An equivalent Reactor sequence: ```text main provider: emit "main-partial" -> error retry: emit "main-partial" -> error fallback: emit "fallback-completion" ``` produces: ```text main-partial, main-partial, fallback-completion ``` Even after #6647 removes the retry replay, an unconditional fallback after the first emitted chunk would still produce: ```text main-partial, fallback-completion ``` ## Impact - Clients receive a syntactically valid SSE stream with semantically corrupted, mixed completion content. - Tool calls or structured output can be duplicated or combined across providers. - Multiple providers may bill for the same client request. - The response is already committed, so the gateway cannot safely replace earlier chunks. ## Expected behavior Provider retry and fallback are allowed only before the first response chunk is emitted. After any chunk is sent, an upstream failure must terminate the current stream without starting another provider completion. ## Acceptance criteria - [ ] Track whether the main stream has emitted a chunk. - [ ] Allow retry/fallback only before the first emission. - [ ] After partial emission, propagate/terminate the stream without invoking fallback. - [ ] Add a regression test where the main provider emits a chunk and then errors while fallback is configured. - [ ] Assert no duplicated prefix and no fallback chunk is appended after partial output. - [ ] Retain the existing error-before-first-chunk fallback behavior. - [ ] Coordinate the retry portion with #6647 rather than implementing two competing fixes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
