matanper opened a new issue, #68973: URL: https://github.com/apache/doris/issues/68973
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.4 (cloud mode, arm64). On 4.1.4, `StreamLoadAction::_handle` also blocks in `ctx->future.get()` and commit. On master, #53144 already moved that into a callback. The other two blocking points are unchanged on master: - `be/src/service/http/action/stream_load.cpp:244-296` → `_on_header` `:386-402`: `begin_txn` (FE `loadTxnBegin`, `:394`) and, in `_process_put`, the FE `streamLoadPut` RPC (`:826-831`) plus the fragment start, all run synchronously from libevent's header callback (`ev_http_server.cpp:72-79`). - `be/src/service/http/action/stream_load.cpp:405-445`: `on_chunk_data` calls `body_sink->append`, which blocks in `StreamLoadPipe::_append` on `_put_cond` while the pipe is full (`be/src/io/fs/stream_load_pipe.cpp:200-212`; the pipe size is `kMaxPipeBufferedBytes`). ### Description Each libevent worker (`webserver_num_workers`, default 128) serves many keep-alive connections. When a callback blocks, every other connection on that worker waits, including other loads' body reads and unrelated endpoints such as `/api/health` or `/metrics`. 1. **Header.** Before any body byte is read, `on_header` performs two synchronous FE RPCs (`begin_txn`, then `streamLoadPut`) and starts the fragment. While one load waits on the FE, the other connections on that loop stall. 2. **Full pipe.** When the scanner is slower than the network, `on_chunk_data` blocks the loop thread until the scanner drains the pipe. Back-pressure should apply to that one connection, not to the whole worker. What we measured: an off-CPU profile with 32 concurrent senders showed `EvHttpServer` threads blocked for 91 s per 2 s window in `StreamLoadPipe::_append`. On 4.1.4 we also saw the finish wait (fixed on master): Stream Load receive p50 rose to 3.0–3.7 s at only 4–5 Gbit/s, because body reads queued behind a neighbouring load's write and commit. We do not have a production measurement of the header part alone. ### Solution We have a patch that keeps each load's work off the loop. Both changes are behind mutable configs: 1. **Pause instead of block.** When the pipe is full, stop reading that connection, and resume once the scanner has drained the pipe to half. Other connections on the loop keep being served. Reads are resumed only while the body is still being read, because libevent aborts if reading is re-enabled in the wrong state. Cancelling the pipe also resumes reads. 2. **Header off the loop.** On the loop, `on_header` only parses and validates the headers. It pauses reads and hands `begin_txn`, the plan and the fragment start to a thread pool. Body bytes that arrive with the headers are held until the pipe exists. The result is posted back to the loop, which resumes reads. On error, the load rolls back and replies exactly as the inline path does. A load that times out while queued fails before `begin_txn`. Add a `HeaderWaitTimeMs` timing to the reply. ### How to Reproduce? 1. Set `webserver_num_workers` low (e.g. 2) to make collisions likely. 2. Send several concurrent keep-alive stream loads, where one load's scanner is slow (a large Arrow/JSON body into a wide table), or make `begin_txn` slow (for example with an FE debug point or a busy FE). 3. Meanwhile, time a second load or `curl /api/health` on the same port. Its latency follows the slow load's header phase or the time its pipe is full. ### Anything Else? Unit tests cover both changes. They show that a full pipe, or a slow `begin_txn` (via a debug point), on one loop no longer stalls a second request on that loop, and that the old path does. A small A/B on a test box (2 event loops, 8 clients) showed no throughput change (541k vs 530k rows/s, within noise). So we expect the gain to be in tail latency and isolation, not in peak throughput. Trade-offs: a pipe can exceed its capacity by up to one network read before reads pause. Errors found during the FE work arrive after `100-continue` has been sent. The `http_stream` endpoint (`HttpStreamAction`) still runs this work on the loop. Related: #53144 (closes #53139) already moved the finish/commit wait off the loop on master; it is not in branch-4.1, where we saw it too. The full-pipe stall was reported as part B of #67777, which was closed by a metrics fix (#67945) that did not change Stream Load. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
