caiconghui edited a comment on pull request #3703: URL: https://github.com/apache/incubator-doris/pull/3703#issuecomment-635199557
As a conclusion, the http server is based on libevent , which use reactor model to process http request, some http requests are processed by one thread. it assumes that every process function in libevent thread is non blocked, but there are some blocked operations in our current implementaion, such as rpc calls to fe and wait for write data operation finish. once a stream load is blocked, it will affect the following stream load, which cannot be processed in time. So This PR manily add some mertics to trace the Potential bottlenecks when stream load get bad performance. and change the default value of http server thread num to reduce the num of http requeset for per libevent server thread for one time. For now. the main time cost for stream load seems to be commit and publis txn, write data operation if we load many datas into doris. If the stream load operation really cost much time, what we can do is to think a way to escape the heavy stream load blocking other http request ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
