caiconghui opened a new issue #3696:
URL: https://github.com/apache/incubator-doris/issues/3696
Now, there are many rpcs between be and fe, and some default config in be
may cause poor performance for query or stream load. In our experiment, we find
that when we execute stream load in doris, the txn can be easily publish
timeout and query can be easily rpc timeout, http request to be can be easily
no response.
And we read the code for server in be, and we find the num of thrift server
threads is too small to handle so many rpc which include exec_plan_fragment and
submit_tasks at the same time, and http server use reactor model to process
request, when the stream load performance is poor (server thread would be
blocked in rpc or other blocked function) and the num of threads in http
server is small, it will cause http server can not handle http request in time.
Finally, we do some experiments after change the some config in be, and we
find that query and stream load both get good performance and http server can
response in time.
we use thread pool in java client to make 10000 stream load requests to fe
with 5 be instance. Every request has one row data, and we have 500 tables in
on db, every table will be inserted 20 rows. the total time cost for java
client is 35398ms, and some stream load result are as following:
{
"TxnId": 276003,
"Label": "86292a00-511b-47de-82e2-6dfa9c2d219d",
"Status": "Success",
"Message": "OK",
"NumberTotalRows": 1,
"NumberLoadedRows": 1,
"NumberFilteredRows": 0,
"NumberUnselectedRows": 0,
"LoadBytes": 12,
"LoadTimeMs": 31,
"BeginTxnMs": 0,
"ReadBufferMs": 0,
"WriteBufferMs": 12,
"StreamLoadPutMs": 0,
"CommitAndPublishMs": 15
}
{
"TxnId": 276006,
"Label": "7d6023a4-5fb5-4551-9327-4c758f231851",
"Status": "Success",
"Message": "OK",
"NumberTotalRows": 1,
"NumberLoadedRows": 1,
"NumberFilteredRows": 0,
"NumberUnselectedRows": 0,
"LoadBytes": 12,
"LoadTimeMs": 31,
"BeginTxnMs": 0,
"ReadBufferMs": 0,
"WriteBufferMs": 12,
"StreamLoadPutMs": 0,
"CommitAndPublishMs": 15
}
{
"TxnId": 275256,
"Label": "3b29cd47-2aa8-4255-9d37-df6148570e93",
"Status": "Success",
"Message": "OK",
"NumberTotalRows": 1,
"NumberLoadedRows": 1,
"NumberFilteredRows": 0,
"NumberUnselectedRows": 0,
"LoadBytes": 11,
"LoadTimeMs": 34,
"BeginTxnMs": 0,
"ReadBufferMs": 0,
"WriteBufferMs": 12,
"StreamLoadPutMs": 0,
"CommitAndPublishMs": 18
}
load all time 35398
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
