vagetablechicken commented on issue #2780: OlapTableSink::send is low efficient? URL: https://github.com/apache/incubator-doris/issues/2780#issuecomment-596974767 https://github.com/apache/incubator-doris/pull/2956#issuecomment-596889947 As mentioned, here‘s the new design of OlapTableSink--add one sender thread to do non-blocking sending. Let me explain the meaning of non-blocking. The origin version of OlapTableSink can be abstracted as one queue(contains all batches in all node channels). One thread consumes the queue's items, one by one. When it wants to send a batch of the NodeChannel which has a in flight packet(rpc hasn’t returned response), it must wait(rpc join). For example: The batch which index id=0 && node id=2, is denoted by "B1(0-2)". The Abstract queue as shown below. B0(0-1) | B1(0-2) | B2(0-1) | B3(1-4) | ... | ... -- | -- | -- | -- | -- | -- When we are sending B2(0-1), we must wait for the B0 response. But if we set aside B2(0-1) and send the next item "B3(1-4)", it won't be blocked. So I used to split the one queue into multi queues(abandoned), as follows.( For details, see https://github.com/apache/incubator-doris/issues/2780#issuecomment-588156273) Batches queue0: B0(0-0) | B1(0-0) | B2(2-0) | B3(4-0) | ... | ... -- | -- | -- | -- | -- | -- Batches queue1: B0(2-1) | B1(0-1) | B2(0-1) | B3(1-1) | ... | ... -- | -- | -- | -- | -- | -- Batches queue2: B0(0-2) | B1(0-2) | B2(3-2) | B3(1-2) | ... | ... -- | -- | -- | -- | -- | -- Each queue needs one thread to consume items. Block time is shared by multiple queues. But it's still a block way. ### The new design is non-blocking. We can save batches in NodeChannels(pending batches), and **try to send** a pending batch. If the current channel has a in flight packet, we just skip sending in this round. The implementation is coming soon.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
