vagetablechicken opened a new pull request #3143: Non blocking OlapTableSink URL: https://github.com/apache/incubator-doris/pull/3143 ## ImplementaItion Notes ### NodeChannel 1. _cur_batch -> _pending_batches: when _cur_batch is filled up, move it to _pending_batches. 2. add_row() just produce batches. 3. try_send_and_fetch_status() tries to consume one pending batch. If has in flight packet, skip send in this round. So we can add one sender thread to be in charge of all node channels try_send. ### IndexChannel 1. init(), open() stay the same. 2. Use for_each_node_channel() to expose the detailed changes of NodeChannel.(It's more easy to read & modify) ### Sender thread See func OlapTableSink::_send_batch_process() #### Why use polling？ If we use wait/notify, it will notify when generate a new batch. We can't skip sending this batch, coz it won't notify the same batch again. So wait/notify can't avoid blocking simply. So I choose polling. It's wasting to continuously try_send(), but it's difficult to set the suitable polling interval. Thus, I add std::this_thread::yield() to give up the time slice, give priority to other process/threads (if there are other process/threads waiting in the queue).
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
