vagetablechicken edited a comment on issue #2780: OlapTableSink::send is low efficient? URL: https://github.com/apache/incubator-doris/issues/2780#issuecomment-575530354 @morningman That means the memory usage of a row batch may be great? So thus, we should stop pushing more batch as follows: https://github.com/apache/incubator-doris/blob/8df63bc191fd123df5986084c1e453db732fda29/be/src/exec/broker_scan_node.cpp#L366-L378 I think even if in the single-thread model, it's more than 1 row batch in memory. Let's consider the worst-case scenarios. ### The single-thread model Divide into three roles: **1 scanner (holds one _batch_queue)**: The mem usage of _batch_queue is denoted by Size<sub>queue</sub>. _batch_queue is not empty and memory exceed limit(aka mem_limit), thus Size<sub>queue</sub> > mem_limit **1 plan_fragment_executor (holds one _row_batch)**： _row_batch comes from _batch_queue, denoted by Size<sub>batch</sub>. If it's too large, we can assume: Size<sub>batch</sub> > mem_limit **N<sub>nc</sub> NodeChannels(holds deep_copy rows, aka "_batch")** If "_batch"s in all channels are near full capacity, every channel has batch_size-1 rows, the mem usage is approximately equal to Size<sub>batch</sub>. The mem usage of all NodeChannels is N<sub>be</sub>*Size<sub>batch</sub>. So, Mem<sub>worst</sub> = Size<sub>queue</sub>+(1+N<sub>nc</sub>)*Size<sub>batch</sub>. Mem<sub>worst</sub> > (2+N<sub>nc</sub>)*mem_limit. ### The multi-thread model If we use a thread pool or whatever to do data sending, we should add a new role, sender. The best parallelism is that one sender is in charge of one channel. **1 scanner (holds one _batch_queue)** SAME **1 plan_fragment_executor (holds one _row_batch)** SAME **N<sub>nc</sub> send_worker** send_workers will buffer rows, this should be strictly controlled of mem usage. The mem usage of one sender is denoted by Size<sub>buf</sub>. The sum is N<sub>nc</sub>*Size<sub>buf</sub>. **N<sub>ch</sub> NodeChannels(holds deep_copy rows, aka "_batch")** SAME So, Mem<sub>worst</sub> = Size<sub>queue</sub>+(1+N<sub>nc</sub>)*Size<sub>batch</sub>+N<sub>nc</sub>*Size<sub>buf</sub>. Mem<sub>worst</sub> > (2+N<sub>nc</sub>)*mem_limit + N<sub>nc</sub>*Size<sub>buf</sub> If Size<sub>buf</sub> is significantly smaller than mem_limit, it can be ignored. If Size<sub>buf</sub> is smaller than the bytes per row, it still might bring some benefits, cause it improves the parallel degree of wait_in_flight_packet.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
