[GitHub] [incubator-doris] zbtzbtzbt opened a new issue #8522: [Enhancement] speed up stream load 50%

[GitBox]

zbtzbtzbt opened a new issue #8522:
URL: https://github.com/apache/incubator-doris/issues/8522


   - pr: https://github.com/apache/incubator-doris/pull/8500
   
   - **speed up data load for no-vec-engine 35%~100% , avg enhance 50%**
   🚀🚀🚀
   
   <img width="1259" alt="数据2" 
src="https://user-images.githubusercontent.com/35688959/158932089-bcd8d7f1-36c4-4e59-bde0-04c112780ae8.png";>
   <img width="958" alt="实验数据1" 
src="https://user-images.githubusercontent.com/35688959/158932106-30eeecfd-8198-4e28-beef-1f8a9c44ef73.png";>
   
   **master:**
   mainly cost is from `insert()` funtion's `get_type_info()`
   
   <img width="1164" alt="load" 
src="https://user-images.githubusercontent.com/35688959/158932834-d5686af5-b937-4612-8cd5-a1458418faf0.png";>
   
   
   
   **opt_data_load:**
   
   via perf I find hotspot function change from `get_type_info` to 
`append_data_in_current_page`
   <img width="1101" alt="优化后" 
src="https://user-images.githubusercontent.com/35688959/158932717-698ac162-834b-4fcb-b772-9d7cb271eb17.png";>
   
   <img width="1216" alt="优化后2" 
src="https://user-images.githubusercontent.com/35688959/158932775-74848efc-a846-4b96-ad7e-0aa16a6e949a.png";>
   
   The perf data may not be so accurate, I tested it again，here is the result:
   <img width="1244" alt="perf2" 
src="https://user-images.githubusercontent.com/35688959/158932984-1b962036-4936-4ca1-ab28-aa5b81d76cfd.png";>
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org