weizuo93 opened a new issue #4882: URL: https://github.com/apache/incubator-doris/issues/4882
The current compaction mechanism is that there is a producer thread that has been producing compaction tasks, and the selected tablet must apply for `permits`. When a tablet could hold `permits`, compaction task for this tablet will be submitted to thread pool. We take compaction score as `permits` which is used for limiting memory consumption. However, `pick_rowset_to_compaction()` will be executed before the file merge in compaction thread, and the number of segment files that actually perform the merge operation is smaller than compaction score. In addition, it is also possible that compaction task exits directly because the tablet doesn't meet the requirements of compaction. Can we execute 'pick rowsets' before applying for permits for a compaction task, calculate the number of segment files that actually participate in the merge operation, and take this number as `permits`. Based on test results, we found that`pick_rowset_to_compaction()` execution time for a tablet is on the time scale of milliseconds. Therefore, there is little pressure on the producer thread. ``` 1112 16:00:08.686122 (+ 982us) cumulative_compaction.cpp:50] rowsets picked 1112 16:00:08.355522 (+ 1033us) cumulative_compaction.cpp:50] rowsets picked 1112 16:00:08.355372 (+ 930us) cumulative_compaction.cpp:50] rowsets picked 1112 16:00:09.707385 (+ 990us) cumulative_compaction.cpp:50] rowsets picked 1112 16:00:32.218772 (+ 1305us) cumulative_compaction.cpp:50] rowsets picked 1112 16:00:32.212823 (+ 1212us) cumulative_compaction.cpp:50] rowsets picked 1112 16:00:32.077322 (+ 2411us) cumulative_compaction.cpp:50] rowsets picked 1112 16:00:33.534394 (+ 2484us) cumulative_compaction.cpp:50] rowsets picked 1112 16:00:52.350023 (+ 1181us) cumulative_compaction.cpp:50] rowsets picked 1112 16:00:54.011435 (+ 1212us) cumulative_compaction.cpp:50] rowsets picked 1112 16:01:14.845957 (+ 294us) cumulative_compaction.cpp:50] rowsets picked 1112 16:00:17.865887 (+ 97us) base_compaction.cpp:44] rowsets picked 1112 16:01:35.193204 (+ 206us) cumulative_compaction.cpp:50] rowsets picked 1112 16:01:39.244388 (+ 198us) cumulative_compaction.cpp:50] rowsets picked 1112 16:02:18.931984 (+ 1627us) cumulative_compaction.cpp:50] rowsets picked 1112 16:02:19.014592 (+ 1394us) cumulative_compaction.cpp:50] rowsets picked 1112 16:03:03.405753 (+ 160us) cumulative_compaction.cpp:50] rowsets picked 1112 16:03:04.777374 (+ 350us) cumulative_compaction.cpp:50] rowsets picked 1112 16:03:07.016017 (+ 164us) cumulative_compaction.cpp:50] rowsets picked 1112 15:59:12.573256 (+ 205us) base_compaction.cpp:44] rowsets picked 1112 16:02:18.932822 (+ 2213us) cumulative_compaction.cpp:50] rowsets picked 1112 16:03:08.975090 (+ 339us) cumulative_compaction.cpp:50] rowsets picked 1112 16:02:18.932387 (+ 1885us) cumulative_compaction.cpp:50] rowsets picked 1112 16:03:11.285206 (+ 402us) cumulative_compaction.cpp:50] rowsets picked 1112 16:03:14.799639 (+ 119us) cumulative_compaction.cpp:50] rowsets picked 1112 16:03:20.540975 (+ 148us) cumulative_compaction.cpp:50] rowsets picked 1112 16:03:23.908919 (+ 198us) cumulative_compaction.cpp:50] rowsets picked 1112 16:03:32.000026 (+ 133us) cumulative_compaction.cpp:50] rowsets picked 1112 16:03:13.347364 (+ 2796us) cumulative_compaction.cpp:50] rowsets picked 1112 16:03:13.722429 (+ 2046us) cumulative_compaction.cpp:50] rowsets picked 1112 16:03:43.050516 (+ 410us) cumulative_compaction.cpp:50] rowsets picked 1112 16:03:43.849807 (+ 387us) cumulative_compaction.cpp:50] rowsets picked 1112 16:02:17.914220 (+ 469us) base_compaction.cpp:44] rowsets picked 1112 16:03:48.663824 (+ 336us) cumulative_compaction.cpp:50] rowsets picked 1112 16:03:49.778246 (+ 241us) cumulative_compaction.cpp:50] rowsets picked 1112 16:03:52.141355 (+ 468us) cumulative_compaction.cpp:50] rowsets picked 1112 16:04:20.670042 (+ 1564us) cumulative_compaction.cpp:50] rowsets picked 1112 16:04:38.078296 (+ 264us) cumulative_compaction.cpp:50] rowsets picked 1112 16:04:36.673882 (+ 1177us) cumulative_compaction.cpp:50] rowsets picked 1112 16:04:37.060312 (+ 1139us) cumulative_compaction.cpp:50] rowsets picked 1112 16:04:55.154987 (+ 1215us) cumulative_compaction.cpp:50] rowsets picked 1112 16:04:55.403399 (+ 1071us) cumulative_compaction.cpp:50] rowsets picked 1112 16:05:02.536426 (+ 1551us) cumulative_compaction.cpp:50] rowsets picked 1112 16:05:32.090439 (+ 1024us) cumulative_compaction.cpp:50] rowsets picked 1112 16:05:18.661826 (+ 1306us) cumulative_compaction.cpp:50] rowsets picked 1112 16:05:18.990078 (+ 1241us) cumulative_compaction.cpp:50] rowsets picked 1112 16:05:42.798424 (+ 143us) cumulative_compaction.cpp:50] rowsets picked 1112 16:05:46.082527 (+ 176us) cumulative_compaction.cpp:50] rowsets picked 1112 16:05:41.727956 (+ 1584us) cumulative_compaction.cpp:50] rowsets picked 1112 16:06:01.083588 (+ 1123us) cumulative_compaction.cpp:50] rowsets picked 1112 16:06:26.622230 (+ 632us) cumulative_compaction.cpp:50] rowsets picked 1112 16:06:35.897463 (+ 92us) cumulative_compaction.cpp:50] rowsets picked ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
