caiconghui opened a new issue #5204: URL: https://github.com/apache/incubator-doris/issues/5204
Now, many user use broker load to load hdfs data into doris, and they will submit many broker job at one time, like #5150 and loading task usually takes long time to finish task while pending task cost little time to finish task. when task queue in loadingLoadTaskScheduler is full, there is no need to begin txn for a new load job, which will just wait for other broker job to finish their loading task. Once start a new transaction, the broker job's status will be transformed from pending to loading, which may be confused for user that it is not load slowly but for the limited thread resource. So I will make two change, 1. split async_load_task_pool_size config into two configs: async_pending_load_task_pool_size, async_loading_load_task_pool_size 2. check the task queue in loadingLoadTaskScheduler is full, once full, we should not begin transaction immediately for bulk load job ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
