wangbo commented on pull request #3604: URL: https://github.com/apache/incubator-doris/pull/3604#issuecomment-632023210
After I run this pr in out production env,I find a new problem. A dim table whose size is 1.1 GB with single replica, and 1.1G is less than perNodeMemLimit(2GB)，so when the fact table is empty the broadcast join is choosen. But when the hash table actually builds, the memory usesage still execeeds which means HASH_TBL_SPACE_OVERHEAD(=1.1) is not so precise. This pr can fix the is problem when user use ```shuffle hint``` because shuffle Node runs more nodes. So I think there may two ways to solve the problem thoroughly; 1 make HASH_TBL_SPACE_OVERHEAD a configurable param,but it seems not a common solution, because I'm not sure what factors can affect the real hash table size. 2 maybe we can disable broadcast when dim table exceeds 100M ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
