walterddr opened a new issue, #12309: URL: https://github.com/apache/pinot/issues/12309
there are several execution path that could cause a legitimate OOM issue when executing a super large result set query such as `select * FROM a JOIN b on a.boolCol = b.boolCol`, this could result in multiplication result size compared to either table's row size. We need to safeguard against these type of OOM. 1. broker is caching all data and return back all at once. which is most likely to OOM 2. we dont have a streaming SortOperator so potentially a singleton server that does the final stage ORDER BY can also get OOM. this not only will cause the servers to crash but also incur very large GC overhead when these types of queries are ran. Related ==== #12304 addresses this partially by not allowing these types of non-limited queries to be accidentally run; but this one is to fix it more fundamentally in a safe-guard manner. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
