itschrispeck opened a new pull request, #15139: URL: https://github.com/apache/pinot/pull/15139
Closes `ChunkReaderContext` in `ProjectionOperator` and scan based iterators to reduce direct buffer usage. Lightly tested in a local staging cluster - will add some more query shapes soon. For the simple query: `select lower(message_logtype) from loggingstaging_wallet_api_gateway where lower(message_logtype) = 'asdf'` we see up to 80GB direct buffer memory usage, which can take 10s of minutes to be collected. e.g., per host metrics: <img width="768" alt="image" src="https://github.com/user-attachments/assets/812751fb-af33-4ef4-87cf-4b8b052af5a3"; /> With the patch, the same query creates no discernable spike in direct buffer usage (query executed at 17:43): <img width="768" alt="image" src="https://github.com/user-attachments/assets/a0863f15-54b3-415a-8bbc-31cb4a4fd215"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
