geekboood commented on issue #10960: URL: https://github.com/apache/pinot/issues/10960#issuecomment-1635724282
@Jackie-Jiang I studied the issue further and found that DirectByteBuffer would be freed eventually when the corresponding Java object on the heap is garbage collected. However, since the Java heap pressure is not heavy, it seems like this would take quite a long time. I discovered that there exists a Java option, `-XX:MaxDirectMemorySize`, which can limit the total allocated off-heap memory. The default value is set to the heap's maximum memory, which means that when I set `-Xmx24G`, it could use a total of 48GB of memory and lead to an OOM (out of memory) error. After I set `-XX:MaxDirectMemorySize=4G`, everything seems fine. However, after setting this option, I encountered an OutOfMemory issue when running a large query. I think explicitly closing the ColumnValueReaders could help with running large queries. For now, I set `deriveNumDocsPerChunkForRawIndex=true` for the large JSON column, and the uncompressed chunk size will be smaller, allowing for smaller DirectByteBuffers to be allocated. Currently, everything is running smoothly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
