jasperjiaguo commented on PR #11496: URL: https://github.com/apache/pinot/pull/11496#issuecomment-1715071399
> > @jasperjiaguo - you may also want to share numbers from the internal production test we did using this, before v/s after numbers / observations from heap dump etc > > ``` > > * Direct memory usage should come down after initial failure > > > > * No repeated retries from Netty > > ``` > > Yes, this would be very helpful. So the direct memory graph looks like <img width="1166" alt="Screenshot 2023-09-11 at 11 22 37 PM" src="https://github.com/apache/pinot/assets/10736840/2edf48fb-b395-48e9-9a81-206e3bacc655";> And error looks like ``` curl 'https://xxxxx:xxxx/pinot-broker/resources/query' --data-urlencode 'q=query' --data-urlencode 'sql=xxxxx' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 138M 0 138M 0 0 788k 0 --:--:-- 0:02:59 --:--:-- 8956k { "metadata": { "columns": [ { "columnType": "STRING", "columnName": "XXX" }, { "columnType": "LONG", "columnName": "count(*)" } ], "hasPartialResults": true, "numEntriesScannedPostFilter": 215036581, "numDocsScanned": 215036581, "errors": [ { "type": "UNKNOWN", "message": "org.apache.pinot.spi.exception.QueryCancelledException: Query cancelled as broker is out of direct memory\n\tat org.apache.pinot.core.transport.DirectOOMHandler.exceptionCaught(DirectOOMHandler.java:80)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)" ``` Also took a heap dump and direct buffer are all clean after the incident -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
