itschrispeck opened a new issue, #10976: URL: https://github.com/apache/pinot/issues/10976
I added some logs to track [`_luceneRealtimeReaders`](https://github.com/apache/pinot/blob/release-0.12.1/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeLuceneIndexReaderRefreshThread.java#L49) size and it looks like this queue is constantly increasing as new readers are added when segments are created. As segments are destroyed the reader is marked to note the segment is destroyed, but it remains in the queue. With ~30000 segments/week/server created we expect an increase of ~5 mins of index lag/week of server uptime (30k * (10ms sleep between reader refeshes + 0 ms refresh time)). `RealtimeLuceneIndexReaderRefreshThread` loops through this queue, and it seems like we could check whether the segment is destroyed before readding the reader to the queue. cc: @chenboat -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
