Yonik, Don't you think that proper codec format can get the comparable gain without changes in design? https://issues.apache.org/jira/browse/LUCENE-5052
On Mon, Jan 13, 2014 at 9:15 PM, Yonik Seeley <ysee...@gmail.com> wrote: > Update on the my initial performance findings for off-heap filters: > http://heliosearch.org/off-heap-filters/ > > -Yonik > http://heliosearch.org -- making solr shine > > > On Tue, Jan 7, 2014 at 1:53 PM, Yonik Seeley <ysee...@gmail.com> wrote: > > Off-Heap Filters: > > JVMs have never been good at dealing with large heaps. Large heaps > > mean the JVM needs to do a lot of garbage collection work, and often > > means some pretty long stop-the-world GC pauses. > > > > Filters (Solr DocSets) stored in the filterCache are now allocated > > off-heap and reference counted so they can be freed as soon as they > > are no longer needed. The JVM no longer needs to waste time copying > > around these potentially long-lived blocks of memory. This should both > > help eliminate the long GC pauses as well as increase request > > throughput. > > > > Performance Results: > > I'm still putting together a blog on the results, but they look good! > > It was pretty trivial to reproduce >1s stop-the-world GC pauses with a > > 4GB heap, and then see those pauses completely go away when I switched > > to off-heap filters. Throughput also increased since much less time > > was spent doing GC. > -- Sincerely yours Mikhail Khludnev Principal Engineer, Grid Dynamics <http://www.griddynamics.com> <mkhlud...@griddynamics.com>
