Roman, Let me briefly explain the design
special RequestParser stores servlet output stream into the context https://github.com/m-khl/solr-patches/compare/streaming#L7R22 then special component injects special PostFilter/DelegatingCollector which writes right into output https://github.com/m-khl/solr-patches/compare/streaming#L2R146 here is how it streams the doc, you see it's lazy enough https://github.com/m-khl/solr-patches/compare/streaming#L2R181 I mention that it disables later collectors https://github.com/m-khl/solr-patches/compare/streaming#L2R57 hence, no facets with streaming, yet as well as memory consumption. This test shows how it works https://github.com/m-khl/solr-patches/compare/streaming#L15R115 all other code purposed for distributed search. On Sat, Jul 27, 2013 at 4:44 PM, Roman Chyla <roman.ch...@gmail.com> wrote: > Mikhail, > If your solution gives lazy loading of solr docs /and thus streaming of > huge result lists/ it should be big YES! > Roman > On 27 Jul 2013 07:55, "Mikhail Khludnev" <mkhlud...@griddynamics.com> > wrote: > > > Otis, > > You gave links to 'deep paging' when I asked about response streaming. > > Let me understand. From my POV, deep paging is a special case for regular > > search scenarios. We definitely need it in Solr. However, if we are > talking > > about data analytic like problems, when we need to select an "endless" > > stream of responses (or store them in file as Roman did), 'deep paging' > is > > a suboptimal hack. > > What's your vision on this? > > > -- Sincerely yours Mikhail Khludnev Principal Engineer, Grid Dynamics <http://www.griddynamics.com> <mkhlud...@griddynamics.com>
