Hi Joel, Thanks for the insight. How can this exception be thrown/forced from client side. Client can't do a System.exit() as it is running as a webapp.
Thanks, Susmit On Fri, May 12, 2017 at 4:44 PM, Joel Bernstein <joels...@gmail.com> wrote: > In this scenario the /export handler continues to export results until it > encounters a "Broken Pipe" exception. This exception is trapped and ignored > rather then logged as it's not considered an exception if the client > disconnects early. > > Joel Bernstein > http://joelsolr.blogspot.com/ > > On Fri, May 12, 2017 at 2:10 PM, Susmit Shukla <shukla.sus...@gmail.com> > wrote: > > > Hi, > > > > I have a question regarding solr /export handler. Here is the scenario - > > I want to use the /export handler - I only need sorted data and this is > the > > fastest way to get it. I am doing multiple level joins using streams > using > > /export handler. I know the number of top level records to be retrieved > but > > not for each individual stream rolling up to the final result. > > I observed that calling close() on a /export stream is too expensive. It > > reads the stream to the very end of hits. Assuming there are 100 million > > hits for each stream ,first 1k records were found after joins and we call > > close() after that, it would take many minutes/hours to finish it. > > Currently I have put close() call in a different thread - basically fire > > and forget. But the cluster is very strained because of the unneccessary > > reads. > > > > Internally streaming uses ChunkedInputStream of HttpClient and it has to > be > > drained in the close() call. But from server point of view, it should > stop > > sending more data once close() has been issued. > > There is a read() call in close() method of ChunkedInputStream that is > > indistinguishable from real read(). If /export handler stops sending more > > data after close it would be very useful. > > > > Another option would be to use /select handler and get into business of > > managing a custom cursor mark that is based on the stream sort and is > reset > > until it fetches the required records at topmost level. > > > > Any thoughts. > > > > Thanks, > > Susmit > > >
