RE: RE: Re: Re: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-03-12 Thread Fikavec F
I checked how an increase in the size of the outputAggregationSize and outputBufferSize in Jetty.xml affects the data transfer speed.   - outputBufferSize - set the size of the buffer into which response content is aggregated before being sent to the client. A larger buffer can improve performance

Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-03-12 Thread Fikavec F
   Continuing my research on the performance of data fetching from Solr, I noticed a significant drop in the transfer rate when the size of stored fields decreased. Below are the results of measuring the data transfer rate (wt=javabin) from a collection of 10 gigabytes in size, but consisting of a

Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-03-12 Thread David Smiley
There is compression of stored data; I don't think it makes sense to disable it. The default compression is LZ4 which is the "BEST_SPEED" option offered by Lucene compared to others. Back in 2015, the article you quoted, this faster option wasn't available. I don't see a no-compression option: h

Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-03-12 Thread Ishan Chattopadhyaya
> Again, it's worth being aware that what you are doing is very far afield > from what a search engine is *for*. So yeah... performance may not be so > great. Solr users want top-X documents sorted by something, and/or maybe > some facets/stats summarizing fields. Not all docs. Optimizing known