Yes, but 581 is the final number you got in the response, which is the
result of the main query intersected with the filter query so I wouldn't
take in account this number. The main and the filter query are executed
separately, so I guess (but I'm guessing because I don't know these
internals) that's here where the "rows" parameter matters.
Again, I'm guessing, I'm sure some Solr committer here can explain you
how things are working.
Best,
Andrea
On 31/07/18 11:12, Fette, Georg wrote:
Hi Andrea,
I agree that receiving too much data in one request is bad. But I was
surprised that the query works with a lower but still very large rows
parameter and that there is a threshold at which it crashes the
server. Furthermore, it seems that the reason for the crash is not the
size of the actual results because those are only 581.
Greetings
Georg
Am 31.07.2018 um 10:53 schrieb Andrea Gazzarini:
Hi Georg,
I would say, without knowing your context, that this is not what Solr
is supposed to do. You're asking to load everything in a single
request/response and this poses a problem.
Since I guess that, even we assume it works, you should then iterate
those results one by one or in blocks, an option would be to do this
part (block scrolling) using Solr [2].
I suggest you to have a look at
* the export endpoint [1]
* the cursor API [2]
Best,
Andrea
[1] https://lucene.apache.org/solr/guide/6_6/exporting-result-sets.html
[2]
https://lucene.apache.org/solr/guide/6_6/pagination-of-results.html#fetching-a-large-number-of-sorted-results-cursors
On 31/07/18 10:44, Georg Fette wrote:
Hello,
We run the server version 7.3.1. on a machine with 32GB RAM in a
mode having -10g.
When requesting a query with
q={!boost
b=sv_int_catalog_count_document}string_catalog_aliases:(*2*)&fq=string_field_type:catalog_entry&rows=2147483647
the server takes all available memory up to 10GB and is then no
longer accessible with one processor at 100%.
When we reduce the rows parameter to 10000000 the query works. The
query returns only 581 results.
The documentation at
https://wiki.apache.org/solr/CommonQueryParameters states that as
the "rows" parameter a "ridiculously large value" may be used, but
this could pose a problem. The number we used was Int.max from Java.
Greetings
Georg
