(11/06/12 8:27), Markus Jelsma wrote:
Hi,
How can i calculate the memory requirements for highlighting documents where
the variables are number of documents to highlight and the total size of the
documents in a single request? I can then multiply a worst-case result set
with the expected concurrency.
For example, with a _limited_ heap highlighting works perfectly well and is
only CPU-bound, which is clearly visible when highlighting a result set with
many very large documents. When such a worst-case query is fired a lot of times
concurrently Solr is guaranteed to run out of memory.
So, now i'm looking for a method to calcualte per-request memory consumption
for set-ups with FVH and all term* settings enabled on the highlighted field in
schema.
Though FVH uses some structured objects described at:
http://lucene.apache.org/java/3_2_0/api/all/org/apache/lucene/search/vectorhighlight/package-summary.html#package_description
during its processing, but these should be relatively small.
So, I think huge stored field data to be highlighted consumes a lot of heap.
To limit it, think of use of copyField with maxChars for the highlight field.
koji
--
http://www.rondhuit.com/en/
