On 10/29/2018 7:24 AM, Sofiya Strochyk wrote:
Actually the smallest server doesn't look bad in terms of performance,
it has been consistently better that the other ones (without
replication) which seems a bit strange (it should be about the same or
slightly worse, right?). I guess the memory being smaller than index
doesn't cause problems due to the fact that we use SSDs.
SSD, while fast, is nowhere near as fast as main memory. As I said, the
memory numbers might cause performance problems, or they might not.
Glad you're in the latter category.
What if we are sending requests to machine which is part of the
cluster but doesn't host any shards? Does it handle the initial
request and merging of the results, or this has to be handled by one
of the shards anyway?
Also i was thinking "more shards -> each shard searches smaller set of
documents -> search is faster". Or is the overhead for merging results
bigger than overhead from searching larger set of documents?
If every shard is on its own machine, many shards might not be a
performance bottleneck with a high query rate. The more shards you
have, the more the machine doing the aggregation must do to produce results.
SolrCloud complicates the situation further. It normally does load
balancing of all requests that come in across the cloud. So the machine
handling the request might not be the machine where you SENT the request.
Very likely the one with a higher load is the one that is aggregating
shard requests for a full result.
Is there a way to confirm this? Maybe the aggregating shard is going
to have additional requests in its solr.log?
The logfiles on your servers should be verbose enough to indicate what
machines are handling which parts of the request.
Most Solr performance issues are memory related. With an extreme
query rate, CPU can also be a bottleneck, but memory will almost
always be the bottleneck you run into first.
This is the advice i've seen often, but how exactly can we run out of
memory if total RAM is 128, heap is 8 and index size is 80. Especially
since node with 64G runs just as fine if not better.
Even when memory is insufficient, "running out" of memory generally
doesn't happen unless the heap is too small.Java will work within the
limits imposed by the system if it can. For OS disk cache, the OS tries
to be as smart as it can about which data stays in the cache and which
data is discarded.
A lot of useful information can be obtained from the GC logs that
Solr's built-in scripting creates. Can you share these logs?
The screenshots described here can also be very useful for
troubleshooting:
https://wiki.apache.org/solr/SolrPerformanceProblems#Asking_for_help_on_a_memory.2Fperformance_issue
I have attached some GC logs and screenshots, hope these are helpful
(can only attach small files)
Only one attachment made it to the list. I'm surprised that ANY of them
made it -- usually they don't. Generally you need to use a file sharing
website and provide links. Dropbox is one site that works well. Gist
might also work.
The GC log that made it through (solr_gc.log.7.1) is only two minutes
long. Nothing useful can be learned from a log that short. It is also
missing the information at the top about the JVM that created it, so I'm
wondering if you edited the file so it was shorter before including it.
Thanks,
Shawn
