Thanks to everyone for the suggestions. We managed to get the
performance to a bearable level by splitting the index into ~20 separate
collections (one collection per country) and spreading them between
existing servers as evenly as possible. The largest country is also
split into 2 shards. Thi
On Mon, 2018-11-12 at 14:19 +0200, Sofiya Strochyk wrote:
> I'll check if the filter queries or the main query tokenizers/filters
> might have anything to do with this, but I'm afraid query
> optimization can only get us so far.
Why do you think that? As you tried eliminating sorting and retrieva
From what I've gathered and what's been my experience docValues should
be enabled, but if you can't think of anything else, I'd try turning
them off to see if it makes any difference. As far as I can recall
turning them off will increase usage of Solr's own caches and that
caused noticeable slo
Thanks for the suggestion Ere. It looks like they are actually enabled;
in schema file the field is only marked as stored (field name="_id"
type="string" multiValued="false" indexed="true" required="true"
stored="true") but the admin UI shows DocValues as enabled, so I guess
this is by default.
Thanks for your suggestions. I'll check if the filter queries or the
main query tokenizers/filters might have anything to do with this, but
I'm afraid query optimization can only get us so far. Since we will have
to add facets later, the queries will only become heavier, and there has
to be a w
Sofiya,
Do you have docValues enabled for the id field? Apparently that can make
a significant difference. I'm failing to find the relevant references
right now, but just something worth checking out.
Regards,
Ere
Sofiya Strochyk kirjoitti 6.11.2018 klo 16.38:
Hi Toke,
sorry for the late r
On Tue, 2018-11-06 at 16:38 +0200, Sofiya Strochyk wrote:
> I have tested all of the suggested changes none of these seem to make
> a noticeable difference (usually response time and other metrics
> fluctuate over time, and the changes caused by different parameters
> are smaller than the fluctuati
Hi Toke,
sorry for the late reply. The query i wrote here is edited to hide
production details, but I can post additional info if this helps.
I have tested all of the suggested changes none of these seem to make a
noticeable difference (usually response time and other metrics fluctuate
over
So far no answer from Sofiya. That's fair enough: My suggestions might
have seemed random. Let me try to qualify them a bit.
What we have to work with is the redacted query
q=&fl=&start=0&sort=&fq=&rows=24&version=2.2&wt=json
and an earlier mention that sorting was complex.
My suggestions were t
On Wed, 2018-10-31 at 13:42 +0200, Sofiya Strochyk wrote:
> q=&fl=&start=0&sort= expression>&fq=&rows=24&version=2.2&wt=json
Not much to see here, perhaps because you are not allowed to share it?
Maybe we can try and isolate the cause? Could you try different runs,
where you change different comp
The logfiles on your servers should be verbose enough to indicate what
machines are handling which parts of the request.
Yes, generally i see the following entries in logs:
1.
df=_text_&distrib=false&fl=_id&fl=score&shards.purpose=4&start=0&fsv=true&sort=fq=&shard.url=&rows=24&version=2&q=&NO
On 10/29/2018 7:24 AM, Sofiya Strochyk wrote:
Actually the smallest server doesn't look bad in terms of performance,
it has been consistently better that the other ones (without
replication) which seems a bit strange (it should be about the same or
slightly worse, right?). I guess the memory be
Sure, here is IO for bigger machine:
https://upload.cc/i1/2018/10/30/tQovyM.png
for smaller machine:
https://upload.cc/i1/2018/10/30/cP8DxU.png
CPU utilization including iowait:
https://upload.cc/i1/2018/10/30/eSs1YT.png
iowait only:
https://upload.cc/i1/2018/10/30/CHgx41.png
On 30.10.18
Please see inline...
Deepak
"The greatness of a nation can be judged by the way its animals are
treated. Please consider stopping the cruelty by becoming a Vegan"
+91 73500 12833
deic...@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, G
On 10/29/2018 8:56 PM, Erick Erickson wrote:
The interval between when a commit happens and all the autowarm
queries are finished if 52 seconds for the filterCache. seen warming
that that long unless something's very unusual. I'd actually be very
surprised if you're really only firing 64 autowarm
My swappiness is set to 10, swap is almost not used (used space is on
scale of a few MB) and there is no swap IO.
There is disk IO like this, though:
https://upload.cc/i1/2018/10/30/43lGfj.png
https://upload.cc/i1/2018/10/30/T3u9oY.png
However CPU iowait is still zero, so not sure if the disk
Yes. Swapping from disk to memory & vice versa
Deepak
"The greatness of a nation can be judged by the way its animals are
treated. Please consider stopping the cruelty by becoming a Vegan"
+91 73500 12833
deic...@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in
Sofiya:
The interval between when a commit happens and all the autowarm
queries are finished if 52 seconds for the filterCache. seen warming
that that long unless something's very unusual. I'd actually be very
surprised if you're really only firing 64 autowarm queries and it's
taking almost 52 sec
Could you please clarify what is memory disk layer? Do you mean swapping
from memory to disk, reading from disk to memory, or something else?
On 29.10.18 17:20, Deepak Goel wrote:
I would then suspect performance is choking in memory disk layer. can
you please check the performance?
On Mon,
Sure, i can test that, will set to zero now :)
We never tried a small number for the autowarming parameter but it has
been running with zero (default value) for a while before being changed
to 64, and the startup after the commit has been a bit slow. But
overall, there was rather little differ
Speaking of your caches... Either it's a problem with the metrics
reporting or your warmup times very, very long. 11 seconds and, er,
52 seconds! My guess is that you have your autowarm counts set to a
very high number and are consuming a lot of CPU time every time a
commit happens. Which will only
I would then suspect performance is choking in memory disk layer. can you
please check the performance?
On Mon, 29 Oct 2018, 20:30 Sofiya Strochyk, wrote:
> Hi Deepak and thanks for your reply,
>
> On 27.10.18 10:35, Deepak Goel wrote:
>
>
> Last, what is the nature of your request. Are the quer
Hi Ere,
Thanks for your advice! I'm aware of the performance problems with deep
paging and unfortunately it is not the case here, as the rows number is
always 24 and next pages are hardly ever requested from what i see in
the logs.
On 29.10.18 11:19, Ere Maijala wrote:
Hi Sofiya,
You've a
Hi Deepak and thanks for your reply,
On 27.10.18 10:35, Deepak Goel wrote:
Last, what is the nature of your request. Are the queries the same? Or
they are very random? Random queries would need more tuning than if
the queries the same.
The search term (q) is different for each query, and fil
Hi Shalin,
these are stats for caches used:
*documentCache*
class:org.apache.solr.search.LRUCache
description:LRU Cache(maxSize=128, initialSize=128)
stats:
CACHE.searcher.documentCache.cumulative_evictions:234923643
CACHE.searcher.documentCache.cumulative_hitratio:0
CACHE.searcher.documentCache
On Mon, 2018-10-29 at 10:55 +0200, Sofiya Strochyk wrote:
> I think we could try that, but most likely it turns out that at some
> point we are receiving 300 requests per second, and are able to
> reasonably handle 150 per second, which means everything else is
> going to be kept in the growing que
Hi Sofiya,
You've already received a lot of ideas, but I think this wasn't yet
mentioned: You didn't specify the number of rows your queries fetch or
whether you're using deep paging in the queries. Both can be real
perfomance killers in a sharded index because a large set of records
have to
I think we could try that, but most likely it turns out that at some
point we are receiving 300 requests per second, and are able to
reasonably handle 150 per second, which means everything else is going
to be kept in the growing queue and increase response times even further..
Also, if one no
Erick,
thanks, i've been pulling my hair out over this for a long time and
gathered a lot of information :)
Doesn't there exist a setting for maxIndexingThreads in solrconfig with
default value of about 8? It's not clear if my updates are being
executed in parallel or not but i would expect
Hi Walter,
yes, after some point it gets really slow (before reaching 100% CPU
usage), so unless G1 or further tuning helps i guess we will have to add
more replicas or shards.
On 26.10.18 20:57, Walter Underwood wrote:
The G1 collector should improve 95th percentile performance, because it
What does your cache statistics look like? What's the hit ratio, size,
evictions etc?
More comments inline:
On Sat, Oct 27, 2018 at 8:23 AM Erick Erickson
wrote:
> Sofiya:
>
> I haven't said so before, but it's a great pleasure to work with
> someone who's done a lot of homework before pinging
On Fri, Oct 26, 2018 at 9:25 PM Sofiya Strochyk
wrote:
> Hi everyone,
>
> We have a SolrCloud setup with the following configuration:
>
>- 4 nodes (3x128GB RAM Intel Xeon E5-1650v2, 1x64GB RAM Intel Xeon
>E5-1650v2, 12 cores, with SSDs)
>- One collection, 4 shards, each has only a sin
On 10/26/2018 9:55 AM, Sofiya Strochyk wrote:
We have a SolrCloud setup with the following configuration:
I'm late to this party. You've gotten some good replies already. I
hope I can add something useful.
* 4 nodes (3x128GB RAM Intel Xeon E5-1650v2, 1x64GB RAM Intel Xeon
E5-1650v
Sofiya Strochyk wrote:
> Target query rate is up to 500 qps, maybe 300, and we need
> to keep response time at <200ms. But at the moment we only
> see very good search performance with up to 100 requests
> per second. Whenever it grows to about 200, average response
> time abruptly increases to 0.
Sofiya:
I haven't said so before, but it's a great pleasure to work with
someone who's done a lot of homework before pinging the list. The only
unfortunate bit is that it usually means the simple "Oh, I can fix
that without thinking about it much" doesn't work ;)
2. I'll clarify a bit here. Any
David Hastings wrote:
> Would adding the docValues in the schema, but not reindexing, cause
> errors? IE, only apply the doc values after the next reindex, but in the
> meantime keep functioning as there were none until then?
As soon as you specify in the schema that a field has docValues=true,
Would adding the docValues in the schema, but not reindexing, cause
errors? IE, only apply the doc values after the next reindex, but in the
meantime keep functioning as there were none until then?
On Fri, Oct 26, 2018 at 2:15 PM Toke Eskildsen wrote:
> Sofiya Strochyk wrote:
> > 5. Yes, docVa
Sofiya Strochyk wrote:
> 5. Yes, docValues are enabled for the fields we sort on
> (except score which is an internal field); [...]
I am currently working on
https://issues.apache.org/jira/browse/LUCENE-8374
which speeds up DocValues-operations for indexes with many documents.
What "many" means
The G1 collector should improve 95th percentile performance, because it limits
the length of pauses.
With the CMS/ParNew collector, I ran very large Eden spaces, 2 Gb out of an 8
Gb heap. Nearly all of the allocations in Solr have the lifetime of one
request, so you don’t want any of those allo
Thanks Erick,
1. We already use Solr 7.5, upgraded some of our nodes only recently to
see if this eliminates the difference in performance (it doesn't, but
I'll test and see if the situation with replicas syncing/recovery has
improved since then)
2. Yes, we only open searcher once every 30 m
Some ideas:
1> What version of Solr? Solr 7.3 completely re-wrote Leader Initiated
Recovery and 7.5 has other improvements for recovery, we're hoping
that the recovery situation is much improved.
2> In the 7x code line, there are TLOG and PULL replicas. As of 7.5,
you can set up so the queries ar
41 matches
Mail list logo
