Thanks Erick,
I have another index with the same infrastructure setup, but only 10m
docs, and never see these slow-downs, that's why my first instinct was
to look at creating more shards.
I'll definitely make a point of investigating further tho with all the
things you and Shawn mentioned, time is unfortunately against me.
Cheers,
Rob
On 19/03/16 19:11, Erick Erickson wrote:
Be _very_ cautious when you're looking at these timings. Random
spikes are often due to opening a new searcher (assuming
you're indexing as you query) and are eminently tunable by
autowarming. Obviously you can't fire the same query again and again,
but if you collect a set of "bad" queries and, say, measure them after
Solr has been running for a while (without any indexing going on) and
the times are radically better, then autowarming is where I'd look first.
Second, what are you measuring? Time until the client displays the
results? There's a bunch of other things going on there, it might be
network issues. The QTime is a rough measure of how long Solr is
taking, although it doesn't include the time spent assembling the return
packet.
Third, "randomly requesting facets" is something of a red flag. Make
really sure the facets are realistic. Fields with high cardinality make
solr work harder. For instance, let's say you have a date field with
millisecond resolution. I'd bet that faceting on that field is not something
you'll ever support. NOTE: I'm talking about just setting
facet.field=datefield
here. A range facet on the field is totally reasonable. Really, I'm saying
to insure that your queries are realistic before jumping into sharding.
Fourth, garbage collection (especially "stop the world" GCs) won't be helped
by just splitting into shards.
And the list goes on and on. Really what both Shawn and I are saying is
that you really need to identify _what's_ slowing you down before trying
a solution like sharding. And you need to be able to quantify that rather
than
"well, sometimes when I put stuff in it seems slow" or you'll spend a large
amount of time chasing the wrong thing (at least I have).
30M docs per shard is well within a reasonable range, although the
complexity of your docs may push that number up or down. You haven't told
us much about how much memory you have on your machine, how much
RAM you're allocating to Solr and the like so it's hard to say much other
than generalities....
Best,
Erick
On Sat, Mar 19, 2016 at 10:41 AM, Shawn Heisey <apa...@elyograg.org> wrote:
On 3/19/2016 11:12 AM, Robert Brown wrote:
I have an index of 60m docs split across 2 shards (each with a replica).
When load testing queries (picking random keywords I know exist), and
randomly requesting facets too, 95% of my responses are under 0.5s.
However, during some random manual tests, sometimes I see searches
taking between 1-2 seconds.
Should I expect a simple shard split to assist with the speed
immediately? Even with the 2 new shards still being on the original
servers?
Will move them to their own dedicated hosts, but just want to
understand what I should expect during the process.
Maybe. It depends on why the responses are slow in the first place.
If your queries are completely CPU-bound, then splitting into more
shards and either putting those shards on additional machines or taking
advantage of idle CPUs will make performance better. Note that if your
query rate is extremely high, you should only have one shard replica on
each server -- all your CPU power will be needed for handling query
volume, so none of your CPUs will be idle.
Most of the time, Solr installations are actually I/O bound, because
there's not enough unused RAM to effectively cache the index. If this
is what's happening and you don't add memory (which you can do by adding
machines and adding/removing replicas to move them), then you'll make
performance worse by splitting into more shards.
Thanks,
Shawn
