Hello esteemed Solr community --
I'm observing some inconsistent performance on our slave servers after
recently optimizing our master server.
Our configuration is as follows:
- all servers are hosted at Amazon EC2, running Ubuntu 8.04
- 1 master with heavy insert/update traffic, about 125K new documents
per day (m1.large, ~8GB RAM)
- autocommit every 1 minute
- 3 slaves (m2.xlarge instance sizes, ~16GB RAM)
- replicate every 5 minutes
- we have configured autowarming queries for these machines
- autowarmCount = 0
- Total index size is ~7M documents
We were seeing increasing, but gradual performance degradation across all
nodes.
So we decided to try optimizing our index to improve performance.
In preparation for the optimize we disabled replication polling on all
slaves. We also turned off all
workers that were writing to the index. Then we ran optimize on the master.
The optimize took 45-60 minutes to complete, and the total size went from
68GB down to 23GB.
We then enabled replication on each slave one at a time.
The first slave we re-enabled took about 15 minutes to copy the new files.
Once the files were copied
the performance of slave plummeted. Average response time went from 0.75 sec
to 45 seconds.
Over the past 18 hours the average response time has gradually gown down to
around 1.2 seconds now.
Before re-enabling replication the second slave, we first removed it from
our load-balanced pool of available search servers.
This server's average query performance also degraded quickly, and then
(unlike the first slave we replicated) did not improve.
It stayed at around 30 secs per query. On the theory that this is a
cache-warming issue, we added this server
back to the pool in hopes that additional traffic would warm the cache. But
what we saw was a quick spike of much worse
performance (50 sec / query on average) followed by a slow/gradual decline
in average response times.
As of now (10 hours after the initial replication) this server is still
reporting an average response time of ~2 seconds.
This is much worse than before the optimize and is a counter-intuitive
result. We expected an index 1/3 the size would be faster, not slower.
On the theory that the index files needed to be loaded into the file system
cache, I used the 'dd' command to copy
the contents of the data/index directory to /dev/null, but that did not
result in any noticeable performance improvement.
At this point, things were not going as expected. We did not expect the
replication after an optimize to result in such horrid
performance. So we decided to let the last slave continue to serve stale
results while we waited 4 hours for the
other two slaves to approach some acceptable performance level.
After the 4 hour break, we re-moved the 3rd and last slave server from our
load-balancing pool, then re-enabled replication.
This time we saw a tiny blip. The average performance went up to 1 second
briefly then went back to the (normal for us)
0.25 to 0.5 second range. We then added this server back to the
load-balancing pool and observed no degradation in performance.
While we were happy to avoid a repeat of the poor performance we saw on the
previous slaves, we are at a loss to explain
why this slave did not also have such poor performance.
At this point we're scratching our heads trying to understand:
(a) Why the performance of the first two slaves was so terrible after the
optimize. We think its cache-warming related, but we're not sure.
> 10 hours seems like a long time to wait for the cache to warm up
(b) Why the performance of the third slave was barely impacted. It should
have hit the same cold-cache issues as the other servers, if that is indeed
the root cause.
(c) Why performance of the first 2 slaves is still much worse after the
optimize than it was before the optimize,
where the performance of the 3rd slave is pretty much unchanged. We
expected the optimize to *improve* performance.
All 3 slave servers are identically configured, and the procedure for
re-enabling replication was identical for the 2nd and 3rd
slaves, with the exception of a 4-hour wait period.
We have confirmed that the 3rd slave did replicate, the number of documents
and total index size matches the master and other slave servers.
I'm writing to fish for an explanation or ideas that might explain this
inconsistent performance. Obviously, we'd like to be able to reproduce the
performance of the 3rd slave, and avoid the poor performance of the first
two slaves the next time we decide it's time to optimize our index.
thanks in advance,
Mason
