Hi all, We launched our new production instance of SolrCloud last week and since then have noticed a trend with regards to disk usage. The non-leader replicas all seem to be self-optimizing their index segments as expected, but the leaders have (on average) around 33% more data on disk. My assumption is that leader's are not self-optimising (or not to the same extent)... but it is still early days of course.
If it helps, there are 45 JVMs in the cloud, with 15 shards and 3 replicas per shard. Each non-leader shard is sitting at between 59GB and 87GB on their SSD, but the leaders are between 84GB and 116GB. We have pretty much constant read and write traffic 24x7, with just 'slow' periods overnight when write traffic is < 1 document per second and searches are between 1 and 2 per second. Is this light level of traffic still too much for the leaders to self-optimise? I'd also be curious to hear about what others are doing in terms of operating procedures. We load test before launch what would happen if we turned off JVMs and forced recovery events. I know that these things all work, just that customers will experience slower search responses whilst they occur. For example, a restore from a leader to a replica under load testing for us takes around 30 minutes and response times drop from around 200-300ms average to 1.5s average. Bottleneck appears to be network I/O on the servers. We haven't explored whether this is specific to the servers replicating, or saturation of the of the infrastructure that all the servers share, because... This performance is acceptable for us, but I'm not sure if I'd like to force that event to occur unless required... this is following the line of reasoning proposed internally that we should periodically rotate leaders by turning them off briefly. We aren't going to do that unless we have a strong reason though. Does anyone try to manipulate production instances that way? Vaguely related to this is leader distribution. We have 9 physical servers and 5 JVMs running on each server. By virtue of the deployment procedures the first 3 servers to come online are all running 5 leaders each. Is there any merit in 'moving' these around (by reboots)? Our planning up to launch was based on lots of mailing list response we'd seen that indicated leaders had no significant performance difference to normal replicas, and all of our testing has agreed with that. The disk size 'issue' (which we aren't worried about... yet. It hasn't been in prod long enough to know for certain) may be the only thing we've seen so far. Ta, Greg
