Erik - This box did have all the leaders for the dozen or so collections we have when the cloud spun up. We were able to force the leaders for other cores onto other nodes using the apis, but did not see this traffic load migrate to the new hosts when leadership changed. All nodes are NRT. The requests are 99% queries to load content on the web front ends, a few intermittent updates with comments, new content creation, etc.
Jason - 1. We are instantiating the cloud client with our VIP Load Balancer url. We ran into a memory leak issue when passing in ZK server addresses that forced this path. 2. No we did not tweak any cache TTLs 3. This codebase interacts with three collections in our cloud, and we are seeing CLUSTERSTATUS checks for all 3. Shawn - Server performance is fine and request time are great. We are tolerating the level of traffic, but the server that is taking all the hits is obviously performing a bit slower than the others. Response times are under 5MS avg for queries on all servers, which is within our perf thresholds. We are running 7.4 on the client and server side, moving to 7.5 was troublesome for us so we are holding off for the time being. Thanks, TZ On 11/6/18, 11:39 AM, "Shawn Heisey" <apa...@elyograg.org> wrote: >On 11/6/2018 9:06 AM, Zimmermann, Thomas wrote: >> For example - 75k request per minute going to this one box, and 3.5k >>RPM to all other nodes in the cloud. >> >> All of those extra requests on the one box are >>"/solr/admin/collections?collection=collectionName&action=CLUSTERSTATUS&w >>t=javabin&version=2" > >That sounds like either a bug or some kind of problem in your setup. >Over a thousand requests per second will overwhelm a single Solr node, >even if the info can be satisfied entirely from memory and doesn't >require complex calculations or large-scale data retrieval like a >regular query does. > >If you manually execute that request, do you get a response, and does it >return quickly or take a significant amount of time? If the request >itself has problems, maybe CloudSolrClient is repeating it frequently >because it's not getting the info it's after. Can you share the full >log entry from solr.log for one of those requests? > >I try to keep an eye on things with CloudSolrClient, but I have very >limited experience with it. I cannot imagine that the behavior you're >seeing is normal. It sounds very wrong to me. > >Since I do not know all that much about how CloudSolrClient's background >threads work, I cannot say for sure whether it's a bug or a problem with >your setup. Can you try upgrading the Solr jars in your client app to >7.5.0 and see if that makes any difference? What version of Solr are >you running on the server side? > >> Our plan right now is to roll back to the basic HTTP client and pass >>all traffic through our load balancer, but would like to understand if >>this is an expected interaction for the Cloud Client, a misconfiguration >>on our end, or a bug > >At least you have that as an option! Some people might not be able to >do that. > >Thanks, >Shawn >
