In 6.5.1, the intra-cluster requests are POST, which makes them easy to distinguish in the request logs. Also, the intra-cluster requests go to a specific core instead of to the collection. So we use the request logs and grep out the GET lines.
We are considering fronting every Solr process with a local nginx server. That will allow us to limit concurrent connections. It will also give us a log of just the client requests. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Dec 5, 2017, at 4:25 AM, Matzdorf, Stefan, Springer SBM DE > <stefan.matzd...@springer.com> wrote: > > To be more precisely and provide some more details, i tried to simplify the > problem by using the Solr-examples that were delivered with the solr.... > So i started bin/solr -e cloud, using 2 nodes, 2 shards and replication of 2. > > To understand the following, it might be important to know, which ports are > used: > node 1: 8983 (leader for shard1 and shard2) > node 2: 7574 (no leader at all) > > > In this example i searched for 3 terms in the following order: first on node > 1 (8983 - leader) and then on node 2 (7574). > > Sample1 (q=test): > http://localhost:8983/solr/gettingstarted/select?indent=on&q=test&wt=json > > produced logs: > 1) [gettingstarted_shard1_replica_n1] webapp=/solr path=/select > params={df=_text_&distrib=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=http://127.0.1.1:7574/solr/gettingstarted_shard1_replica_n1/|http://127.0.1.1:8983/solr/gettingstarted_shard1_replica_n2/&rows=10&version=2&q=test&NOW=1512474523045&isShard=true&wt=javabin} > hits=0 status=0 QTime=1 > 2) [gettingstarted_shard2_replica_n4] webapp=/solr path=/select > params={df=_text_&distrib=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=http://127.0.1.1:7574/solr/gettingstarted_shard2_replica_n4/|http://127.0.1.1:8983/solr/gettingstarted_shard2_replica_n6/&rows=10&version=2&q=test&NOW=1512474523045&isShard=true&wt=javabin} > hits=0 status=0 QTime=1 > > > > http://localhost:7574/solr/gettingstarted/select?indent=on&q=test&wt=json > > produced logs: > 1) [gettingstarted_shard1_replica_n1] webapp=/solr path=/select > params={q=test&indent=on&wt=json} hits=0 status=0 QTime=17 > > ############################################################################################################################## > ############################################################################################################################## > > Sample2 (q=foo): > http://localhost:8983/solr/gettingstarted/select?indent=on&q=foo&wt=json > > produced logs: > 1) [gettingstarted_shard1_replica_n1] webapp=/solr path=/select > params={df=_text_&distrib=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=http://127.0.1.1:7574/solr/gettingstarted_shard1_replica_n1/|http://127.0.1.1:8983/solr/gettingstarted_shard1_replica_n2/&rows=10&version=2&q=foo&NOW=1512474569299&isShard=true&wt=javabin} > hits=0 status=0 QTime=0 > > > > http://localhost:7574/solr/gettingstarted/select?indent=on&q=foo&wt=json > > produced logs: > 1) [gettingstarted_shard2_replica_n4] webapp=/solr path=/select > params={q=foo&indent=on&wt=json} hits=0 status=0 QTime=13 > > ############################################################################################################################## > ############################################################################################################################## > > Sample3 (q=test) NOTE- its the same query as in sample1: > http://localhost:8983/solr/gettingstarted/select?indent=on&q=test&wt=json > > produced logs: > 1) [gettingstarted_shard2_replica_n4] webapp=/solr path=/select > params={df=_text_&distrib=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=http://127.0.1.1:7574/solr/gettingstarted_shard2_replica_n4/|http://127.0.1.1:8983/solr/gettingstarted_shard2_replica_n6/&rows=10&version=2&q=test&NOW=1512474643732&isShard=true&wt=javabin} > hits=0 status=0 QTime=0 > > > http://localhost:7574/solr/gettingstarted/select?indent=on&q=test&wt=json > > produced logs: > 1) [gettingstarted_shard2_replica_n4] webapp=/solr path=/select > params={df=_text_&distrib=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=http://127.0.1.1:7574/solr/gettingstarted_shard2_replica_n4/|http://127.0.1.1:8983/solr/gettingstarted_shard2_replica_n6/&rows=10&version=2&q=test&NOW=1512474627254&isShard=true&wt=javabin} > hits=0 status=0 QTime=0 > 2) [gettingstarted_shard1_replica_n1] webapp=/solr path=/select > params={q=test&indent=on&wt=json} hits=0 status=0 QTime=13 > > ############################################################################################################################## > ############################################################################################################################## > > Sample4 (q=baa): > http://localhost:8983/solr/gettingstarted/select?indent=on&q=baa&wt=json > > produced logs: > 1) [gettingstarted_shard2_replica_n4] webapp=/solr path=/select > params={df=_text_&distrib=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=http://127.0.1.1:7574/solr/gettingstarted_shard2_replica_n4/|http://127.0.1.1:8983/solr/gettingstarted_shard2_replica_n6/&rows=10&version=2&q=baa&NOW=1512474709460&isShard=true&wt=javabin} > hits=0 status=0 QTime=0 > > > http://localhost:7574/solr/gettingstarted/select?indent=on&q=baa&wt=json > > produced logs: > 1) [gettingstarted_shard1_replica_n1] webapp=/solr path=/select > params={q=baa&indent=on&wt=json} hits=0 status=0 QTime=12 > > > > > Sorry for this messy logs. > I'll try to sumarize.... > > For queries against the node 1, the leading node, i never got those "short > logs". just containing what i was querying. Instead i recieve logs containing > all these sharding information. Sometimes 2 equivalent ones (see sample 1) > and sometimes just one log (sample 2-4). Mentioned that i got different logs > for the same query/request (sample1 vs sample3). > > For queries against the node 2, not leading anything, i got those "short > logs" everytime. In addition to that, i also recievie sometimes an additional > log, containing these sharding information (sample 3). > > > My assumptions: > afaik, the queried shard is choosen randomly, that would explain why i got > different amount of logs for same queries/request. But what i everytime the > same, is that the "non-leading node" is the only one that returns the logs i > want. I tried some other example (that i do not write down here), and i saw > the same result. different amount of logs, that look spreaded randomly, but i > never got those "short-logs" from the leading node, just from the > "non-leading node". > > > So the idea of trying to find the difference between logs is not that > suitable, because sometimes they aren't present (facing the leading node). > > So the question still reminds: how could i log the incoming request in a > proper way? > > > I hope that my problem was clearly described, and someone got an idea whats > going on here.
