I've managed to replicate this issue with the 7.5.0 release as well by
starting up a single instance of solr in cloud mode (on windows) and
uploading the security.json file below to it.
After a short while, the "could not get tags from node..." messages start
coming through every 60 seconds. The accompanying logged error and
expecting stacktrace are also included below.
Is there a JIRA ticket for this issue (or a directly related one)? I
couldn't seem to find one.
Thanks,
Chris
*security.json:*
{
"authentication":{"blockUnknown":true,"class":"solr.BasicAuthPlugin",
"credentials":{
"solradmin":"...",
"solrreader":"...",
"solrwriter":"..."}
},
"authorization":{"class":"solr.RuleBasedAuthorizationPlugin",
"permissions":[
{"name":"read","role":"reader"},
{"name":"security-read","role":"reader"},
{"name":"schema-read","role":"reader"},
{"name":"config-read","role":"reader"},
{"name":"core-admin-read","role":"reader"},
{"name":"collection-admin-read","role":"reader"},
{"name":"update","role":"writer"},
{"name":"security-edit","role":"admin"},
{"name":"schema-edit","role":"admin"},
{"name":"config-edit","role":"admin"},
{"name":"core-admin-edit","role":"admin"},
{"name":"collection-admin-edit","role":"admin"},
{"name":"all","role":"admin"}],
"user-role":{
"solradmin":["reader","writer","admin"],
"solrreader":["reader"],
"solrwriter":["reader","writer"]}
}
}
*StackTrace:*
2018-10-31 16:20:01.994 WARN (MetricsHistoryHandler-12-thread-1) [ ]
o.a.s.c.s.i.SolrClientNodeStateProvider could not get tags from node
ip:8080_solr
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://ip:8080/solr: Expected mime type
application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 401 require authentication</title>
</head>
<body><h2>HTTP ERROR 401</h2>
<p>Problem accessing /solr/admin/metrics. Reason:
<pre> require authentication</pre></p>
</body>
</html>
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$ClientSnitchCtx.invoke(SolrClientNodeStateProvider.java:342)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchReplicaMetrics(SolrClientNodeStateProvider.java:195)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$AutoScalingSnitch.getRemoteInfo(SolrClientNodeStateProvider.java:241)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:58]
at
org.apache.solr.common.cloud.rule.ImplicitSnitch.getTags(ImplicitSnitch.java:76)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchTagValues(SolrClientNodeStateProvider.java:139)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.getNodeValues(SolrClientNodeStateProvider.java:128)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:58]
at
org.apache.solr.handler.admin.MetricsHistoryHandler.collectGlobalMetrics(MetricsHistoryHandler.java:498)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:55]
at
org.apache.solr.handler.admin.MetricsHistoryHandler.collectMetrics(MetricsHistoryHandler.java:371)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:55]
at
org.apache.solr.handler.admin.MetricsHistoryHandler.lambda$new$0(MetricsHistoryHandler.java:231)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:55]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_121]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[?:1.8.0_121]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_121]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[?:1.8.0_121]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_121]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
On Wed, Oct 17, 2018 at 8:32 AM Chris Ulicny <culicny@iq.media> wrote:
> Hi all,
>
> Recently in a 7.4.0 test cluster, we ran into SOLR-12814
> <https://issues.apache.org/jira/browse/SOLR-12814> which we fixed by
> slightly increasing the request header size. However, there were some other
> log messages along with the "URI size >8192" message which we thought were
> related, but have not abated since increasing the header size. A full
> shutdown of the solr processes and bringing them back up one at a time did
> not solve the issue.
>
> The overseer node seems to not be authenticating any of the requests to
> /solr/admin/metrics on any node (including itself). Every minute, there are
> two warning per node
>
> 10/17/2018, 7:53:45 AM WARN SolrClientNodeStateProvider could not
> get tags from node host1:port1_solr
> 10/17/2018, 7:53:45 AM WARN SolrClientNodeStateProvider could not
> get tags from node host1:port1_solr
> 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could not
> get tags from node host2:port2_solr
> 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could not
> get tags from node host2:port2_solr
> 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could not
> get tags from node host3:port3_solr
> 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could not
> get tags from node host3:port3_solr
>
> There are two slightly different stack traces that appear with each pair:
> https://pastebin.com/2Z1C5rXr
>
> The warning message possibly comes from
> solrj.impl.SolrClientNodeStateProvider.fetchMetrics which both of the
> attempted requests call in their stack trace.
>
> However, we already have a 7.4.0 production cluster running that also has
> security enabled with similar replica density where we have not seen this
> issue.
>
> *Test:*
> -- 10 collections (9 with 2 shards, 1 with 43 shards)
> -- replication factor of 2 for all collections
> -- 3 hosts with 40 or 41 replicas each
>
> *Production:*
> -- 9 collections with 14 shards
> -- replication factor of 2 for all collections
> -- 7 hosts with 36 replicas each
>
> I've enabled TRACE logging in our test environment on most options related
> to metrics and authentication. So far the only new message I've gotten is
> the challenge from the target server for the necessary credentials right
> before the warning and stack trace.
>
> 2018-10-17 12:20:46.368 DEBUG (MetricsHistoryHandler-8-thread-1) [ ]
> o.a.h.i.a.HttpAuthenticator Authentication required
> 2018-10-17 12:20:46.368 DEBUG (MetricsHistoryHandler-8-thread-1) [ ]
> o.a.h.i.a.HttpAuthenticator host3:port3 requested authentication
>
> I suspect the creation and balancing of the large collection on test might
> have something to do with it since the problem started happening after
> that.
>
> Are there any other specific log settings I should turn on that might
> produce some useful information?
>
> Thanks,
> Chris
>
