Vadim, Thanks for the suggestion.
For the test case, of spinning up solr 7.5 locally with the security.json upload, I went ahead and tried to change both the request and response header sizes as you mentioned and restarting. Unfortunately, it does not seem to fix the issue for me. The error that is being displayed in the logs is still the 401 error when accessing /solr/admin/metrics. To me, it seems that this internal request is just ignoring the security altogether. On Wed, Oct 31, 2018 at 4:54 PM Vadim Ivanov < vadim.iva...@spb.ntk-intourist.ru> wrote: > Hi, Chris > I had the same messages in solr log while testing 7.4 and 7.5 > The only remedy I've found - increasing header size: > /opt/solr/server/etc/jetty.xml > <Set name="requestHeaderSize"><Property > name="solr.jetty.request.header.size" default="81920" /></Set> > <Set name="responseHeaderSize"><Property > name="solr.jetty.response.header.size" default="81920" /></Set> > After solr restart - no more annoying messages > > > -----Original Message----- > > From: Chris Ulicny [mailto:culicny@iq.media] > > Sent: Wednesday, October 31, 2018 7:40 PM > > To: solr-user > > Subject: Re: Overseer could not get tags > > > > I've managed to replicate this issue with the 7.5.0 release as well by > > starting up a single instance of solr in cloud mode (on windows) and > > uploading the security.json file below to it. > > > > After a short while, the "could not get tags from node..." messages start > > coming through every 60 seconds. The accompanying logged error and > > expecting stacktrace are also included below. > > > > Is there a JIRA ticket for this issue (or a directly related one)? I > > couldn't seem to find one. > > > > Thanks, > > Chris > > > > *security.json:* > > { > > "authentication":{"blockUnknown":true,"class":"solr.BasicAuthPlugin", > > "credentials":{ > > "solradmin":"...", > > "solrreader":"...", > > "solrwriter":"..."} > > }, > > "authorization":{"class":"solr.RuleBasedAuthorizationPlugin", > > "permissions":[ > > {"name":"read","role":"reader"}, > > {"name":"security-read","role":"reader"}, > > {"name":"schema-read","role":"reader"}, > > {"name":"config-read","role":"reader"}, > > {"name":"core-admin-read","role":"reader"}, > > {"name":"collection-admin-read","role":"reader"}, > > {"name":"update","role":"writer"}, > > {"name":"security-edit","role":"admin"}, > > {"name":"schema-edit","role":"admin"}, > > {"name":"config-edit","role":"admin"}, > > {"name":"core-admin-edit","role":"admin"}, > > {"name":"collection-admin-edit","role":"admin"}, > > {"name":"all","role":"admin"}], > > "user-role":{ > > "solradmin":["reader","writer","admin"], > > "solrreader":["reader"], > > "solrwriter":["reader","writer"]} > > } > > } > > > > *StackTrace:* > > 2018-10-31 16:20:01.994 WARN (MetricsHistoryHandler-12-thread-1) [ ] > > o.a.s.c.s.i.SolrClientNodeStateProvider could not get tags from node > > ip:8080_solr > > org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: > Error > > from server at http://ip:8080/solr: Expected mime type > > application/octet-stream but got text/html. <html> > > <head> > > <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> > > <title>Error 401 require authentication</title> > > </head> > > <body><h2>HTTP ERROR 401</h2> > > <p>Problem accessing /solr/admin/metrics. Reason: > > <pre> require authentication</pre></p> > > </body> > > </html> > > > > at > > > org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.ja > > va:607) > > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > > jimczi - 2018-09-18 13:07:58] > > at > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255) > > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > > jimczi - 2018-09-18 13:07:58] > > at > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244) > > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > > jimczi - 2018-09-18 13:07:58] > > at > > org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219) > > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > > jimczi - 2018-09-18 13:07:58] > > at > > > org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$ClientSnitchCtx.in > > voke(SolrClientNodeStateProvider.java:342) > > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > > jimczi - 2018-09-18 13:07:58] > > at > > > org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchReplicaMetri > > cs(SolrClientNodeStateProvider.java:195) > > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > > jimczi - 2018-09-18 13:07:58] > > at > > > org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$AutoScalingSnitc > > h.getRemoteInfo(SolrClientNodeStateProvider.java:241) > > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > > jimczi - 2018-09-18 13:07:58] > > at > > > org.apache.solr.common.cloud.rule.ImplicitSnitch.getTags(ImplicitSnitch.java:7 > > 6) > > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > > jimczi - 2018-09-18 13:07:58] > > at > > > org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchTagValues(S > > olrClientNodeStateProvider.java:139) > > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > > jimczi - 2018-09-18 13:07:58] > > at > > > org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.getNodeValues(S > > olrClientNodeStateProvider.java:128) > > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > > jimczi - 2018-09-18 13:07:58] > > at > > > org.apache.solr.handler.admin.MetricsHistoryHandler.collectGlobalMetrics(Me > > tricsHistoryHandler.java:498) > > ~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > > jimczi - 2018-09-18 13:07:55] > > at > > > org.apache.solr.handler.admin.MetricsHistoryHandler.collectMetrics(MetricsHi > > storyHandler.java:371) > > ~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > > jimczi - 2018-09-18 13:07:55] > > at > > org.apache.solr.handler.admin.MetricsHistoryHandler.lambda$new$0(Metrics > > HistoryHandler.java:231) > > ~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > > jimczi - 2018-09-18 13:07:55] > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > [?:1.8.0_121] > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > > [?:1.8.0_121] > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.acces > > s$301(ScheduledThreadPoolExecutor.java:180) > > [?:1.8.0_121] > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(S > > cheduledThreadPoolExecutor.java:294) > > [?:1.8.0_121] > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: > > 1142) > > [?:1.8.0_121] > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java > > :617) > > [?:1.8.0_121] > > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] > > > > > > On Wed, Oct 17, 2018 at 8:32 AM Chris Ulicny <culicny@iq.media> wrote: > > > > > Hi all, > > > > > > Recently in a 7.4.0 test cluster, we ran into SOLR-12814 > > > <https://issues.apache.org/jira/browse/SOLR-12814> which we fixed by > > > slightly increasing the request header size. However, there were some > other > > > log messages along with the "URI size >8192" message which we thought > > were > > > related, but have not abated since increasing the header size. A full > > > shutdown of the solr processes and bringing them back up one at a time > did > > > not solve the issue. > > > > > > The overseer node seems to not be authenticating any of the requests to > > > /solr/admin/metrics on any node (including itself). Every minute, > there are > > > two warning per node > > > > > > 10/17/2018, 7:53:45 AM WARN SolrClientNodeStateProvider could > not > > > get tags from node host1:port1_solr > > > 10/17/2018, 7:53:45 AM WARN SolrClientNodeStateProvider could > not > > > get tags from node host1:port1_solr > > > 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could > not > > > get tags from node host2:port2_solr > > > 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could > not > > > get tags from node host2:port2_solr > > > 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could > not > > > get tags from node host3:port3_solr > > > 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could > not > > > get tags from node host3:port3_solr > > > > > > There are two slightly different stack traces that appear with each > pair: > > > https://pastebin.com/2Z1C5rXr > > > > > > The warning message possibly comes from > > > solrj.impl.SolrClientNodeStateProvider.fetchMetrics which both of the > > > attempted requests call in their stack trace. > > > > > > However, we already have a 7.4.0 production cluster running that also > has > > > security enabled with similar replica density where we have not seen > this > > > issue. > > > > > > *Test:* > > > -- 10 collections (9 with 2 shards, 1 with 43 shards) > > > -- replication factor of 2 for all collections > > > -- 3 hosts with 40 or 41 replicas each > > > > > > *Production:* > > > -- 9 collections with 14 shards > > > -- replication factor of 2 for all collections > > > -- 7 hosts with 36 replicas each > > > > > > I've enabled TRACE logging in our test environment on most options > related > > > to metrics and authentication. So far the only new message I've gotten > is > > > the challenge from the target server for the necessary credentials > right > > > before the warning and stack trace. > > > > > > 2018-10-17 12:20:46.368 DEBUG (MetricsHistoryHandler-8-thread-1) [ ] > > > o.a.h.i.a.HttpAuthenticator Authentication required > > > 2018-10-17 12:20:46.368 DEBUG (MetricsHistoryHandler-8-thread-1) [ ] > > > o.a.h.i.a.HttpAuthenticator host3:port3 requested authentication > > > > > > I suspect the creation and balancing of the large collection on test > might > > > have something to do with it since the problem started happening after > > > that. > > > > > > Are there any other specific log settings I should turn on that might > > > produce some useful information? > > > > > > Thanks, > > > Chris > > > > >
