Issues with Graphite reporter config
Hi, I am running Solrcloud version: 6.6.1 I have been trying to use graphite to report solr metrics and seem to get the below error while doing so in the solr logs: > java.lang.NullPointerException > at > com.codahale.metrics.graphite.PickledGraphite.pickleMetrics(PickledGraphite.java:313) > at > com.codahale.metrics.graphite.PickledGraphite.writeMetrics(PickledGraphite.java:255) > at > com.codahale.metrics.graphite.PickledGraphite.send(PickledGraphite.java:213) > at > com.codahale.metrics.graphite.GraphiteReporter.reportGauge(GraphiteReporter.java:345) > at > com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:243) > at > com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:251) > at > com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:174) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > 2017-11-07 15:28:47.543 WARN (metrics-graphite-reporter-3-thread-1) [ ] > c.c.m.g.GraphiteReporter Unable to report to Graphite > java.net.SocketException: Socket closed > at > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118) > at java.net.SocketOutputStream.write(SocketOutputStream.java:143) > at > com.codahale.metrics.graphite.PickledGraphite.writeMetrics(PickledGraphite.java:261) > at > com.codahale.metrics.graphite.PickledGraphite.send(PickledGraphite.java:213) > at > com.codahale.metrics.graphite.GraphiteReporter.sendIfEnabled(GraphiteReporter.java:328) > at > com.codahale.metrics.graphite.GraphiteReporter.reportTimer(GraphiteReporter.java:288) > at > com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:259) > at > com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:251) > at > com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:174) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > 2017-11-07 15:28:47.543 ERROR (metrics-graphite-reporter-1-thread-1) [ ] > c.c.m.ScheduledReporter Exception thrown from GraphiteReporter#report. > Exception was suppressed. > java.lang.NullPointerException > at java.util.LinkedList$ListItr.next(LinkedList.java:893) > at > com.codahale.metrics.graphite.PickledGraphite.pickleMetrics(PickledGraphite.java:305) > at > com.codahale.metrics.graphite.PickledGraphite.writeMetrics(PickledGraphite.java:255) > at > com.codahale.metrics.graphite.PickledGraphite.send(PickledGraphite.java:213) > at > com.codahale.metrics.graphite.GraphiteReporter.sendIfEnabled(GraphiteReporter.java:328) > at > com.codahale.metrics.graphite.GraphiteReporter.reportMetered(GraphiteReporter.java:304) > at > com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:255) > at > com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:251) > at > com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:174) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > (END) Kindly let me know if this needs to be reported as a bug.
Re: Issues with Graphite reporter config
Thank you Cassandra. Does seem like a thread unsafe operation issue. But what confuses me is the error occurs every time and only occurs when I have multiple metrics group configured. Also the exception is null pointer on the linked list instead of already connected exception Regards Sudershan Madhavan On 7 Nov 2017 6:18 pm, "Cassandra Targett" wrote: > I believe this is https://issues.apache.org/jira/browse/SOLR-11413, > which has a fix already slated for Solr 7.2. > > On Tue, Nov 7, 2017 at 10:44 AM, sudershan madhavan > wrote: > > Hi, > > I am running Solrcloud version: 6.6.1 > > I have been trying to use graphite to report solr metrics and seem to get > > the below error while doing so in the solr logs: > >> > >> java.lang.NullPointerException > >> at > >> com.codahale.metrics.graphite.PickledGraphite.pickleMetrics( > PickledGraphite.java:313) > >> at > >> com.codahale.metrics.graphite.PickledGraphite.writeMetrics( > PickledGraphite.java:255) > >> at > >> com.codahale.metrics.graphite.PickledGraphite.send( > PickledGraphite.java:213) > >> at > >> com.codahale.metrics.graphite.GraphiteReporter.reportGauge( > GraphiteReporter.java:345) > >> at > >> com.codahale.metrics.graphite.GraphiteReporter.report( > GraphiteReporter.java:243) > >> at > >> com.codahale.metrics.ScheduledReporter.report( > ScheduledReporter.java:251) > >> at > >> com.codahale.metrics.ScheduledReporter$1.run( > ScheduledReporter.java:174) > >> at > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > >> at > >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > >> at > >> java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > >> at > >> java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > >> at > >> java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > >> at > >> java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > >> at java.lang.Thread.run(Thread.java:745) > >> 2017-11-07 15:28:47.543 WARN (metrics-graphite-reporter-3-thread-1) > [ ] > >> c.c.m.g.GraphiteReporter Unable to report to Graphite > >> java.net.SocketException: Socket closed > >> at > >> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118) > >> at java.net.SocketOutputStream.write(SocketOutputStream.java: > 143) > >> at > >> com.codahale.metrics.graphite.PickledGraphite.writeMetrics( > PickledGraphite.java:261) > >> at > >> com.codahale.metrics.graphite.PickledGraphite.send( > PickledGraphite.java:213) > >> at > >> com.codahale.metrics.graphite.GraphiteReporter.sendIfEnabled( > GraphiteReporter.java:328) > >> at > >> com.codahale.metrics.graphite.GraphiteReporter.reportTimer( > GraphiteReporter.java:288) > >> at > >> com.codahale.metrics.graphite.GraphiteReporter.report( > GraphiteReporter.java:259) > >> at > >> com.codahale.metrics.ScheduledReporter.report( > ScheduledReporter.java:251) > >> at > >> com.codahale.metrics.ScheduledReporter$1.run( > ScheduledReporter.java:174) > >> at > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > >> at > >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > >> at > >> java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > >> at > >> java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > >> at > >> java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > >> at > >> java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > >> at java.lang.Thread.run(Thread.java:745) > >> 2017-11-07 15:28:47.543 ERROR (metrics-graphite-reporter-1-thread-1) > [ ] > >> c.c.m.ScheduledReporter Exception thrown from GraphiteReporter#report. > >> Exception was suppressed. > >> java.lang.NullPointerException > >>
Re: Issues with Graphite reporter config
Can someone confirm if this needs to be reported as a bug? As the exception in the patch seems to be different than that of the the SOLR-11413? Also the issue is not sporadic but occurs every time the Graphite Reporter is invoked for multiple metrics. Regards Sudershan Madahavan On Tue, Nov 7, 2017 at 7:10 PM, sudershan madhavan < sudershan.madha...@gmail.com> wrote: > Thank you Cassandra. Does seem like a thread unsafe operation issue. But > what confuses me is the error occurs every time and only occurs when I have > multiple metrics group configured. Also the exception is null pointer on > the linked list instead of already connected exception > > Regards > Sudershan Madhavan > > On 7 Nov 2017 6:18 pm, "Cassandra Targett" wrote: > >> I believe this is https://issues.apache.org/jira/browse/SOLR-11413, >> which has a fix already slated for Solr 7.2. >> >> On Tue, Nov 7, 2017 at 10:44 AM, sudershan madhavan >> wrote: >> > Hi, >> > I am running Solrcloud version: 6.6.1 >> > I have been trying to use graphite to report solr metrics and seem to >> get >> > the below error while doing so in the solr logs: >> >> >> >> java.lang.NullPointerException >> >> at >> >> com.codahale.metrics.graphite.PickledGraphite.pickleMetrics( >> PickledGraphite.java:313) >> >> at >> >> com.codahale.metrics.graphite.PickledGraphite.writeMetrics(P >> ickledGraphite.java:255) >> >> at >> >> com.codahale.metrics.graphite.PickledGraphite.send(PickledGr >> aphite.java:213) >> >> at >> >> com.codahale.metrics.graphite.GraphiteReporter.reportGauge(G >> raphiteReporter.java:345) >> >> at >> >> com.codahale.metrics.graphite.GraphiteReporter.report(Graphi >> teReporter.java:243) >> >> at >> >> com.codahale.metrics.ScheduledReporter.report(ScheduledRepor >> ter.java:251) >> >> at >> >> com.codahale.metrics.ScheduledReporter$1.run(ScheduledReport >> er.java:174) >> >> at >> >> java.util.concurrent.Executors$RunnableAdapter.call( >> Executors.java:511) >> >> at >> >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) >> >> at >> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu >> tureTask.access$301(ScheduledThreadPoolExecutor.java:180) >> >> at >> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu >> tureTask.run(ScheduledThreadPoolExecutor.java:294) >> >> at >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> Executor.java:1142) >> >> at >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> lExecutor.java:617) >> >> at java.lang.Thread.run(Thread.java:745) >> >> 2017-11-07 15:28:47.543 WARN (metrics-graphite-reporter-3-thread-1) >> [ ] >> >> c.c.m.g.GraphiteReporter Unable to report to Graphite >> >> java.net.SocketException: Socket closed >> >> at >> >> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118) >> >> at java.net.SocketOutputStream.wr >> ite(SocketOutputStream.java:143) >> >> at >> >> com.codahale.metrics.graphite.PickledGraphite.writeMetrics(P >> ickledGraphite.java:261) >> >> at >> >> com.codahale.metrics.graphite.PickledGraphite.send(PickledGr >> aphite.java:213) >> >> at >> >> com.codahale.metrics.graphite.GraphiteReporter.sendIfEnabled >> (GraphiteReporter.java:328) >> >> at >> >> com.codahale.metrics.graphite.GraphiteReporter.reportTimer(G >> raphiteReporter.java:288) >> >> at >> >> com.codahale.metrics.graphite.GraphiteReporter.report(Graphi >> teReporter.java:259) >> >> at >> >> com.codahale.metrics.ScheduledReporter.report(ScheduledRepor >> ter.java:251) >> >> at >> >> com.codahale.metrics.ScheduledReporter$1.run(ScheduledReport >> er.java:174) >> >> at >> >> java.util.concurrent.Executors$RunnableAdapter.call( >> Executors.java:511) >> >> at >> >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) >> >> at >> >> java.util.concurrent.ScheduledThreadPoolExecutor$S
Query on recovering from Shard Loss
Hi, Good day. I have a solrcloud cluster which has a collection with RF=2 and NumShards=3 on 6 Nodes. We want to test how to recover from unexpected situations like shard loss. So we will probably execute an rm -rf on the solr data directory on one of the replica or master. Now the question is, how will this shredded node recover from the shard loss? Are manual steps required(if yes, then what needs to be done), or will it automatically recover from the replica? Regards, SUDERSHAN MADHAVAN | Digital River World Payments | Senior Database Administrator m: +46 733 312525 | smadha...@digitalriver.com<mailto:smadha...@digitalriver.com> | digitalriver.com<http://www.digitalriver.com/> Managing Director: Michael Roos | Registered Office: Stockholm | Company Number: 556548-0026 Textilgatan 31, SE-120 30 Stockholm, Sweden [In-2C-14px]<https://www.linkedin.com/company-beta/2991085/> Follow us on LinkedIn<https://www.linkedin.com/company-beta/2991085/>
