On Mon, Jul 20, 2009 at 12:31 PM, Ryan McKinley<ryan...@gmail.com> wrote: > > On Jul 20, 2009, at 9:16 AM, Edward Capriolo wrote: > >> On Mon, Jul 20, 2009 at 11:53 AM, Ryan McKinley<ryan...@gmail.com> wrote: >>> >>> On Jul 20, 2009, at 8:47 AM, Edward Capriolo wrote: >>> >>>> Hey all, >>>> >>>> We have several deployments of Solr across our enterprise. Our largest >>>> one is a several GB and when enough documents are added an OOM >>>> exception is occurring. >>>> >>>> To debug this problem I have enable JMX. My goal is to write some >>>> cacti templates similar to the ones I have done for hadoop. >>>> http://www.jointhegrid.com/hadoop/. The only cacti template for solr I >>>> have found is old, broken and is using curl and PHP to try and read >>>> the values off the web interface. I have a few general >>>> questions/comments and also would like to know how others are dealing >>>> with this. >>>> >>>> 1) SNMP has counters/gauges. With JMX it is hard to know what a >>>> variable is without watching it for a while. Some fields are obvious, >>>> (total_x) (cumulative_x) it is worth wild to add some notes in the >>>> MBEAN info to say "works like counter" "works like gauge". This way a >>>> "network engineer" like me does not have to go code surfing to figure >>>> out how to graph them. >>>> >>>> Has anyone written up a list of what the attributes are, types, and >>>> what they mean? >>>> >>>> 2) The values that are not counter style I am assuming are sampled, >>>> what is the sampling rate and is it adjustable? >>>> >>>> Any tips are helpful. Thank you, >>> >>> Check: >>> >>> http://svn.apache.org/repos/asf/lucene/solr/trunk/src/java/org/apache/solr/handler/RequestHandlerBase.java >>> >>> For cacti, you should probably ignore the two 'rate' based calculations >>> as >>> they are just derivatives: >>> lst.add("avgTimePerRequest", (float) totalTime / (float) >>> this.numRequests); >>> lst.add("avgRequestsPerSecond", (float) numRequests*1000 / >>> (float)(System.currentTimeMillis()-handlerStart)); >>> >>> >>> >>> >> Thanks Ryan, >> >> Actually, I typically graph the derivatives directly. As graphing a >> derivative is usually easier then writing cacti CDEF's which can be >> fickle when exporting the templates between versions, but I see your >> point. >> >> However, do you see the point I was getting at? Without MBEAN info >> stating that these values are derivatives I have to dig through source >> code. It is not a complaint, just a note that their seems to be so >> much work on JMX counters, but just a few word description in the >> MBEAN info would eliminate the need to dig through the source tree >> when it actually comes time to for someone to render these counters. >> > > no doubt -- i am unfamiliar with how these get passed to JMX (or where extra > docs would be helpful) -- feel free to submit a patch that adds this info, > perhaps to the wiki? javadoc? this way it will be easier for the next guy > > >> Also one more question on my mind, how are the JMX objects effected by >> a multi core deployment. Does each core have its own objects or are >> they shared? >> > > each core/handler gets its own object -- they are not shared across cores. > > >> Thank you, >> Edward > > Ryan,
After adding a <jmx> in the solconfig.xml and setting some command line -D options. JMX is available on a tcp port. From that point a java tools can read the value directly. I have console programs that output the value so cacti 'data input methods' can read the data in. I just subclass this ...http://www.jointhegrid.com/svn/hadoop-cacti-jtg/trunk/src/com/jointhegrid/hadoopjmx/JMXBase.java The jconsole GUI tool then allows you to browse the JMX tree. If the MBEAN info is filled in it is displayed directly to the user. Patching the attributes to have a more verbose descriptions would be very helpful. I will open a Jira for that. Thanks, Edward
