On 4/11/2019 1:03 PM, Hari Nakka wrote:
I mean the light weight processes (lwp) which were taking high cpu. I
pulled the actual threads taking high cpu.
full thread dump: *tdump.out*
linux lwps: *high-cpu.out*
top high cpu lwps mapped to thread nid: *high-cpu-dump.out
(included threads taking more than 50% virtual core cpu)*
I don't know how to read any of the files you shared and come to any
useful conclusions. I did notice that the thread names in the
"high-cpu.out" file are truncated to 15 characters, so it is not
possible to map them to the thread dump.
----------------
If I look at the "high-cpu-tdump.out" file and find threads in that dump
which are actually running (not waiting), here's what I see:
Threads that are doing lookups in one of Solr's caches, probably the
documentCache, but I am not sure:
qtp1650813924-32709
Not sure what the thread is doing, but it's running Lucene code:
qtp1650813924-54372
qtp1650813924-30839
qtp1650813924-44304
qtp1650813924-54344
qtp1650813924-40150
qtp1650813924-46829
qtp1650813924-41093
Threads that are doing a Pivot facet (Solr code):
qtp1650813924-43735
qtp1650813924-51360
Threads where Jetty is waiting for a request:
qtp1650813924-43734
qtp1650813924-52846
qtp1650813924-46575
qtp1650813924-55623
qtp1650813924-45996
qtp1650813924-48982
If the "jetty waiting" threads are causing the high CPU, then upgrading
Java should have taken care of it. If the Lucene or Pivot Facet threads
are to blame, then you may be facing a general performance problem,
which additional memory MIGHT help with.
----------------
What requests are you sending to Solr when high CPU happens? How long
does the high CPU last?
It might be useful to gather the screenshot described at the following
web page:
https://wiki.apache.org/solr/SolrPerformanceProblems#Asking_for_help_on_a_memory.2Fperformance_issue
One piece of information that could be useful and will not show up on
the screenshot is how many documents are being handled by that Solr
instance. You will need to look at all of the index cores in that
instance and add up the "maxDoc" numbers for each. The maxDoc number
includes deleted docs, which do affect performance.
Thanks,
Shawn
