Hi Steve,
Fluctuation is OK. 100% utilization for more than a moment is not :)
Not sure what tool(s) you use for monitoring your Solr servers, but look
under "JVM Pool Utilization" in SPM if you're using SPM.
Or this live demo of a Solr system:
* click on https://apps.sematext.com/demo to get in
On 12/9/2015 7:56 AM, Steven White wrote:
> Thanks Erick!! Your summary and the blog by Uwe (thank you too Uwe) are
> very helpful.
>
> A follow up question. I also noticed the "JVM-Memory" report off Solr's
> home page is fluctuating. I expect some fluctuation, but it kinda worries
> me when it
-
> From:Steven White
> Sent: Wednesday 9th December 2015 15:56
> To: solr-user@lucene.apache.org
> Subject: Re: Solr memory usage
>
> Thanks Erick!! Your summary and the blog by Uwe (thank you too Uwe) are
> very helpful.
>
> A follow up question. I also noticed
Thanks Erick!! Your summary and the blog by Uwe (thank you too Uwe) are
very helpful.
A follow up question. I also noticed the "JVM-Memory" report off Solr's
home page is fluctuating. I expect some fluctuation, but it kinda worries
me when it fluctuates up / down in a range of 4 GB and maybe mo
You're doing nothing wrong, that particular bit of advice has
always needed a bit of explanation.
Solr (well, actually Lucene) uses MMapDirectory for much of
the index structure which uses the OS memory rather than
the JVM heap. See Uwe's excellent:
http://blog.thetaphi.de/2012/07/use-lucenes-mma
And keep in mind that starving the OS of memory to
give it to the JVM is an anti-pattern, see Uwe's
excellent blog on MMapDirectory here:
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Best,
Erick
On Wed, Jan 7, 2015 at 5:55 AM, Shawn Heisey wrote:
> On 1/6/2015 1:10 PM
On 1/6/2015 1:10 PM, Abhishek Sharma wrote:
> *Q* - I am forced to set Java Xmx as high as 3.5g for my solr app.. If i
> keep this low, my CPU hits 100% and response time for indexing increases a
> lot.. And i have hit OOM Error as well when this value is low..
>
> Is this too high? If so, how can
Abhishek Sharma [abhishe...@unbxd.com] wrote:
> *Q* - I am forced to set Java Xmx as high as 3.5g for my solr app.. If i
> keep this low, my CPU hits 100% and response time for indexing increases a
> lot.. And i have hit OOM Error as well when this value is low..
[...]
> 2. Index Size - 2 g
>
On Wed, 2014-10-29 at 23:37 +0100, Will Martin wrote:
> This command only touches OS level caches that hold pages destined for (or
> not) the swap cache. Its use means that disk will be hit on future requests,
> but in many instances the pages were headed for ejection anyway.
>
> It does not have
On 10/29/2014 1:05 PM, Toke Eskildsen wrote:
> We did have some problems on a 256GB machine churning terabytes of data
> through 40 concurrent Tika processes and into Solr. After some days,
> performance got really bad. When we did a top, we noticed that most of the
> time was used in the kernel
;solr-user@lucene.apache.org'
Subject: RE: Solr Memory Usage
This command only touches OS level caches that hold pages destined for (or
not) the swap cache. Its use means that disk will be hit on future requests,
but in many instances the pages were headed for ejection anyway.
It does not have anyt
. from people who don't even research the matter.
-Original Message-
From: Toke Eskildsen [mailto:t...@statsbiblioteket.dk]
Sent: Wednesday, October 29, 2014 3:06 PM
To: solr-user@lucene.apache.org
Subject: RE: Solr Memory Usage
Vijay Kokatnur [kokatnur.vi...@gmail.com] wrote:
&g
Vijay Kokatnur [kokatnur.vi...@gmail.com] wrote:
> For the Solr Cloud setup, we are running a cron job with following command
> to clear out the inactive memory. It is working as expected. Even though
> the index size of Cloud is 146GB, the used memory is always below 55GB.
> Our response times
On 10/29/2014 11:43 AM, Vijay Kokatnur wrote:
> I am observing some weird behavior with how Solr is using memory. We are
> running both Solr and zookeeper on the same node. We tested memory
> settings on Solr Cloud Setup of 1 shard with 146GB index size, and 2 Shard
> Solr setup with 44GB index s
thanks!
On Tue, Mar 18, 2014 at 4:37 PM, Erick Erickson wrote:
> Avishai:
>
> It sounds like you already understand mmap. Even so you might be
> interested in this excellent writeup of MMapDirectory and Lucene by
> Uwe:
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
On 3/18/2014 8:37 AM, Erick Erickson wrote:
> It sounds like you already understand mmap. Even so you might be
> interested in this excellent writeup of MMapDirectory and Lucene by
> Uwe: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
There is some actual bad memory report
Avishai:
It sounds like you already understand mmap. Even so you might be
interested in this excellent writeup of MMapDirectory and Lucene by
Uwe: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Best,
Erick
On Tue, Mar 18, 2014 at 7:23 AM, Avishai Ish-Shalom
wrote:
> aha
aha! mmap explains it. thank you.
On Tue, Mar 18, 2014 at 3:11 PM, Shawn Heisey wrote:
> On 3/18/2014 5:30 AM, Avishai Ish-Shalom wrote:
> > My solr instances are configured with 10GB heap (Xmx) but linux shows
> > resident size of 16-20GB. even with thread stack and permgen taken into
> > acco
On 3/18/2014 5:30 AM, Avishai Ish-Shalom wrote:
> My solr instances are configured with 10GB heap (Xmx) but linux shows
> resident size of 16-20GB. even with thread stack and permgen taken into
> account i'm still far off from these numbers. Could it be that jvm IO
> buffers take so much space? doe
How large is your index on disk? Solr memory maps the index into
memory. Thus the virtual memory used will often be quite large. Your
numbers don't sound inconceivable.
A good reference point is Grant Ingersoll's blog post on searchhub:
http://searchhub.org/2011/09/14/estimating-memory-and-storage
More on this, I think I found something...
*Slave admin console- --> stats.jsp#cache**, FieldCache**
*
...
entries count: 22
entry#0 :
'MMapIndexInput(path="/home/agazzarini/solr-indexes/slave-data-dir/cbt/main/data/index/*_mp*.frq")'=>*'title_sort'*,class
...
entry#9 :
'MMapIndexInput(path=
Yeah, I sent a note to the web folks there about the images.
I'll leave the rest to people who really _understand_ all that stuff
On Thu, Sep 20, 2012 at 8:31 AM, Bernd Fehling
wrote:
> Hi Erik,
>
> thanks for the link.
> Now if we could see the images in that article that would be great
Hi Erik,
thanks for the link.
Now if we could see the images in that article that would be great :-)
By the way, one cause for the memory jumps was located as "killer search" from
a user.
The interesting part is that the verbose gc.log showed a "hiccup" in the GC.
Which means that during a GC r
Here's a wonderful writeup about GC and memory in Solr/Lucene:
http://searchhub.org/dev/2011/03/27/garbage-collection-bootcamp-1-0/
Best
Erick
On Thu, Sep 20, 2012 at 5:49 AM, Robert Muir wrote:
> On Thu, Sep 20, 2012 at 3:09 AM, Bernd Fehling
> wrote:
>
>> By the way while looking for upgradi
On Thu, Sep 20, 2012 at 3:09 AM, Bernd Fehling
wrote:
> By the way while looking for upgrading to JDK7, the release notes say under
> section
> "known issues" about the "PorterStemmer" bug:
> "...The recommended workaround is to specify -XX:-UseLoopPredicate on the
> command line."
> Is this st
That is the problem with a jvm, it is a virtual machine.
Ask 10 experts about a good jvm settings and you get 15 answers. May be a
tradeoff
of the flexibility of jvm's. There is always a right setting for any application
running on a jvm but you just have to find it.
How about a Solr Wiki page abo
I have used this setting to reduce gc pauses with CMS - java 6 u23
XX:+ParallelRefProcEnabled
With this setting, jvm does gc of weakrefs with multiple threads and pauses are
low.
Please use this option only when you have multiple cores.
For me, CMS gives better results
Sent from my iPhone
On
Ooh, that is a nasty one. Is this JDK 7 only or also in 6?
It looks like the "-XX:ConcGCThreads=1" option is a workaround, is that right?
We've had some 1.6 JVMs behave in the same way that bug describes, but I
haven't verified it is because of finalizer problems.
wunder
On Sep 19, 2012, at 5:
Two in one morning
The JVM bug I'm familiar with is here:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7112034
FWIW,
Erick
On Wed, Sep 19, 2012 at 8:20 AM, Shawn Heisey wrote:
> On 9/18/2012 9:29 PM, Lance Norskog wrote:
>>
>> There is a known JVM garbage collection bug that causes th
On 9/18/2012 9:29 PM, Lance Norskog wrote:
There is a known JVM garbage collection bug that causes this. It has to do with
reclaiming Weak references, I think in WeakHashMap. Concurrent garbage
collection collides with this bug and the result is that old field cache data
is retained after clos
he problem.)
- Original Message -
| From: "Bernd Fehling"
| To: solr-user@lucene.apache.org
| Sent: Tuesday, September 18, 2012 11:29:56 PM
| Subject: Re: SOLR memory usage jump in JVM
|
| Hi Lance,
|
| thanks for this hint. Something I also see, a sawtooth. This is
| coming from Eden
Hi Otis,
because I see this on my slave without replication there is no index file
change.
I have also tons of logged data to dig in :-)
I took dumps from different stages, fresh installed, after 5GB jump, after the
system was hanging right after replication,...
The last one was interesting when
- Original Message -
> | From: "Yonik Seeley"
> | To: solr-user@lucene.apache.org
> | Sent: Tuesday, September 18, 2012 7:38:41 AM
> | Subject: Re: SOLR memory usage jump in JVM
> |
> | On Tue, Sep 18, 2012 at 7:45 AM, Bernd Fehling
> | wrote:
> | > I use
Hi Bernd,
On Tue, Sep 18, 2012 at 3:09 AM, Bernd Fehling
wrote:
> Hi Otis,
>
> not really a problem because I have plenty of memory ;-)
> -Xmx25g -Xms25g -Xmn6g
Good.
> I'm just interested into this.
> Can you report similar jumps within JVM with your monitoring at sematext?
Yes. More importan
| Sent: Tuesday, September 18, 2012 7:38:41 AM
| Subject: Re: SOLR memory usage jump in JVM
|
| On Tue, Sep 18, 2012 at 7:45 AM, Bernd Fehling
| wrote:
| > I used GC in different situations and tried back and forth.
| > Yes, it reduces the used heap memory, but not by 5GB.
| > Even so t
On Tue, Sep 18, 2012 at 7:45 AM, Bernd Fehling
wrote:
> I used GC in different situations and tried back and forth.
> Yes, it reduces the used heap memory, but not by 5GB.
> Even so that GC from jconsole (or jvisualvm) is "Full GC".
Whatever "Full GC" means ;-)
In the past at least, I've found th
I used GC in different situations and tried back and forth.
Yes, it reduces the used heap memory, but not by 5GB.
Even so that GC from jconsole (or jvisualvm) is "Full GC".
But while you bring GC into this, there is another interesting thing.
- I have one slave running for a week which ends up aro
What happens if you attach jconsole (should ship with your SDK) and force a GC?
Does the extra 5G go away?
I'm wondering if you get a couple of warming searchers going simultaneously
and happened to measure after that.
Uwe has an interesting blog about memory, he recommends using as
little as pos
Hi Otis,
not really a problem because I have plenty of memory ;-)
-Xmx25g -Xms25g -Xmn6g
I'm just interested into this.
Can you report similar jumps within JVM with your monitoring at sematext?
Actually I would assume to see jumps of 0.5GB or even 1GB, but 5GB?
And what is the cause, a cache?
A
Hi Bernd,
But is this really (causing) a problem? What -Xmx are you using?
Otis
Search Analytics - http://sematext.com/search-analytics/index.html
Performance Monitoring - http://sematext.com/spm/index.html
On Tue, Sep 18, 2012 at 2:50 AM, Bernd Fehling
wrote:
> Hi list,
>
> while monitoring
Faceting and sorting are the two biggest places people get into trouble.
You've been asking questions about Solr Cloud, so I assume you're
working on a trunk release. Note that most everything people know
about memory consumption painfully gained over the years is...wrong
on trunk.
Or at least may
Mike
Actually i'm not able to tell you what each value stands for .. but what i can
tell you is, where the information is coming from.
The interface requests /admin/system which is using
https://svn.apache.org/repos/asf/lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/admin/SystemInf
Hi Bai,
Solr doesn't try to load the whole index into memory, no.
You can control how much memory Tomcat uses with -Xmx Java command line
parameter.
Â
Otis
Performance Monitoring SaaS for Solr -
http://sematext.com/spm/solr-performance-monitoring/index.html
>___
On Tue, 2010-12-14 at 06:07 +0100, Cameron Hurst wrote:
[Cameron expected 150MB overhead]
> As I start to index data and passing queries to the database I notice a
> steady rise in the RAM but it doesn't stop at 150MB. If I continue to
> reindex the exact same data set with no additional data ent
44 matches
Mail list logo
