_Have_ they crashed due to OOMs? It’s quite normal for Java to create
a sawtooth pattern of memory consumption. If you attach, say, jconsole
to the running Solr and hit the GC button, does the memory drop back?
To answer your question, though, no there’s no reason memory should creep.
That said, t
On 9/7/2018 8:39 AM, Pavel Micka wrote:
I found on wiki (https://wiki.apache.org/solr/SolrPerformanceProblems#RAM) that
optimal amount of RAM for SOLR is equal to index size. This is lets say the
ideal case to have everything in memory.
I wrote that page.
We plan to have small installation w
)
On Aug 28, 2017, at 8:48 AM, Markus Jelsma
wrote:
It is, unfortunately, not committed for 6.7.
-Original message-
From:Markus Jelsma
Sent: Monday 28th August 2017 17:46
To: solr-user@lucene.apache.org
Subject: RE: Solr memory leak
See https://issues.apache.org/jira
gt;>
>>>> It is, unfortunately, not committed for 6.7.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> -Original message-
>>>>>
>>>>> From:Markus Jelsma
>>>>> Sent: Mon
apache.org
Subject: RE: Solr memory leak
See https://issues.apache.org/jira/browse/SOLR-10506
Fixed for 7.0
Markus
-Original message-
From:Hendrik Haddorp
Sent: Monday 28th August 2017 17:42
To: solr-user@lucene.apache.org
Subject: Solr memory leak
Hi,
we noticed that triggering
solr-user@lucene.apache.org
Subject: RE: Solr memory leak
See https://issues.apache.org/jira/browse/SOLR-10506
Fixed for 7.0
Markus
-Original message-
From:Hendrik Haddorp
Sent: Monday 28th August 2017 17:42
To: solr-user@lucene.apache.org
Subject: Solr memory leak
Hi,
we noticed
blog)
>
>
>> On Aug 28, 2017, at 8:48 AM, Markus Jelsma
>> wrote:
>>
>> It is, unfortunately, not committed for 6.7.
>>
>>
>>
>>
>>
>> -Original message-
>>> From:Markus Jelsma
>>> Sent: Monday 28t
> -Original message-
>> From:Markus Jelsma
>> Sent: Monday 28th August 2017 17:46
>> To: solr-user@lucene.apache.org
>> Subject: RE: Solr memory leak
>>
>> See https://issues.apache.org/jira/browse/SOLR-10506
>> Fixed for 7.0
>>
>> Markus
>>
It is, unfortunately, not committed for 6.7.
-Original message-
> From:Markus Jelsma
> Sent: Monday 28th August 2017 17:46
> To: solr-user@lucene.apache.org
> Subject: RE: Solr memory leak
>
> See https://issues.apache.org/jira/browse/SOLR-10506
> Fixe
See https://issues.apache.org/jira/browse/SOLR-10506
Fixed for 7.0
Markus
-Original message-
> From:Hendrik Haddorp
> Sent: Monday 28th August 2017 17:42
> To: solr-user@lucene.apache.org
> Subject: Solr memory leak
>
> Hi,
>
> we noticed that triggering collection reloads on many
Hi Steve,
Fluctuation is OK. 100% utilization for more than a moment is not :)
Not sure what tool(s) you use for monitoring your Solr servers, but look
under "JVM Pool Utilization" in SPM if you're using SPM.
Or this live demo of a Solr system:
* click on https://apps.sematext.com/demo to get in
On 12/9/2015 7:56 AM, Steven White wrote:
> Thanks Erick!! Your summary and the blog by Uwe (thank you too Uwe) are
> very helpful.
>
> A follow up question. I also noticed the "JVM-Memory" report off Solr's
> home page is fluctuating. I expect some fluctuation, but it kinda worries
> me when it
-
> From:Steven White
> Sent: Wednesday 9th December 2015 15:56
> To: solr-user@lucene.apache.org
> Subject: Re: Solr memory usage
>
> Thanks Erick!! Your summary and the blog by Uwe (thank you too Uwe) are
> very helpful.
>
> A follow up question. I also noticed
Thanks Erick!! Your summary and the blog by Uwe (thank you too Uwe) are
very helpful.
A follow up question. I also noticed the "JVM-Memory" report off Solr's
home page is fluctuating. I expect some fluctuation, but it kinda worries
me when it fluctuates up / down in a range of 4 GB and maybe mo
You're doing nothing wrong, that particular bit of advice has
always needed a bit of explanation.
Solr (well, actually Lucene) uses MMapDirectory for much of
the index structure which uses the OS memory rather than
the JVM heap. See Uwe's excellent:
http://blog.thetaphi.de/2012/07/use-lucenes-mma
And keep in mind that starving the OS of memory to
give it to the JVM is an anti-pattern, see Uwe's
excellent blog on MMapDirectory here:
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Best,
Erick
On Wed, Jan 7, 2015 at 5:55 AM, Shawn Heisey wrote:
> On 1/6/2015 1:10 PM
On 1/6/2015 1:10 PM, Abhishek Sharma wrote:
> *Q* - I am forced to set Java Xmx as high as 3.5g for my solr app.. If i
> keep this low, my CPU hits 100% and response time for indexing increases a
> lot.. And i have hit OOM Error as well when this value is low..
>
> Is this too high? If so, how can
Abhishek Sharma [abhishe...@unbxd.com] wrote:
> *Q* - I am forced to set Java Xmx as high as 3.5g for my solr app.. If i
> keep this low, my CPU hits 100% and response time for indexing increases a
> lot.. And i have hit OOM Error as well when this value is low..
[...]
> 2. Index Size - 2 g
>
On Wed, 2014-10-29 at 23:37 +0100, Will Martin wrote:
> This command only touches OS level caches that hold pages destined for (or
> not) the swap cache. Its use means that disk will be hit on future requests,
> but in many instances the pages were headed for ejection anyway.
>
> It does not have
On 10/29/2014 1:05 PM, Toke Eskildsen wrote:
> We did have some problems on a 256GB machine churning terabytes of data
> through 40 concurrent Tika processes and into Solr. After some days,
> performance got really bad. When we did a top, we noticed that most of the
> time was used in the kernel
;solr-user@lucene.apache.org'
Subject: RE: Solr Memory Usage
This command only touches OS level caches that hold pages destined for (or
not) the swap cache. Its use means that disk will be hit on future requests,
but in many instances the pages were headed for ejection anyway.
It does not have anyt
. from people who don't even research the matter.
-Original Message-
From: Toke Eskildsen [mailto:t...@statsbiblioteket.dk]
Sent: Wednesday, October 29, 2014 3:06 PM
To: solr-user@lucene.apache.org
Subject: RE: Solr Memory Usage
Vijay Kokatnur [kokatnur.vi...@gmail.com] wrote:
&g
Vijay Kokatnur [kokatnur.vi...@gmail.com] wrote:
> For the Solr Cloud setup, we are running a cron job with following command
> to clear out the inactive memory. It is working as expected. Even though
> the index size of Cloud is 146GB, the used memory is always below 55GB.
> Our response times
On 10/29/2014 11:43 AM, Vijay Kokatnur wrote:
> I am observing some weird behavior with how Solr is using memory. We are
> running both Solr and zookeeper on the same node. We tested memory
> settings on Solr Cloud Setup of 1 shard with 146GB index size, and 2 Shard
> Solr setup with 44GB index s
On 8/1/2014 3:17 PM, Ethan wrote:
> Our SolrCloud setup : 3 Nodes with Zookeeper, 2 running SolrCloud.
>
> Current dataset size is 97GB, JVM is 10GB, but 6GB is used(for less garbage
> collection time). RAM is 96GB,
>
> Our softcommit is set to 2secs and hardcommit is set to 1 hour.
>
> We are sud
4.5.0.
We are trying to free memory by deleting data from 2010. But that hasn't
helped so far.
On Fri, Aug 1, 2014 at 3:13 PM, Otis Gospodnetic wrote:
> Which version of Solr?
>
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sema
Which version of Solr?
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Fri, Aug 1, 2014 at 11:17 PM, Ethan wrote:
> Our SolrCloud setup : 3 Nodes with Zookeeper, 2 running SolrCloud.
>
> Current dataset size is 97GB, JVM
thanks!
On Tue, Mar 18, 2014 at 4:37 PM, Erick Erickson wrote:
> Avishai:
>
> It sounds like you already understand mmap. Even so you might be
> interested in this excellent writeup of MMapDirectory and Lucene by
> Uwe:
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
On 3/18/2014 8:37 AM, Erick Erickson wrote:
> It sounds like you already understand mmap. Even so you might be
> interested in this excellent writeup of MMapDirectory and Lucene by
> Uwe: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
There is some actual bad memory report
Avishai:
It sounds like you already understand mmap. Even so you might be
interested in this excellent writeup of MMapDirectory and Lucene by
Uwe: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Best,
Erick
On Tue, Mar 18, 2014 at 7:23 AM, Avishai Ish-Shalom
wrote:
> aha
aha! mmap explains it. thank you.
On Tue, Mar 18, 2014 at 3:11 PM, Shawn Heisey wrote:
> On 3/18/2014 5:30 AM, Avishai Ish-Shalom wrote:
> > My solr instances are configured with 10GB heap (Xmx) but linux shows
> > resident size of 16-20GB. even with thread stack and permgen taken into
> > acco
On 3/18/2014 5:30 AM, Avishai Ish-Shalom wrote:
> My solr instances are configured with 10GB heap (Xmx) but linux shows
> resident size of 16-20GB. even with thread stack and permgen taken into
> account i'm still far off from these numbers. Could it be that jvm IO
> buffers take so much space? doe
How large is your index on disk? Solr memory maps the index into
memory. Thus the virtual memory used will often be quite large. Your
numbers don't sound inconceivable.
A good reference point is Grant Ingersoll's blog post on searchhub:
http://searchhub.org/2011/09/14/estimating-memory-and-storage
More on this, I think I found something...
*Slave admin console- --> stats.jsp#cache**, FieldCache**
*
...
entries count: 22
entry#0 :
'MMapIndexInput(path="/home/agazzarini/solr-indexes/slave-data-dir/cbt/main/data/index/*_mp*.frq")'=>*'title_sort'*,class
...
entry#9 :
'MMapIndexInput(path=
any input on this?
thanks
Jie
--
View this message in context:
http://lucene.472066.n3.nabble.com/solr-memory-leak-prevent-tomcat-shutdown-tp4014788p4015265.html
Sent from the Solr - User mailing list archive at Nabble.com.
found a solr/lucene bug : TimeLimitingCollector starts thread in static {}
with no way to stop them
https://issues.apache.org/jira/browse/LUCENE-2822
is this the same issue? it is fixed in Luence 3.5. but I am using solr3.5
with lucene 2.9.3 (matched lucene version).
can anyone shed some light
by the way, I am running tomcat 6, solr 3.5 on redhat 2.6.18-274.el5 #1 SMP
Fri Jul 8 17:36:59 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
--
View this message in context:
http://lucene.472066.n3.nabble.com/solr-memory-leak-prevent-tomcat-shutdown-tp4014788p4014792.html
Sent from the Solr - User ma
Yeah, I sent a note to the web folks there about the images.
I'll leave the rest to people who really _understand_ all that stuff
On Thu, Sep 20, 2012 at 8:31 AM, Bernd Fehling
wrote:
> Hi Erik,
>
> thanks for the link.
> Now if we could see the images in that article that would be great
Hi Erik,
thanks for the link.
Now if we could see the images in that article that would be great :-)
By the way, one cause for the memory jumps was located as "killer search" from
a user.
The interesting part is that the verbose gc.log showed a "hiccup" in the GC.
Which means that during a GC r
Here's a wonderful writeup about GC and memory in Solr/Lucene:
http://searchhub.org/dev/2011/03/27/garbage-collection-bootcamp-1-0/
Best
Erick
On Thu, Sep 20, 2012 at 5:49 AM, Robert Muir wrote:
> On Thu, Sep 20, 2012 at 3:09 AM, Bernd Fehling
> wrote:
>
>> By the way while looking for upgradi
On Thu, Sep 20, 2012 at 3:09 AM, Bernd Fehling
wrote:
> By the way while looking for upgrading to JDK7, the release notes say under
> section
> "known issues" about the "PorterStemmer" bug:
> "...The recommended workaround is to specify -XX:-UseLoopPredicate on the
> command line."
> Is this st
That is the problem with a jvm, it is a virtual machine.
Ask 10 experts about a good jvm settings and you get 15 answers. May be a
tradeoff
of the flexibility of jvm's. There is always a right setting for any application
running on a jvm but you just have to find it.
How about a Solr Wiki page abo
I have used this setting to reduce gc pauses with CMS - java 6 u23
XX:+ParallelRefProcEnabled
With this setting, jvm does gc of weakrefs with multiple threads and pauses are
low.
Please use this option only when you have multiple cores.
For me, CMS gives better results
Sent from my iPhone
On
Ooh, that is a nasty one. Is this JDK 7 only or also in 6?
It looks like the "-XX:ConcGCThreads=1" option is a workaround, is that right?
We've had some 1.6 JVMs behave in the same way that bug describes, but I
haven't verified it is because of finalizer problems.
wunder
On Sep 19, 2012, at 5:
Two in one morning
The JVM bug I'm familiar with is here:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7112034
FWIW,
Erick
On Wed, Sep 19, 2012 at 8:20 AM, Shawn Heisey wrote:
> On 9/18/2012 9:29 PM, Lance Norskog wrote:
>>
>> There is a known JVM garbage collection bug that causes th
On 9/18/2012 9:29 PM, Lance Norskog wrote:
There is a known JVM garbage collection bug that causes this. It has to do with
reclaiming Weak references, I think in WeakHashMap. Concurrent garbage
collection collides with this bug and the result is that old field cache data
is retained after clos
he problem.)
- Original Message -
| From: "Bernd Fehling"
| To: solr-user@lucene.apache.org
| Sent: Tuesday, September 18, 2012 11:29:56 PM
| Subject: Re: SOLR memory usage jump in JVM
|
| Hi Lance,
|
| thanks for this hint. Something I also see, a sawtooth. This is
| coming from Eden
Hi Otis,
because I see this on my slave without replication there is no index file
change.
I have also tons of logged data to dig in :-)
I took dumps from different stages, fresh installed, after 5GB jump, after the
system was hanging right after replication,...
The last one was interesting when
- Original Message -
> | From: "Yonik Seeley"
> | To: solr-user@lucene.apache.org
> | Sent: Tuesday, September 18, 2012 7:38:41 AM
> | Subject: Re: SOLR memory usage jump in JVM
> |
> | On Tue, Sep 18, 2012 at 7:45 AM, Bernd Fehling
> | wrote:
> | > I use
Hi Bernd,
On Tue, Sep 18, 2012 at 3:09 AM, Bernd Fehling
wrote:
> Hi Otis,
>
> not really a problem because I have plenty of memory ;-)
> -Xmx25g -Xms25g -Xmn6g
Good.
> I'm just interested into this.
> Can you report similar jumps within JVM with your monitoring at sematext?
Yes. More importan
| Sent: Tuesday, September 18, 2012 7:38:41 AM
| Subject: Re: SOLR memory usage jump in JVM
|
| On Tue, Sep 18, 2012 at 7:45 AM, Bernd Fehling
| wrote:
| > I used GC in different situations and tried back and forth.
| > Yes, it reduces the used heap memory, but not by 5GB.
| > Even so t
On Tue, Sep 18, 2012 at 7:45 AM, Bernd Fehling
wrote:
> I used GC in different situations and tried back and forth.
> Yes, it reduces the used heap memory, but not by 5GB.
> Even so that GC from jconsole (or jvisualvm) is "Full GC".
Whatever "Full GC" means ;-)
In the past at least, I've found th
I used GC in different situations and tried back and forth.
Yes, it reduces the used heap memory, but not by 5GB.
Even so that GC from jconsole (or jvisualvm) is "Full GC".
But while you bring GC into this, there is another interesting thing.
- I have one slave running for a week which ends up aro
What happens if you attach jconsole (should ship with your SDK) and force a GC?
Does the extra 5G go away?
I'm wondering if you get a couple of warming searchers going simultaneously
and happened to measure after that.
Uwe has an interesting blog about memory, he recommends using as
little as pos
Hi Otis,
not really a problem because I have plenty of memory ;-)
-Xmx25g -Xms25g -Xmn6g
I'm just interested into this.
Can you report similar jumps within JVM with your monitoring at sematext?
Actually I would assume to see jumps of 0.5GB or even 1GB, but 5GB?
And what is the cause, a cache?
A
Hi Bernd,
But is this really (causing) a problem? What -Xmx are you using?
Otis
Search Analytics - http://sematext.com/search-analytics/index.html
Performance Monitoring - http://sematext.com/spm/index.html
On Tue, Sep 18, 2012 at 2:50 AM, Bernd Fehling
wrote:
> Hi list,
>
> while monitoring
Check your cores' "status" page and see if you're running the
MMapDirectory (you probably are.)
In that case, you probably want to devote even less RAM to Tomcat's
heap because the index files are being read out of memory-mapped pages
that don't reside on the heap, so you'd be devoting more memory
Le 22/08/2012 16:57, Bruno Mannina a écrit :
Dear users,
I try to know if my add in the setenv.sh (which I need to create
because it didn't exist) file has been set but when I click on the
link Java Properties on Admin Solr web page
I can't see the variable CATALINA_OPTS.
In fact, I would li
Faceting and sorting are the two biggest places people get into trouble.
You've been asking questions about Solr Cloud, so I assume you're
working on a trunk release. Note that most everything people know
about memory consumption painfully gained over the years is...wrong
on trunk.
Or at least may
Mike
Actually i'm not able to tell you what each value stands for .. but what i can
tell you is, where the information is coming from.
The interface requests /admin/system which is using
https://svn.apache.org/repos/asf/lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/admin/SystemInf
Hi Bai,
Solr doesn't try to load the whole index into memory, no.
You can control how much memory Tomcat uses with -Xmx Java command line
parameter.
Otis
Performance Monitoring SaaS for Solr -
http://sematext.com/spm/solr-performance-monitoring/index.html
>___
> Commits are divided into 2 groups:
> - often but small (last changed
> info)
1) Make sure that it's not too often and you don't have commit
overlapping problem.
http://wiki.apache.org/solr/FAQ#What_does_.22PERFORMANCE_WARNING:_Overlapping_onDeckSearchers.3DX.22_mean_in_my_logs.3F
2) You may
> Hey Denis,
> * How big is your index in terms of number of documents and index size?
5 cores, average 250.000 documents, one with about 1 million (but
without text, just int/float fields), one with about 10 million
id/name documents, but with n-gram.
Size: 4 databases about 1G (sum),
I ran out of memory on some big indexes when using solr 1.4. Found out that
increasing
termInfosIndexDivisor
in solrconfig.xml could help a lot.
It may slow down your searching your index.
cheers,
:-Dennis
On 02/06/2011, at 01.16, Alexey Serba wrote:
> Hey Denis,
>
> * How big is your in
Hey Denis,
* How big is your index in terms of number of documents and index size?
* Is it production system where you have many search requests?
* Is there any pattern for OOM errors? I.e. right after you start your
Solr app, after some search activity or specific Solr queries, etc?
* What are 1)
There were no parameters at all, and java hitted "out of memory"
almost every day, then i tried to add parameters but nothing changed.
Xms/Xmx - did not solve the problem too. Now i try the MaxPermSize,
because it's the last thing i didn't try yet :(
Wednesday, June 1, 2011, 9:00:56 PM,
Could be related to your crazy high MaxPermSize like Marcus said.
I'm no JVM tuning expert either. Few people are, it's confusing. So if
you don't understand it either, why are you trying to throw in very
non-standard parameters you don't understand? Just start with whatever
the Solr example
PermSize and MaxPermSize don't need to be higher than 64M. You should read on
JVM tuning. The permanent generation is only used for the code that's being
executed.
> So what should i do to evoid that error?
> I can use 10G on server, now i try to run with flags:
> java -Xms6G -Xmx6G -XX:MaxPer
Overall memory on server is 24G, and 24G of swap, mostly all the time
swap is free and is not used at all, that's why "no free swap" sound
strange to me..
> There is no simple answer.
> All I can say is you don't usually want to use an Xmx that's more than
> you actually have available RAM, a
There is no simple answer.
All I can say is you don't usually want to use an Xmx that's more than
you actually have available RAM, and _can't_ use more than you have
available ram+swap, and the Java error seems to be suggesting you are
using more than is available in ram+swap. That may not be
So what should i do to evoid that error?
I can use 10G on server, now i try to run with flags:
java -Xms6G -Xmx6G -XX:MaxPermSize=1G -XX:PermSize=512M -D64
Or should i set xmx to lower numbers and what about other params?
Sorry, i don't know much about java/jvm =(
Wednesday, June 1, 2011, 7:29:
Are you in fact out of swap space, as the java error suggested?
The way JVM's work always, if you tell it -Xmx6g, it WILL use all 6g
eventually. The JVM doesn't Garbage Collect until it's going to run out
of heap space, until it gets to your Xmx. It will keep using RAM until
it reaches your
Here is output after about 24 hours running solr. Maybe there is some
way to limit memory consumption? :(
test@d6 ~/solr/example $ java -Xms3g-Xmx6g-D64
-Dsolr.solr.home=/home/test/solr/example/multicore/ -jar start.jar
2011-05-31 17:05:14.265:INFO::Logging to STDERR via
My OS is also CentOS (5.4). If it were 10gb all the time it would be
ok, but it grows for 13-15gb, and hurts other services =\
> It could be environment specific (specific of your "top" command
> implementation, OS, etc)
> I have on CentOS 2986m "virtual" memory showing although -Xmx2g
> You
It could be environment specific (specific of your "top" command
implementation, OS, etc)
I have on CentOS 2986m "virtual" memory showing although -Xmx2g
You have 10g "virtual" although -Xmx6g
Don't trust it too much... "top" command may count OS buffers for opened
files, network sockets, JVM D
On Tue, 2010-12-14 at 06:07 +0100, Cameron Hurst wrote:
[Cameron expected 150MB overhead]
> As I start to index data and passing queries to the database I notice a
> steady rise in the RAM but it doesn't stop at 150MB. If I continue to
> reindex the exact same data set with no additional data ent
On Mon, Sep 13, 2010 at 6:29 PM, Burton-West, Tom wrote:
> Thanks Robert and everyone!
>
> I'm working on changing our JVM settings today, since putting Solr 1.4.1 into
> production will take a bit more work and testing. Hopefully, I'll be able to
> test the setTermIndexDivisor on our test serv
o see if we can provide you with our tii/tis
data. I'll let you know as soon as I hear anything.
Tom
-Original Message-
From: Robert Muir [mailto:rcm...@gmail.com]
Sent: Sunday, September 12, 2010 10:48 AM
To: solr-user@lucene.apache.org; simon.willna...@gmail.com
Subject: Re: Solr
On Sun, Sep 12, 2010 at 9:57 AM, Simon Willnauer <
simon.willna...@googlemail.com> wrote:
> > To change the divisor in your solrconfig, for example to 4, it looks like
> > you need to do this.
> >
> > > class="org.apache.solr.core.StandardIndexReaderFactory">
> >4
> >
>
> Ah, thanks robert
On Sun, Sep 12, 2010 at 12:42 PM, Robert Muir wrote:
> On Sat, Sep 11, 2010 at 7:51 PM, Michael McCandless <
> luc...@mikemccandless.com> wrote:
>
>> On Sat, Sep 11, 2010 at 11:07 AM, Burton-West, Tom
>> wrote:
>> > Is there an example of how to set up the divisor parameter in
>> solrconfig.xml
On Sat, Sep 11, 2010 at 7:51 PM, Michael McCandless <
luc...@mikemccandless.com> wrote:
> On Sat, Sep 11, 2010 at 11:07 AM, Burton-West, Tom
> wrote:
> > Is there an example of how to set up the divisor parameter in
> solrconfig.xml somewhere?
>
> Alas I don't know how to configure terms index d
One thing that the Codec API makes possible ("in theory", anyway)...
is variable gap terms index.
Ie, Lucene today makes an indexed term at regular (every N -- 128 in
3.x, 32 in 4.0) intervals.
But this is rather silly. Imagine the terms you are going through are
all singletons (happen only in o
On Sun, Sep 12, 2010 at 1:51 AM, Michael McCandless
wrote:
> On Sat, Sep 11, 2010 at 11:07 AM, Burton-West, Tom wrote:
>> Is there an example of how to set up the divisor parameter in
>> solrconfig.xml somewhere?
>
> Alas I don't know how to configure terms index divisor from Solr...
You can s
On Sat, Sep 11, 2010 at 11:07 AM, Burton-West, Tom wrote:
> Is there an example of how to set up the divisor parameter in solrconfig.xml
> somewhere?
Alas I don't know how to configure terms index divisor from Solr...
>>>In 4.0, w/ flex indexing, the RAM efficiency is much better -- we use lar
There is a trick: facets with only one occurrence tend to be mispellings
or dirt. You write a program to fetch the terms (Lucene's CheckIndex is
a great starting point) create a stopwords file.
Here's a data mining project: which languages are more vulnerable to
dirty OCR?
Burton-West, Tom w
Thanks Mike,
>>Do you use a terms index divisor? Setting that to 2 would halve the
>>amount of RAM required but double (on average) the seek time to locate
>>a given term (but, depending on your queries, that seek time may still
>>be a negligible part of overall query time, ie the tradeoff could
Unfortunately, the terms index (before 4.0) is not RAM efficient -- I
wrote about this here:
http://chbits.blogspot.com/2010/07/lucenes-ram-usage-for-searching.html
Every indexed term that's loaded into RAM creates 4 objects (TermInfo,
Term, String, char[]), as you see in your profiler output
I've never paid attention to post/commit ration. I usually do a commit
after maybe 100 posts. Is there a guideline about this? Thanks.
On Wed, May 13, 2009 at 1:10 PM, Otis Gospodnetic
wrote:
> 2) ramBufferSizeMB dictates, more or less, how much Lucene/Solr will consume
> during indexing. Ther
I think that if you have in your index any documents with norms, you
will still use norms for those fields even if the schema is changed
later. Did you wipe and re-index after all your schema changes?
-Peter
On Fri, May 15, 2009 at 9:14 PM, vivek sar wrote:
> Some more info,
>
> Profiling the
Some more info,
Profiling the heap dump shows
"org.apache.lucene.index.ReadOnlySegmentReader" as the biggest object
- taking up almost 80% of total memory (6G) - see the attached screen
shot for a smaller dump. There is some norms object - not sure where
are they coming from as I've omitnorms=tr
Thanks Mark.
I checked all the items you mentioned,
1) I've omitnorms=true for all my indexed fields (stored only fields I
guess doesn't matter)
2) I've tried commenting out all caches in the solrconfig.xml, but
that doesn't help much
3) I've tried commenting out the first and new searcher listen
800 million docs is on the high side for modern hardware.
If even one field has norms on, your talking almost 800 MB right there.
And then if another Searcher is brought up well the old one is serving
(which happens when you update)? Doubled.
Your best bet is to distribute across a couple mac
Some update on this issue,
1) I attached jconsole to my app and monitored the memory usage.
During indexing the memory usage goes up and down, which I think is
normal. The memory remains around the min heap size (4 G) for
indexing, but as soon as I run a search the tenured heap usage jumps
up to 6
I don't know if field type has any impact on the memory usage - does it?
Our use cases require complete matches, thus there is no need of any
analysis in most cases - does it matter in terms of memory usage?
Also, is there any default caching used by Solr if I comment out all
the caches under que
only 20K and you said this is a large index? That doesn't smell
> right...
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> - Original Message
>> From: vivek sar
>> To: solr-user@lucene.apache.org
>> Sent:
On May 13, 2009, at 6:53 PM, vivek sar wrote:
Disabling first/new searchers did help for the initial load time, but
after 10-15 min the heap memory start climbing up again and reached
max within 20 min. Now the GC is coming up all the time, which is
slowing down the commit and search cycles.
T
t.com/ -- Lucene - Solr - Nutch
- Original Message
> From: vivek sar
> To: solr-user@lucene.apache.org
> Sent: Wednesday, May 13, 2009 5:12:00 PM
> Subject: Re: Solr memory requirements?
>
> Otis,
>
> In that case, I'm not sure why Solr is taking up so mu
May 13, 2009 5:53:45 PM
> Subject: Re: Solr memory requirements?
>
> Just an update on the memory issue - might be useful for others. I
> read the following,
>
> http://wiki.apache.org/solr/SolrCaching?highlight=(SolrCaching)
>
> and looks like the first and new
Even a simple command like this will help:
jmap -histo:live | head -30
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
- Original Message
> From: vivek sar
> To: solr-user@lucene.apache.org
> Sent: Wednesday, May 13, 2009 6:53:29 PM
> Subject: Re:
Warning: I'm wy out of my competency range when I comment
on SOLR, but I've seen the statement that string fields are NOT
tokenized while text fields are, and I notice that almost all of your fields
are string type.
Would someone more knowledgeable than me care to comment on whether
this is at
1 - 100 of 109 matches
Mail list logo
