Ok, thanks Shawn!
That makes sense. We'll be experimenting with it.
Best,
Eric
On Wed, Oct 7, 2015 at 5:54 PM, Shawn Heisey wrote:
> On 10/7/2015 12:00 PM, Eric Torti wrote:
>> Can we read "high reopen rate" as "frequent soft commits"? (In our
>> case, hard commits do not open a searcher. But s
On 10/7/2015 12:00 PM, Eric Torti wrote:
> Can we read "high reopen rate" as "frequent soft commits"? (In our
> case, hard commits do not open a searcher. But soft commits do).
>
> Considering it does mean "frequent soft commits", I'd say that it
> doesn't fit our setup because we have an index rat
S and Java version support it. If support isn't there,
>> it will use conventional file access methods. As far as I know, all
>> 64-bit Java versions and 64-bit operating systems will support MMap.
>
> Considering our JVM is 64-bit, that probably explains why we're
>
ting systems will support MMap.
Considering our JVM is 64-bit, that probably explains why we're
experiencing MMapDirectory like behaviour on our cluster (i.e. high
non-JVM related memory use).
As to NRTCachingDirectoryFactory, when looking up the docs we were in
doubt about what it means to have
On 10/7/2015 8:48 AM, Eric Torti wrote:
> class="${solr.directoryFactory:solr.StandardDirectoryFactory}"
> name="DirectoryFactory"/>
>
> I'm just starting to grasp different strategies for Directory
> implementation. Can I assume that solr.Stand
Hello,
I'm running a 5.2.1 SolrCloud cluster and I see that one of my cores
is configured under solrconfig.xml to use
I'm just starting to grasp different strategies for Directory
implementation. Can I assume that solr.StandardDirectoryFactory is a
MMapDirectory as described by Uwe
On 7/6/2015 2:06 PM, Vincenzo D'Amore wrote:
> just a simple question, if I change from
> solr.NRTCachingDirectoryFactory to solr.MMapDirectoryFactory should I
> reindex the entire collection?
>
> I have tried in a test environment and the change seems to be applied
> seamless.
No, reindexing is n
Hi All,
just a simple question, if I change from
solr.NRTCachingDirectoryFactory to solr.MMapDirectoryFactory should I
reindex the entire collection?
I have tried in a test environment and the change seems to be applied
seamless.
Best regards,
Vincenzo
e reading!
Cheers
Si
-Original Message-
From: Shawn Heisey [mailto:apa...@elyograg.org]
Sent: 08 October 2014 21:09
To: solr-user@lucene.apache.org
Subject: Re: Solr configuration, memory usage and MMapDirectory
On 10/8/2014 4:02 AM, Simon Fairey wrote:
> I'm currently setting
the size of the index data
that has been opened on the disk. This is not memory that's actually
allocated. There's a very good reason that mmap has been the default in
Lucene and Solr for more than two years.
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
You stat
large for these, are these what are using up the memory?
Thanks
Si
-Original Message-
From: Shawn Heisey [mailto:apa...@elyograg.org]
Sent: 06 October 2014 16:56
To: solr-user@lucene.apache.org
Subject: Re: Solr configuration, memory usage and MMapDirectory
On 10/6/2014 9:24 AM, Simon F
On 10/6/2014 10:07 AM, Simon Fairey wrote:
> Thanks and yeah I thought it might be crazy, the image is just the JVM memory
> usage you get from the dashboard on the solr admin pages, the JVM on has what
> appears to be a light grey then dark grey band then some blank space, those
> are the numbe
Thanks I will have a read and digest this.
-Original Message-
From: Shawn Heisey [mailto:apa...@elyograg.org]
Sent: 06 October 2014 16:56
To: solr-user@lucene.apache.org
Subject: Re: Solr configuration, memory usage and MMapDirectory
On 10/6/2014 9:24 AM, Simon Fairey wrote:
> I
e max heap setting?
Will digest the blog.
Cheers
Si
-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com]
Sent: 06 October 2014 16:58
To: solr-user@lucene.apache.org
Subject: Re: Solr configuration, memory usage and MMapDirectory
First, the e-mail programs tend to
s excellent blog ont he subject, with hints on how to read top:
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Meanwhile, Shawn gave you some very good info so I won't repeat any
On Mon, Oct 6, 2014 at 8:24 AM, Simon Fairey wrote:
> Hi
>
> I've in
>
>
>
>
> 90
>
>
>
> Currently each node has around 25M docs in with an index size of 45GB,
> we prune the data every few weeks so it never gets much above 35M docs
> per node.
>
> On reading I've seen a recommendation that we should be usin
every few weeks so it never gets much above 35M docs per
node.
On reading I've seen a recommendation that we should be using MMapDirectory,
currently it's set to NRTCachingDirectoryFactory. However currently the JVM
is configured with -Xmx131072m, and for MMapDirectory I've read you s
Hi All,
I was testing my solr on MMapDirectory, and while indexing, I get this error
lines in the log:
10:27:41.003 [commitScheduler-4-thread-1] ERROR
org.apache.solr.update.CommitTracker - auto commit
error...:org.apache.solr.common.SolrException: Error opening new searcher
at
You may well have already seen this, but in case not:
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
FWIW,
Erick
On Wed, Oct 24, 2012 at 9:51 PM, Shawn Heisey wrote:
> On 10/24/2012 6:29 PM, Aaron Daubman wrote:
>>
>> Let me be clear that that I am no
On 10/24/2012 6:29 PM, Aaron Daubman wrote:
Let me be clear that that I am not interested in RAMDirectory.
However, I would like to better understand the oft-recommended and
currently-default MMapDirectory, and what the tradeoffs would be, when
using a 64-bit linux server dedicated to this
e.
>
> There has been a lot written about this topic on the list. Basically it come
> down to using MMapDirectory (which is the default), make sure your index is
> smaller than your RAM, and allocate just enough memory to the Java VM. That
> last part requires some benchmarki
nce the system got to a steady state.
There has been a lot written about this topic on the list. Basically it come
down to using MMapDirectory (which is the default), make sure your index is
smaller than your RAM, and allocate just enough memory to the Java VM. That
last part requires s
this
well?
- Original Message -
| From: "Mikhail Khludnev"
| To: solr-user@lucene.apache.org
| Sent: Thursday, September 20, 2012 11:19:58 AM
| Subject: Re: MMapDirectory
|
| My limited understanding, confirmed by profiler though, is that doing
| mmap
| IO cost you a copying
ity question pop up and wanted to check it out.
> Solr has the documentCache, designed to hold stored fields while
> various parts of a requestHandler do their tricks, keeping the stored
> content from having to be re-fetched from disk. When using
> MMapDirectory, is this even something
So I just had a curiosity question pop up and wanted to check it out.
Solr has the documentCache, designed to hold stored fields while
various parts of a requestHandler do their tricks, keeping the stored
content from having to be re-fetched from disk. When using
MMapDirectory, is this even
My colleague and I thought the same thing - that this is an O/S
configuration issue.
/proc/sys/vm/max_map_count = 65536
I honestly don't know how many segments were in the index. Our merge factor
is 10 and there were around 4.4 million docs indexed. The OOME was raised
when the MMapDirector
I hit similar issue recently.
Not sure if MMapDirectory is right way to go.
When index file be map to ram, JVM will call OS file mapping function.
The memory usage is in share memory, it may not be calculate to JVM process
space.
I saw one problem is if the index file bigger then physical ram
On Tue, Sep 20, 2011 at 12:32 PM, Michael McCandless
wrote:
>
> Or: is it possible you reopened the reader several times against the
> index (ie, after committing from Solr)? If so, I think 2.9.x never
> unmaps the mapped areas, and so this would "accumulate" against the
> system limit.
In order
t address space.
Your virtual memory is unlimited (from "ulimit" output), so that's good.
Mike McCandless
http://blog.mikemccandless.com
On Wed, Sep 7, 2011 at 12:25 PM, Rich Cariens wrote:
> Ahoy ahoy!
>
> I've run into the dreaded OOM error with MMapDirectory on
> files
> >> ulimit. Do the MultiMMapIndexInput ByteBuffer arrays each consume a file
> >> handle/descriptor?
> >>
> >> On Thu, Sep 8, 2011 at 5:19 PM, Rich Cariens
> >> wrote:
> >>
> >> > FWiW I optimized the index down to a single segmen
> On Thu, Sep 8, 2011 at 5:19 PM, Rich Cariens
>> wrote:
>>
>> > FWiW I optimized the index down to a single segment and now I have no
>> > trouble opening an MMapDirectory on that index, even though the 23G cfx
>> > segment file remains.
>> >
>>
wrote:
>
> > FWiW I optimized the index down to a single segment and now I have no
> > trouble opening an MMapDirectory on that index, even though the 23G cfx
> > segment file remains.
> >
> >
> > On Thu, Sep 8, 2011 at 4:27 PM, Rich Cariens >wrote:
&g
open files
ulimit. Do the MultiMMapIndexInput ByteBuffer arrays each consume a file
handle/descriptor?
On Thu, Sep 8, 2011 at 5:19 PM, Rich Cariens wrote:
> FWiW I optimized the index down to a single segment and now I have no
> trouble opening an MMapDirectory on that index, even though
FWiW I optimized the index down to a single segment and now I have no
trouble opening an MMapDirectory on that index, even though the 23G cfx
segment file remains.
On Thu, Sep 8, 2011 at 4:27 PM, Rich Cariens wrote:
> Thanks for the response. "free -g" reports:
>
>
iettecatte
> My memory of this is a little rusty but isn't mmap also limited by mem +
> swap on the box? What does 'free -g' report?
>
> François
>
> On Sep 7, 2011, at 12:25 PM, Rich Cariens wrote:
>
> > Ahoy ahoy!
> >
> > I've run
My memory of this is a little rusty but isn't mmap also limited by mem + swap
on the box? What does 'free -g' report?
François
On Sep 7, 2011, at 12:25 PM, Rich Cariens wrote:
> Ahoy ahoy!
>
> I've run into the dreaded OOM error with MMapDirectory on a 23G cfs
Ahoy ahoy!
I've run into the dreaded OOM error with MMapDirectory on a 23G cfs compound
index segment file. The stack trace looks pretty much like every other trace
I've found when searching for OOM & "map failed"[1]. My configuration
follows:
Solr 1.4.1/Lucene 2.9
thank you. I will try it.
On Mon, Aug 8, 2011 at 11:18 PM, Rich Cariens wrote:
> We patched our 1.4.1 build with
> SOLR-1969<https://issues.apache.org/jira/browse/SOLR-1969>(making
> MMapDirectory configurable) and realized a 64% search performance
> boost on our Linux hosts.
We patched our 1.4.1 build with
SOLR-1969<https://issues.apache.org/jira/browse/SOLR-1969>(making
MMapDirectory configurable) and realized a 64% search performance
boost on our Linux hosts.
On Mon, Aug 8, 2011 at 10:05 AM, Dyer, James wrote:
> If you want to try MMapDirectory with Solr
If you want to try MMapDirectory with Solr 1.4, then copy the class
org.apache.solr.core.MMapDirectoryFactory from 3.x or Trunk, and either add it
to the .war file (you can just add it under "src/java" and re-package the war),
or you can put it in its own .jar file in the "lib&q
hi all,
I read Apache Solr 3.1 Released Note today and found that
MMapDirectory is now the default implementation in 64 bit Systems.
I am now using solr 1.4 with 64-bit jvm in Linux. how can I use
MMapDirectory? will it improve performance?
Hey I've been testing MMapDirectory with a Debian and with just couple of
cores (1G each index) in the box it seems to give a bit better performance
with a cold index. But I don't think is good enough (not in my use case)
knowing the memory it requires.
When doing an snapinstaller (
Is there a benefit to using MMapDirectory instead of, say, tmpfs (RAM disk)
under Linux?
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
- Original Message
> From: Mark Miller
> To: solr-user@lucene.apache.org
> Sent: Monday, July 6, 2009 9:28:43 AM
>
Marc Sturlese wrote:
Hey there,
For testing purpose I am trying to use lucene's MMapDirectory in a recent
Solr nightly instance. I have read in Lucene's documentation:
"To use MMapDirectory, invoke Java with the System property
org.apache.lucene.FSDirecto
Hey there,
For testing purpose I am trying to use lucene's MMapDirectory in a recent
Solr nightly instance. I have read in Lucene's documentation:
"To use MMapDirectory, invoke Java with the System property
org.apache.lucene.FSDirectory.class set to
org.apache.lucene.store.MMap
: Is there a way to use a MMapDirectory instead of FSDirectory within Solr ?
i'm not very familiar with MMapDirectory but according to the javadocs...
To use this, invoke Java with the System property
org.apache.lucene.FSDirectory.class s
Hi !
Is there a way to use a MMapDirectory instead of FSDirectory within Solr ?
Our index is quite big and It takes a long time to go up in the OS
cached memory. I'm wondering if an MMapDirectory could help to have
our data in memory quicker (our index on disk is bigger than our
m
47 matches
Mail list logo
