7:02, Shalin Shekhar Mangar wrote:
ramBufferSizeMB is independent of the maxIndexingThreads. If you set
it to 100MB then any lucene segment (or part of a segment) exceeding
100MB will be flushed to disk.
On Wed, Jan 20, 2016 at 3:50 AM, Angel Todorov wrote:
hi guys,
quick question - is
ramBufferSizeMB is independent of the maxIndexingThreads. If you set
it to 100MB then any lucene segment (or part of a segment) exceeding
100MB will be flushed to disk.
On Wed, Jan 20, 2016 at 3:50 AM, Angel Todorov wrote:
> hi guys,
>
> quick question - is the ramBufferSizeMB the maxi
hi guys,
quick question - is the ramBufferSizeMB the maximum value no matter how
maxIndexingThreads I have, or is it multiplied by the number if indexing
threads? So, if I have ramBufferSizeMB set to 100 MB, and 8 indexing
threads, does this mean the total ram buffer will be 100 MB or 800 MB
y be get merge together to form a 10GB segment, as I have defined
> the maxMergeSegmentMB to be 10240MB. Then there will be other new small
> segments that are formed from the latest batch of indexing. Is that the way
> it works?
>
> Regards,
> Edwin
>
>
> On 14 January
together to form a 10GB segment, as I have defined
the maxMergeSegmentMB to be 10240MB. Then there will be other new small
segments that are formed from the latest batch of indexing. Is that the way
it works?
Regards,
Edwin
On 14 January 2016 at 10:38, Erick Erickson wrote:
> ramBufferSizeMB i
ramBufferSizeMB is a _limit_ that flushes the buffer when
it is reached (actually, I think, it indexes a doc _then_
checks the size and if it's > the setting, flushes the
buffer. So technically you can exceed the buffer size by
your biggest doc's addition to the index).
But I digre
Hi,
I would like to check, if I have make the following settings for
ramBufferSizeMB, and I am using TieredMergePolicy, am I supposed to get
each segment size of at least 320MB?
320
10
10
10240
I have this setting in my
files and these again into 3,5GB files.
> >
> > Best regards Trym
> >
> > Den 19-09-2012 14:49, Erick Erickson skrev:
> >
> >> I _think_ the getReader calls are being triggered by the autoSoftCommit
> >> being
> >> at one second. If so
e second. If so, this is probably OK. But bumping that up would nail
>> whether that's the case...
>>
>> About RamBufferSizeMB. This has nothing to do with the size of the
>> segments!
>> It's just how much memory is consumed before the RAMBuffer is flushed to
>&g
these again into 3,5GB files.
Best regards Trym
Den 19-09-2012 14:49, Erick Erickson skrev:
I _think_ the getReader calls are being triggered by the autoSoftCommit being
at one second. If so, this is probably OK. But bumping that up would nail
whether that's the case...
About RamBufferS
I _think_ the getReader calls are being triggered by the autoSoftCommit being
at one second. If so, this is probably OK. But bumping that up would nail
whether that's the case...
About RamBufferSizeMB. This has nothing to do with the size of the segments!
It's just how much memory i
Hi
Using SolrCloud I have added the following to solrconfig.xml (actually
the node in zookeeper)
512
After that I expected that my Lucene index segment files would be a bit
bigger than 1KB as I'm indexing very small documents
Enabling the infoStream I see a lot of "flush at getReader" (on
On 8/30/2011 12:57 AM, roz dev wrote:
Thanks Shawn.
If Solr writes this info to Disk as soon as possible (which is what I am
seeing) then ramBuffer setting seems to be misleading.
Anyone else has any thoughts on this?
The stored fields are only two of the eleven Lucene files in each
segment.
Thanks Shawn.
If Solr writes this info to Disk as soon as possible (which is what I am
seeing) then ramBuffer setting seems to be misleading.
Anyone else has any thoughts on this?
-Saroj
On Mon, Aug 29, 2011 at 6:14 AM, Shawn Heisey wrote:
> On 8/28/2011 11:18 PM, roz dev wrote:
>
>> I notic
On 8/28/2011 11:18 PM, roz dev wrote:
I notice that even though InfoStream does not mention that data is being
flushed to disk, new segment files were created on the server.
Size of these files kept growing even though there was enough Heap available
and 856MB Ram was not even used.
With the ca
Hi All,
I am trying to tune ramBufferSizeMB and merge factor for my setup.
So, i enabled Lucene Index Writer's log info stream and started monitoring
Data folder where index files are created.
I started my test with following
Heap: 3GB
Solr 1.4.1,
Index Size = 20 GB,
ramBufferSizeMB=856
On Thu, Dec 2, 2010 at 4:31 PM, Burton-West, Tom wrote:
> We turned on infostream. Is there documentation about how to interpret it,
> or should I just grep through the codebase?
There isn't any documentation... and it changes over time as we add
new diagnostics.
> Is the excerpt below what
On Wed, Dec 1, 2010 at 3:01 PM, Shawn Heisey wrote:
> I have seen this. In Solr 1.4.1, the .fdt, .fdx, and the .tv* files do not
> segment, but all the other files do. I can't remember whether it behaves
> the same under 3.1, or whether it also creates these files in each segment.
Yep, that's t
-user@lucene.apache.org
Subject: Re: ramBufferSizeMB not reflected in segment sizes in index
On Wed, Dec 1, 2010 at 3:16 PM, Burton-West, Tom wrote:
> Thanks Mike,
>
> Yes we have many unique terms due to dirty OCR and 400 languages and probably
> lots of low doc freq terms as well (altho
On Wed, Dec 1, 2010 at 3:16 PM, Burton-West, Tom wrote:
> Thanks Mike,
>
> Yes we have many unique terms due to dirty OCR and 400 languages and probably
> lots of low doc freq terms as well (although with the ICUTokenizer and
> ICUFoldingFilter we should get fewer terms due to bad tokenization a
n the production indexer. If it doesn't I'll turn it on and post
here.
Tom
-Original Message-
From: Michael McCandless [mailto:luc...@mikemccandless.com]
Sent: Wednesday, December 01, 2010 2:43 PM
To: solr-user@lucene.apache.org
Subject: Re: ramBufferSizeMB not reflected in s
On 12/1/2010 12:13 PM, Burton-West, Tom wrote:
We have set the ramBufferSizeMB to 320 in both the indexDefaults and the
mainIndex sections of our solrconfig.xml:
320
20
We expected that this would mean that the index would not write to disk until
it reached somewhere approximately over 300MB
g place.
Mike
On Wed, Dec 1, 2010 at 2:13 PM, Burton-West, Tom wrote:
> We are using a recent Solr 3.x (See below for exact version).
>
> We have set the ramBufferSizeMB to 320 in both the indexDefaults and the
> mainIndex sections of our solrconfig.xml:
>
> 320
> 20
>
We are using a recent Solr 3.x (See below for exact version).
We have set the ramBufferSizeMB to 320 in both the indexDefaults and the
mainIndex sections of our solrconfig.xml:
320
20
We expected that this would mean that the index would not write to disk until
it reached somewhere
essage
> >> From: Tom Burton-West
> >> To: solr-user@lucene.apache.org
> >> Sent: Thu, February 18, 2010 3:30:05 PM
> >> Subject: Re: What is largest reasonable setting for ramBufferSizeMB?
> >>
> >>
> >> Thanks Otis,
> >>
>
i, February 19, 2010 11:41:07 AM
> > Subject: Re: What is largest reasonable setting for ramBufferSizeMB?
> >
> > On Fri, Feb 19, 2010 at 5:03 AM, Glen Newton wrote:
> > > You may consider using LuSql[1] to create the indexes, if your source
> > > content is in
r-user@lucene.apache.org
> Sent: Fri, February 19, 2010 11:41:07 AM
> Subject: Re: What is largest reasonable setting for ramBufferSizeMB?
>
> On Fri, Feb 19, 2010 at 5:03 AM, Glen Newton wrote:
> > You may consider using LuSql[1] to create the indexes, if your source
> > content is in a JDB
sage in context:
http://old.nabble.com/What-is-largest-reasonable-setting-for-ramBufferSizeMB--tp27631231p27658384.html
Sent from the Solr - User mailing list archive at Nabble.com.
On Fri, Feb 19, 2010 at 5:03 AM, Glen Newton wrote:
> You may consider using LuSql[1] to create the indexes, if your source
> content is in a JDBC accessible db. It is quite a bit faster than
> Solr, as it is a tool specifically created and tuned for Lucene
> indexing.
Any idea why it's faster?
A
h
> Hadoop ecosystem search :: http://search-hadoop.com/
>
>
>
> - Original Message
>> From: Tom Burton-West
>> To: solr-user@lucene.apache.org
>> Sent: Thu, February 18, 2010 3:30:05 PM
>> Subject: Re: What is largest reasonable setting for ramBufferSiz
t; To: solr-user@lucene.apache.org
> Sent: Thu, February 18, 2010 3:30:05 PM
> Subject: Re: What is largest reasonable setting for ramBufferSizeMB?
>
>
> Thanks Otis,
>
> I don't know enough about Hadoop to understand the advantage of using Hadoop
> in this use case. How wo
that.
I think you misread Tom's email - it sounds like you are talking about
JVM heap sizes, not ramBufferSizeMB?
32MB is certainly not low, and 320MB is not medium.
-Yonik
http://www.lucidimagination.com
> Otis
>
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Hadoop ecosystem search :: http://search-hadoop.com/
>
>
--
View this message in context:
http://old.nabble.com/What-is-largest-reasonable-setting-for-ramBufferSizeMB--tp27631231p27645167.ht
6:26 PM
> Subject: What is largest reasonable setting for ramBufferSizeMB?
>
> Hello all,
>
> At some point we will need to re-build an index that totals about 2
> terrabytes
> in size (split over 10 shards). At our current indexing speed we estimate
> that
> this wil
t appears that our main bottleneck is disk I/O.
> We currently have ramBufferSizeMB set to 32 and our merge factor is 10. If
> we increase ramBufferSizeMB to 320, we avoid a merge and the 9 disk writes
> and reads to merge 9+1 32MB segments into a 320MB segment.
>
> Assuming we
currently have ramBufferSizeMB set to 32 and our merge factor is 10. If we
increase ramBufferSizeMB to 320, we avoid a merge and the 9 disk writes and
reads to merge 9+1 32MB segments into a 320MB segment.
Assuming we allocate enough memory to the JVM, would it make sense to increase
ramBufferSize
Reply-To:
>> Date: Thu, 05 Nov 2009 22:50:47 +0900
>> To:
>> Subject: Regarding to ramBufferSizeMB and mergeFactor
>>
>> Hello, everybody
>>
>> I am a new Solr user.
>> I have a question about ramBufferSizeMB and mergeFactor.
>> I wo
37 matches
Mail list logo
