Hi Erick,
I read mergeFactor Policy for indexing. By default, mergerFactor
is 10. As said in document,
High value merge factor (e.g., 25):
- Pro: Generally improves indexing speed
- Con: Less frequent merges, resulting in a collection with more index
files which may slow searc
On Fri, Mar 20, 2015 at 1:35 AM, Erick Erickson
wrote:
> That or even hard commit to 60 seconds. It's strictly a matter of how often
> you want to close old segments and open new ones.
>
> On Thu, Mar 19, 2015 at 3:12 AM, Nitin Solanki
> wrote:
> > Hi Erick..
> > I read your Articl
That or even hard commit to 60 seconds. It's strictly a matter of how often
you want to close old segments and open new ones.
On Thu, Mar 19, 2015 at 3:12 AM, Nitin Solanki wrote:
> Hi Erick..
> I read your Article. Really nice...
> Inside that you said that for bulk indexing. Set s
Hi Erick..
I read your Article. Really nice...
Inside that you said that for bulk indexing. Set soft commit = 10 mins and
hard commit = 15sec. Is it also okay for my scenario?
On Thu, Mar 19, 2015 at 1:53 AM, Erick Erickson
wrote:
> bq: As you said, do commits after 6 seconds
>
Hi Alxeandre,
Number of segment counts are different but document
counts are same.
With (soft commit - 300 and hardcommit - 6000) = No. of segment - 43
AND
With (soft commit - 6 and hardcommit - 6) = No. of segment - 31
I dont' have any idea related to segment count
bq: As you said, do commits after 6 seconds
No, No, No. I'm NOT saying 6 seconds! That time is in _milliseconds_
as Shawn said. So setting it to 6 is every minute.
>From solrconfig.xml, conveniently located immediately above the
tag:
maxTime - Maximum amount of time in ms that is al
Probably merged somewhat differently with some terms indexes repeating
between segments. Check the number of segments in data directory.And
do search for *:* and make sure both do have the same document counts.
Also, In all these discussions, you still haven't answered about how
fast after indexin
When I kept my configuration to 300 for soft commit and 3000 for hard
commit and indexed some amount of data, I got the data size of the whole
index to be 6GB after completing the indexing.
When I changed the configuration to 6 for soft commit and 6 for
hard commit and indexed same data th
On 3/18/2015 9:44 AM, Nitin Solanki wrote:
> I am just saying. I want to be sure on commits difference..
> What if I do frequent commits or not? And why I am saying that I need to
> commit things so very quickly because I have to index 28GB of data which
> takes 7-8 hours(frequent comm
Hi Erick,
I am just saying. I want to be sure on commits difference..
What if I do frequent commits or not? And why I am saying that I need to
commit things so very quickly because I have to index 28GB of data which
takes 7-8 hours(frequent commits).
As you said, do commits after 6
Don't do it. Really, why do you want to do this? This seems like
an "XY" problem, you haven't explained why you need to commit
things so very quickly.
I suspect you haven't tried _searching_ while committing at such
a rate, and you might as well turn all your top-level caches off
in solrconfig.xml
Hi,
If I do very very fast indexing(softcommit = 300 and hardcommit =
3000) v/s slow indexing (softcommit = 6 and hardcommit = 6) as you
both said. Will fast indexing fail to index some data?
Any suggestion on this ?
On Tue, Mar 17, 2015 at 2:29 AM, Ramkumar R. Aiyengar <
andyetitmo
Yes, and doing so is painful and takes lots of people and hardware
resources to get there for large amounts of data and queries :)
As Erick says, work backwards from 60s and first establish how high the
commit interval can be to satisfy your use case..
On 16 Mar 2015 16:04, "Erick Erickson" wrote
First start by lengthening your soft and hard commit intervals
substantially. Start with 6 and work backwards I'd say.
Ramkumar has tuned the heck out of his installation to get the commit
intervals to be that short ;).
I'm betting that you'll see your RAM usage go way down, but that' s a
gue
Hi Erick,
You are saying correct. Something, **"overlapping searchers"
warning messages** are coming in logs.
**numDocs numbers** are changing when documents are adding at the time of
indexing.
Any help?
On Sat, Mar 14, 2015 at 11:24 PM, Erick Erickson
wrote:
> First, the soft commit
First, the soft commit interval is very short. Very, very, very, very
short. 300ms is
just short of insane unless it's a typo ;).
Here's a long background:
https://lucidworks.com/blog/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
But the short form is that you're opening sear
Hi Alexandre,
*Hard Commit* is :
${solr.autoCommit.maxTime:3000}
false
*Soft Commit* is :
${solr.autoSoftCommit.maxTime:300}
And I am committing 2 documents each time.
Is it good config for committing?
Or I am good something wrong ?
On Fri, Mar 13, 2015 a
What's your commit strategy? Explicit commits? Soft commits/hard
commits (in solrconfig.xml)?
Regards,
Alex.
Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
http://www.solr-start.com/
On 12 March 2015 at 23:19, Nitin Solanki wrote:
> Hello,
> I have written a
Hello,
I have written a python script to do 2 documents indexing
each time on Solr. I have 28 GB RAM with 8 CPU.
When I started indexing, at that time 15 GB RAM was freed. While indexing,
all RAM is consumed but **not** a single document is indexed. Why so?
And it through *HTTPError:
19 matches
Mail list logo
