RE: [E] Re: Questions about Disk space Usage

2016-10-31 Thread Jamal, Sarfaraz
PM To: solr-user@lucene.apache.org Subject: [E] Re: Questions about Disk space Usage If it works the way I think it does, an empty segment should take the same amount of time to read in as a full segment, but zero time to write out. wunder > On Oct 29, 2016, at 9:21 AM, Erick Erickson wrote

Re: Questions about Disk space Usage

2016-10-29 Thread Walter underwood
If it works the way I think it does, an empty segment should take the same amount of time to read in as a full segment, but zero time to write out. wunder > On Oct 29, 2016, at 9:21 AM, Erick Erickson wrote: > > I would also expect a totally empty segment to be merged very quickly > as the pe

Re: Questions about Disk space Usage

2016-10-29 Thread Erick Erickson
I would also expect a totally empty segment to be merged very quickly as the percent deleted documents weighs heavily when determining whether to merge a segment but that's based on principle, not deep code knowledge. Best, Erick On Fri, Oct 28, 2016 at 6:02 PM, Walter Underwood wrote: > Aft

Re: Questions about Disk space Usage

2016-10-28 Thread Walter Underwood
After the merge. That is what merges do, clean up segments. I expect it is very rare for a segment to be 100% deleted docs, so it isn’t worth handling that case. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Oct 28, 2016, at 5:54 PM, Alexandre Raf

Re: Questions about Disk space Usage

2016-10-28 Thread Alexandre Rafalovitch
Don't the segment that only has deleted documents just gets dropped? Or does it get dropped _after_ the merge and therefore still sits around? Regards, Alex. Solr Example reading group is starting November 2016, join us at http://j.mp/SolrERG Newsletter and resources for Solr beginners and

Re: Questions about Disk space Usage

2016-10-28 Thread Walter Underwood
It is normal for disk usage to double. Under controlled circumstances, it can triple, but that probably won’t happen. This is the second time today that I’ve sent this information to the list. It can use nearly 2X the space whenever the largest segment(s) are merged, especially if there are only

Re: Questions about Disk space Usage

2016-10-28 Thread Alexandre Rafalovitch
2) Is probably a merge operation. Lucene index segments are not rewritable in place, so the merge creates a new file, does everything to it, then switches to it. I remember the number was that the space could temporarily triple (?!?) though that may have been before the tiered merge policy. 3) It

Questions about Disk space Usage

2016-10-28 Thread Jamal, Sarfaraz
Hi Guys, I am currently investigating an instance of Solr's Disk space usage and I had a few questions I thought you guys might be able to help answer. First Question * There is 30 gb's worth of autosuggest data in the /tmp folder. Each file is half of a gigabyte Is it safe to delete those file