FW: Sort index by size

2018-11-27 Thread Srinivas Kashyap
Hi Shawn and everyone who replied to the thread, The solr version is 5.2.1 and each document is returning multi-valued fields for majority of fields defined in schema.xml. I'm in the process of pasting the content of my files to a paste website and soon will update. Thanks, Srinivas On 11/19/

Re: Sort index by size

2018-11-21 Thread Gus Heck
Just as a sanity check, is this getting replicated many times, or further scaled up... it sounds like about $3.50/mo of disk space on AWS and it should all fit in ram on any decent sized server.. (i.e. any server that looks like half or quarter of a decent laptop) As a question, it's interesting b

Re: Sort index by size

2018-11-19 Thread Edward Ribeiro
One more tidbit: are you really sure you need all 20 fields to be indexed and stored? Do you really need all those 20 fields? See this blog post, for example: https://www.garysieling.com/blog/tuning-solr-lucene-disk-usage On Mon, Nov 19, 2018 at 1:45 PM Walter Underwood wrote: > > Worst case is

Re: Sort index by size

2018-11-19 Thread Walter Underwood
Worst case is 3X. That happens when there are no merges until the commit. With tlogs, worst case is more than that. I’ve seen humongous tlogs with a batch load and no hard commit until the end. If you do that several times, then you have a few old humongous tlogs. Bleah. wunder Walter Underwood

Re: Sort index by size

2018-11-19 Thread David Hastings
Also a full import, assuming the documents were already indexed, will just double your index size until a merge/optimize is ran since you are just marking a document as deleted, not taking back any space, and then adding another completely new document on top of it. On Mon, Nov 19, 2018 at 10:36 A

Re: Sort index by size

2018-11-19 Thread Shawn Heisey
On 11/19/2018 2:31 AM, Srinivas Kashyap wrote: I have a solr core with some 20 fields in it.(all are stored and indexed). For an environment, the number of documents are around 0.29 million. When I run the full import through DIH, indexing is completing successfully. But, it is occupying the d

Sort index by size

2018-11-19 Thread Srinivas Kashyap
Hello, I have a solr core with some 20 fields in it.(all are stored and indexed). For an environment, the number of documents are around 0.29 million. When I run the full import through DIH, indexing is completing successfully. But, it is occupying the disk space of around 5 GB. Is there a poss