I typically end up with about 60-70 segments after indexing. What configuration do you use to bring it down to 16?
> On 2 Mar 2017, at 7:42 pm, Michael Joyner <mich...@newsrx.com> wrote: > > You can solve the disk space and time issues by specifying multiple segments > to optimize down to instead of a single segment. > > When we reindex we have to optimize or we end up with hundreds of segments > and very horrible performance. > > We optimize down to like 16 segments or so and it doesn't do the 3x disk > space thing and usually runs in a decent amount of time. (we have >50 million > articles in one of our solr indexes). > > >> On 03/02/2017 10:20 AM, David Hastings wrote: >> Agreed, and since it takes three times the space is part of the reason it >> takes so long, so that 190gb index ends up writing another 380 gb until it >> compresses down and deletes the two left over files. its a pretty hefty >> operation >> >> On Thu, Mar 2, 2017 at 10:13 AM, Alexandre Rafalovitch <arafa...@gmail.com> >> wrote: >> >>> Optimize operation is no longer recommended for Solr, as the >>> background merges got a lot smarter. >>> >>> It is an extremely expensive operation that can require up to 3-times >>> amount of disk during the processing. >>> >>> This is not to say yours is a valid question, which I am leaving to >>> others to respond. >>> >>> Regards, >>> Alex. >>> ---- >>> http://www.solr-start.com/ - Resources for Solr users, new and experienced >>> >>> >>>> On 2 March 2017 at 10:04, Caruana, Matthew <mcaru...@icij.org> wrote: >>>> I’m currently performing an optimise operation on a ~190GB index with >>> about 4 million documents. The process has been running for hours. >>>> This is surprising, because the machine is an EC2 r4.xlarge with four >>> cores and 30GB of RAM, 24GB of which is allocated to the JVM. >>>> The load average has been steady at about 1.3. Memory usage is 25% or >>> less the whole time. iostat reports ~6% util. >>>> What gives? >>>> >>>> Running Solr 6.4.1. >
