It could be that when this index was first reconstructed, it was optimized to one segment before packed and shipped.
How about optimizing it again, with maxSegments set to ten, it should recover right? -----Original message----- > From:Shawn Heisey <apa...@elyograg.org> > Sent: Friday 5th January 2018 14:34 > To: solr-user@lucene.apache.org > Subject: Re: Very high number of deleted docs, part 2 > > On 1/5/2018 5:33 AM, Markus Jelsma wrote: > > Another collection, now on 7.1, also shows this problem and has default TMP > > settings. This time size is different, each shard of this collection is > > over 40 GB, and each shard has about 50 % deleted documents. Each shard's > > largest segment is just under 20 GB with about 75 % deleted documents. > > After that are a few five/six GB segments with just under 50 % deleted > > documents. > > > > What do i need to change to make Lucene believe that at least that twenty > > GB and three month old segment should be merged away. And how what would > > the predicted indexing performance penalty be? > > Quick answer: Erick's statements in the previous thread can be > summarized as this: On large indexes that do a lot of deletes or > updates, once you do an optimize, you have to continue to do optimizes > regularly, or you're going to have this problem. > > TL;DR: > > I think Erick covered most of this (possibly all of it) in the previous > thread. > > If you've got a 20GB segment and TMP's settings are default, then that > means at some point in the past, you've done an optimize. The default > TMP settings have a maximum segment size of 5GB, so if you never > optimize, then there will never be a segment larger than 5GB, and the > deleted document percentage would be less likely to get out of control. > The optimize operation ignores the maximum segment size and reduces the > index to a single large segment with zero deleted docs. > > TMP's behavior with really big segments is apparently completely as the > author intended, but this specific problem wasn't ever addressed. > > If you do an optimize once and then don't ever do it again, any very > large segments are going to be vulnerable to this problem, and the only > way (currently) to fix it is to do another optimize. > > See this issue for a more in-depth discussion and an attempt to figure > out how to avoid it: > > https://issues.apache.org/jira/browse/LUCENE-7976 > > Thanks, > Shawn > >
