Maybe you should consider creating different generations of indexes and not keep everything in one index. If the likelihood of documents being deleted is rather high in, e.g., the first week or so, you could have one index for the high-probability of deletion documents (the fresh ones) and a second one for the potentially longer-lived documents. Without knowing the temporal distribution of deletion probabilities, it is hard to say what would be the ideal index topology.
Apart from that, I have made the experience that in some cases where Solr would produce the notorious out-of-memory exceptions, Elasticsearch seems to be a bit more robust. You may want to give it a try as well. Best regards, --Jürgen On 11.01.2015 07:46, ig01 wrote: > Thank you all for your response, > The thing is that we have 180G index while half of it are deleted documents. > We tried to run an optimization in order to shrink index size but it > crashes on ‘out of memory’ when the process reaches 120G. > Is it possible to optimize parts of the index? > Please advise what can we do in this situation. > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Frequent-deletions-tp4176689p4178700.html > Sent from the Solr - User mailing list archive at Nabble.com. -- Mit freundlichen Grüßen/Kind regards/Cordialement vôtre/Atentamente/С уважением *i.A. Jürgen Wagner* Head of Competence Center "Intelligence" & Senior Cloud Consultant Devoteam GmbH, Industriestr. 3, 70565 Stuttgart, Germany Phone: +49 6151 868-8725, Fax: +49 711 13353-53, Mobile: +49 171 864 1543 E-Mail: juergen.wag...@devoteam.com <mailto:juergen.wag...@devoteam.com>, URL: www.devoteam.de <http://www.devoteam.de/> ------------------------------------------------------------------------ Managing Board: Jürgen Hatzipantelis (CEO) Address of Record: 64331 Weiterstadt, Germany; Commercial Register: Amtsgericht Darmstadt HRB 6450; Tax Number: DE 172 993 071
