Thank you Alex for your answer.
The reference on deleted files are only on index files (with .fdt, .doc.
dvd, ... extensions).
sudo lsof | grep DEL
java 1366 kookel DEL REG 253,8 15360013
/opt/kookel/data/searchSolrNode/solrindex/fr1_green/index/_2508z.cfs
java 1366 kookel DEL REG 253,8 15360035
/opt/kookel/data/searchSolrNode/solrindex/fr1_green/index/_25091.fdt
java 1366 kookel DEL REG 253,8 15425603
/opt/kookel/data/searchSolrNode/solrindex/fr1_green/index/_25091_Lucene50_0.tim
java 1366 kookel DEL REG 253,8 11624982
/opt/kookel/data/searchSolrNode/solrindex/fr1_green/index/_2508y.fdt
...
We have tested to optimize the collection with Solr Admin but without
effect on it.
Elodie
On 03/07/2017 04:11 PM, Alexandre Rafalovitch wrote:
More sanity checks: what are the extensions/types of the files that
are not deleted?
If they are index files, optimize command (even if no longer
recommended for production) should really blow all the old ones away.
So, are they other kinds of files?
Regards,
Alex.
----
http://www.solr-start.com/ - Resources for Solr users, new and experienced
On 7 March 2017 at 09:55, Erick Erickson <erickerick...@gmail.com> wrote:
Just as a sanity check, if you restart the Solr JVM, do the files
disappear from disk?
Do you have any custom code anywhere in this chain? If so, do you open
any searchers but
fail to close them? Although why 6.4 would manifest the problem but
other code wouldn't
is a mystery, just another sanity check.
Best,
Erick
On Tue, Mar 7, 2017 at 6:44 AM, Elodie Sannier <elodie.sann...@kelkoo.fr> wrote:
Hello,
We have migrated from Solr 5.4.1 to Solr 6.4.0 and the disk usage has
increased.
We found hundreds of references to deleted index files being held by solr.
Before the migration, we had 15-30% of disk space used, after the migration
we have 60-90% of disk space used.
We are using Solr Cloud with 2 collections.
The commands applied on the collections are:
- for incremental indexation mode: add, deleteById with commitWithin of 30
minutes
- for full indexation mode: add, deleteById, commit
- for switch between incremental and full mode: deleteByQuery, createAlias,
reload
- there is also an autocommit every 15 minutes
We have seen the email "Solr leaking references to deleted files"
2016-05-31 which describe the same problem but the mentioned bugs are fixed.
We manually tried to force a commit, a reload and an optimize on the
collections without effect.
Is a problem of configuration (merge / delete policy) or a possible
regression in the Solr code ?
Thank you
Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 158 Ter Rue du Temple 75003 Paris
425 093 069 RCS Paris
Ce message et les pièces jointes sont confidentiels et établis à l'attention
exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce
message, merci de le détruire et d'en avertir l'expéditeur.
--
Elodie Sannier
Software engineer
<http://www.kelkoo.com/>
*E*elodie.sann...@kelkoo.fr* Skype*kelkooelodies
*T*+33 (0)4 56 09 07 55
*A*Parc Sud Galaxie, 6, rue des Méridiens, 38130 Echirolles
Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 158 Ter Rue du Temple 75003 Paris
425 093 069 RCS Paris
Ce message et les pièces jointes sont confidentiels et établis à l'attention
exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce
message, merci de le détruire et d'en avertir l'expéditeur.
