Interestingly many of these tlog files (5428 out of 8007) are have 0 length!? What would cause that? As I stated this is a cdcr target collection.
On Thu, Oct 5, 2017 at 1:19 PM, Webster Homer <webster.ho...@sial.com> wrote: > I wouldn't call it massive. The index is ~9 million documents. So not too > big, the documents themselves are pretty small > > On Thu, Oct 5, 2017 at 12:23 PM, Erick Erickson <erickerick...@gmail.com> > wrote: > >> Well, Lucene keeps an open file handle for _every_ file in _every_ >> index directory. So, for instance, let's say a replica has 10 >> segments. Each segment is 10-15 individual files. So that's 100-150 >> file handles right there. And indexes can have many segments. >> >> Check to see if "cfs" extensions are in your indexing directory, >> that's "compound file system" and if present will reduce the number of >> file handles needed. >> >> A second thing you might be able to do is increase the maximum segment >> size by setting maxMergedSegmentMB in your solrconfig file for >> TieredMergePolicy, something like >> <double name="maxMergedSegmentMB">10000</double> >> eventually that'll merge segments into fewer, but that'll take a while. >> >> As to your question, we usually recommend to set the file limit to >> "unlimited". You do have to monitor it however, at some point there's >> a lot of bookkeeping. >> >> one replica trying to open > 8,000 files seems very odd though. Is it >> a massive index? The default max segment size is 5G, so you could have >> a gazillion small segments in which case you might want to split that >> shard up and move the sub-shards to some other machine. >> >> Best, >> Erick >> >> On Thu, Oct 5, 2017 at 10:02 AM, Webster Homer <webster.ho...@sial.com> >> wrote: >> > We have begun to see errors around too many open files on one of our >> > solrcloud nodes. One replica tries to open >8000 files. This replica >> tries >> > to startup and then fails the open files are exceeded upon startup as it >> > tries to recover. >> > >> > >> > Our solrclouds have 12 distinct collections. I would think that the >> number >> > of open files would depend upon the number of collections as well as >> > numbers of files per index etc... >> > >> > Our current setting is 8192 open files per process. >> > >> > What values are recommended? is there a normal number of open files? >> > >> > What would lead to there being lots of open files? >> > >> > -- >> > >> > >> > This message and any attachment are confidential and may be privileged >> or >> > otherwise protected from disclosure. If you are not the intended >> recipient, >> > you must not copy this message or attachment or disclose the contents to >> > any other person. If you have received this transmission in error, >> please >> > notify the sender immediately and delete the message and any attachment >> > from your system. Merck KGaA, Darmstadt, Germany and any of its >> > subsidiaries do not accept liability for any omissions or errors in this >> > message which may arise as a result of E-Mail-transmission or for >> damages >> > resulting from any unauthorized changes of the content of this message >> and >> > any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its >> > subsidiaries do not guarantee that this message is free of viruses and >> does >> > not accept liability for any damages caused by any virus transmitted >> > therewith. >> > >> > Click http://www.emdgroup.com/disclaimer to access the German, French, >> > Spanish and Portuguese versions of this disclaimer. >> > > -- This message and any attachment are confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, you must not copy this message or attachment or disclose the contents to any other person. If you have received this transmission in error, please notify the sender immediately and delete the message and any attachment from your system. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not accept liability for any omissions or errors in this message which may arise as a result of E-Mail-transmission or for damages resulting from any unauthorized changes of the content of this message and any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not guarantee that this message is free of viruses and does not accept liability for any damages caused by any virus transmitted therewith. Click http://www.emdgroup.com/disclaimer to access the German, French, Spanish and Portuguese versions of this disclaimer.
