It seems that there was a networking error just prior to the creation of
the 0 length files:
The files from Sep 27 are all written at 17:56.
There was minor packet loss (1 out of 10 packets per 60 second interval)
just prior to that time.
On Thu, Oct 5, 2017 at 3:11 PM, Webster Homer <webster.ho...@sial.com>
wrote:
> buffering is disabled. Indeed we disable it everywhere as all it seems to
> do is leave tlogs around forever.
>
> Autocommit is set to 60 seconds.
>
> The source cdcr request handler looks like this. The first target is the
> problematic one
>
> {"requestHandler":{"/cdcr":{
> "name":"/cdcr",
> "class":"solr.CdcrRequestHandler",
> "replica":[
> {
>
> "zkHost":"ae1a-ecomqa-mzk01:2181,ae1a-ecomqa-mzk02:2181,ae1a-ecomqa-mzk03:2181/solr",
> "source":"sial-content-citations",
> "target":"sial-content-citations"},
> {
>
> "zkHost":"uc1b-ecomqa-mzk01:2181,uc1b-ecomqa-mzk02:2181,uc1b-ecomqa-mzk03:2181/solr",
> "source":"sial-content-citations",
> "target":"sial-content-citations"}],
> "replicator":{
> "threadPoolSize":2,
> "schedule":1000,
> "batchSize":250},
> "updateLogSynchronizer":{"schedule":60000}}}}
>
>
>
> The target looks like:
>
> "requestHandler":{"/cdcr":{
> "name":"/cdcr",
> "class":"solr.CdcrRequestHandler",
> "buffer":{"defaultState":"disabled"}}
>
>
> These are all in our QA environment
>
>
> On Thu, Oct 5, 2017 at 2:43 PM, Erick Erickson <erickerick...@gmail.com>
> wrote:
>
>> OK, never mind about the file handle limits, let's deal with the
>> tlogs. Although unlimited is a good thing.
>>
>> Do you have buffering disabled on the target cluster?
>>
>> Best
>> Erick
>>
>> On Thu, Oct 5, 2017 at 11:19 AM, Webster Homer <webster.ho...@sial.com>
>> wrote:
>> > I wouldn't call it massive. The index is ~9 million documents. So not
>> too
>> > big, the documents themselves are pretty small
>> >
>> > On Thu, Oct 5, 2017 at 12:23 PM, Erick Erickson <
>> erickerick...@gmail.com>
>> > wrote:
>> >
>> >> Well, Lucene keeps an open file handle for _every_ file in _every_
>> >> index directory. So, for instance, let's say a replica has 10
>> >> segments. Each segment is 10-15 individual files. So that's 100-150
>> >> file handles right there. And indexes can have many segments.
>> >>
>> >> Check to see if "cfs" extensions are in your indexing directory,
>> >> that's "compound file system" and if present will reduce the number of
>> >> file handles needed.
>> >>
>> >> A second thing you might be able to do is increase the maximum segment
>> >> size by setting maxMergedSegmentMB in your solrconfig file for
>> >> TieredMergePolicy, something like
>> >> <double name="maxMergedSegmentMB">10000</double>
>> >> eventually that'll merge segments into fewer, but that'll take a while.
>> >>
>> >> As to your question, we usually recommend to set the file limit to
>> >> "unlimited". You do have to monitor it however, at some point there's
>> >> a lot of bookkeeping.
>> >>
>> >> one replica trying to open > 8,000 files seems very odd though. Is it
>> >> a massive index? The default max segment size is 5G, so you could have
>> >> a gazillion small segments in which case you might want to split that
>> >> shard up and move the sub-shards to some other machine.
>> >>
>> >> Best,
>> >> Erick
>> >>
>> >> On Thu, Oct 5, 2017 at 10:02 AM, Webster Homer <webster.ho...@sial.com
>> >
>> >> wrote:
>> >> > We have begun to see errors around too many open files on one of our
>> >> > solrcloud nodes. One replica tries to open >8000 files. This replica
>> >> tries
>> >> > to startup and then fails the open files are exceeded upon startup
>> as it
>> >> > tries to recover.
>> >> >
>> >> >
>> >> > Our solrclouds have 12 distinct collections. I would think that the
>> >> number
>> >> > of open files would depend upon the number of collections as well as
>> >> > numbers of files per index etc...
>> >> >
>> >> > Our current setting is 8192 open files per process.
>> >> >
>> >> > What values are recommended? is there a normal number of open files?
>> >> >
>> >> > What would lead to there being lots of open files?
>> >> >
>> >> > --
>> >> >
>> >> >
>> >> > This message and any attachment are confidential and may be
>> privileged or
>> >> > otherwise protected from disclosure. If you are not the intended
>> >> recipient,
>> >> > you must not copy this message or attachment or disclose the
>> contents to
>> >> > any other person. If you have received this transmission in error,
>> please
>> >> > notify the sender immediately and delete the message and any
>> attachment
>> >> > from your system. Merck KGaA, Darmstadt, Germany and any of its
>> >> > subsidiaries do not accept liability for any omissions or errors in
>> this
>> >> > message which may arise as a result of E-Mail-transmission or for
>> damages
>> >> > resulting from any unauthorized changes of the content of this
>> message
>> >> and
>> >> > any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its
>> >> > subsidiaries do not guarantee that this message is free of viruses
>> and
>> >> does
>> >> > not accept liability for any damages caused by any virus transmitted
>> >> > therewith.
>> >> >
>> >> > Click http://www.emdgroup.com/disclaimer to access the German,
>> French,
>> >> > Spanish and Portuguese versions of this disclaimer.
>> >>
>> >
>> > --
>> >
>> >
>> > This message and any attachment are confidential and may be privileged
>> or
>> > otherwise protected from disclosure. If you are not the intended
>> recipient,
>> > you must not copy this message or attachment or disclose the contents to
>> > any other person. If you have received this transmission in error,
>> please
>> > notify the sender immediately and delete the message and any attachment
>> > from your system. Merck KGaA, Darmstadt, Germany and any of its
>> > subsidiaries do not accept liability for any omissions or errors in this
>> > message which may arise as a result of E-Mail-transmission or for
>> damages
>> > resulting from any unauthorized changes of the content of this message
>> and
>> > any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its
>> > subsidiaries do not guarantee that this message is free of viruses and
>> does
>> > not accept liability for any damages caused by any virus transmitted
>> > therewith.
>> >
>> > Click http://www.emdgroup.com/disclaimer to access the German, French,
>> > Spanish and Portuguese versions of this disclaimer.
>>
>
>
--
This message and any attachment are confidential and may be privileged or
otherwise protected from disclosure. If you are not the intended recipient,
you must not copy this message or attachment or disclose the contents to
any other person. If you have received this transmission in error, please
notify the sender immediately and delete the message and any attachment
from your system. Merck KGaA, Darmstadt, Germany and any of its
subsidiaries do not accept liability for any omissions or errors in this
message which may arise as a result of E-Mail-transmission or for damages
resulting from any unauthorized changes of the content of this message and
any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its
subsidiaries do not guarantee that this message is free of viruses and does
not accept liability for any damages caused by any virus transmitted
therewith.
Click http://www.emdgroup.com/disclaimer to access the German, French,
Spanish and Portuguese versions of this disclaimer.
