Thanks Jorn on getting back to me. Please find attached memory/cpu
utilization and complete update process chain.
Please let me know your thoughts.
On Sat, Nov 16, 2019 at 9:09 AM Jörn Franke <[email protected]> wrote:
> Can you please provide the whole update chain ? Below is the
> directoryFactory and not the update processor chain.
>
> > Am 15.11.2019 um 17:34 schrieb Fiz N <[email protected]>:
> >
> >
> > Thanks for your response.
> >
> > <directoryFactory
> name="DirectoryFactory"class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
> >
> > attaching the screenshot of physical memory and cpu.
> > Please let me know your thoughts on the below issue.
> >
> >
> >> On Fri, Nov 15, 2019 at 2:18 AM Jörn Franke <[email protected]>
> wrote:
> >> Do you use a updateprocess factory? How does it look like?
> >>
> >> What is the physical memory size and CPU?
> >> What do you mean by “there are 64 cores sending concurrently?” An
> application has 64 threats that send concurrently those updates?
> >>
> >> > Am 15.11.2019 um 02:14 schrieb Fiz N <[email protected]>:
> >> >
> >> > Hi Solr Experts,
> >> >
> >> > SOLR Version : 6.6.2
> >> >
> >> > OS – Linux 3.1.2
> >> >
> >> > JDK – 1.8
> >> >
> >> >
> >> >
> >> > Shard – 16 – All are active.
> >> >
> >> > Xms – 16 gb
> >> >
> >> > Xmx – 16 gb
> >> >
> >> >
> >> >
> >> > Schema fields count – 91
> >> >
> >> > Dynamic fields – 83.
> >> >
> >> >
> >> >
> >> > There are multiple threads sending products to solr. tested with
> batch size
> >> > per thread 50 & 100, it worked fine without error and if the batch
> size is
> >> > 1000 the following error occurs.
> >> >
> >> > there are 64 cores that are sending batches concurrently.
> >> >
> >> >
> >> >
> >> > *I am getting the following error when the batch size is 1000. Please
> >> > advise.*
> >> >
> >> >
> >> >
> >> > 2019-11-14T17:45:44,646 - ERROR [qtp876213901-1169:SolrException@YYY]
> -
> >> > {collection=c:ducts, core=x:ducts_shard15_replica1,
> >> > node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node30,
> shard=s:shard15}
> >> > -
> >> >
> org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
> >> > 2 Async exceptions during distributed update:
> >> >
> >> > 10.YYY.40.62:8983 failed to respond
> >> >
> >> > 10.YYY.40.62:8983 failed to respond
> >> >
> >> >
> >> >
> >> > 2019-11-14T17:45:48,821 - ERROR [qtp876213901-202:SolrException@YYY]
> -
> >> > {collection=c:ducts, core=x:ducts_shard7_replica1,
> >> > node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node29,
> shard=s:shard7}
> >> > -
> >> >
> org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
> >> > Async exception during distributed update: 10.YYY.40.81:8983 failed to
> >> > respond
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > 2019-11-14T19:36:11,599 - ERROR
> >> > [updateExecutor-2-thread-176-processing-x:ducts_shard3_replica2
> >> > http:////10.YYY.40.68:8983//solr//ducts_shard11_replica2
> >> > <http://10.148.40.68:8983/solr/products_shard11_replica2>
> r:core_node26
> >> > n:10.YYY.40.68:8983_solr c:ducts
> >> >
> s:shard3:StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient@131
> ]
> >> > - {ConcurrentUpdateSolrClient.url=
> >> > http://10.YYY.40.68:8983/solr/ducts_shard11_replica2
> >> > <http://10.148.40.68:8983/solr/products_shard11_replica2>,
> >> > collection=c:ducts, core=x:ducts_shard3_replica2,
> >> > node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node26,
> shard=s:shard3}
> >> > - error
> >> >
> >> > org.apache.http.NoHttpResponseException: 10.YYY.40.68:8983 failed to
> respond
> >> >
> >> > at
> >> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
> >> > ~[httpclient-4.4.1.jar:4.4.1]
> >> >
> >> > at
> >> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
> >> > ~[httpclient-4.4.1.jar:4.4.1]
> >> >
> >> > at
> >> > org.apache.http.impl.io
> .AbstractMessageParser.parse(AbstractMessageParser.java:261)
> >> > ~[httpcore-4.4.1.jar:4.4.1]
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > 2019-11-14T19:36:14,567 - ERROR
> >> > [updateExecutor-2-thread-189-processing-x:ducts_shard11_replica2
> >> > http:////10.YYY.40.62:8983//solr//ducts_shard2_replica1
> >> > <http://10.148.40.62:8983/solr/products_shard2_replica1>
> r:core_node25
> >> > n:10.YYY.40.68:8983_solr c:ducts
> >> >
> s:shard11:StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient@131
> ]
> >> > - {ConcurrentUpdateSolrClient.url=
> >> > http://10.YYY.40.62:8983/solr/ducts_shard2_replica1
> >> > <http://10.148.40.62:8983/solr/products_shard2_replica1>,
> >> > collection=c:ducts, core=x:ducts_shard11_replica2,
> >> > node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node25,
> shard=s:shard11}
> >> > - error
> >> >
> >> > java.net.SocketException: Broken pipe (Write failed)
> >> >
> >> > at java.net.SocketOutputStream.socketWrite0(Native Method)
> >> > ~[?:1.8.0_232]
> >> >
> >> > at
> >> > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
> >> > ~[?:1.8.0_232]
> >> >
> >> > at
> java.net.SocketOutputStream.write(SocketOutputStream.java:155)
> >> > ~[?:1.8.0_232]
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > 2019-11-14T19:36:38,851 - ERROR [qtp876213901-542:SolrException@159]
> -
> >> > {collection=c:ducts, core=x:ducts_shard11_replica2,
> >> > node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node25,
> shard=s:shard11}
> >> > -
> >> >
> null:org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
> >> > Async exception during distributed update: Broken pipe (Write failed)
> >> >
> >> > at
> >> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:972)
> >> >
> >> > at
> >> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1911)
> >> >
> >> >
> >> >
> >> >
> >> > Thanks
> >> >
> >> > Fiz..
>
<!-- Add unknown fields to the schema
An example field type guessing update processor that will
attempt to parse string-typed field values as Booleans, Longs,
Doubles, or Dates, and then add schema fields with the guessed
field types.
This requires that the schema is both managed and mutable, by
declaring schemaFactory as ManagedIndexSchemaFactory, with
mutable specified as true.
See http://wiki.apache.org/solr/GuessingFieldTypes
-->
<updateRequestProcessorChain name="add-unknown-fields-to-the-schema">
<!-- UUIDUpdateProcessorFactory will generate an id if none is present in
the incoming document -->
<processor class="solr.UUIDUpdateProcessorFactory"/>
<processor class="solr.RemoveBlankFieldUpdateProcessorFactory"/>
<processor class="solr.FieldNameMutatingUpdateProcessorFactory">
<str name="pattern">[^\w-\.]</str>
<str name="replacement">_</str>
</processor>
<processor class="solr.ParseBooleanFieldUpdateProcessorFactory"/>
<processor class="solr.ParseLongFieldUpdateProcessorFactory"/>
<processor class="solr.ParseDoubleFieldUpdateProcessorFactory"/>
<processor class="solr.ParseDateFieldUpdateProcessorFactory">
<arr name="format">
<str>yyyy-MM-dd'T'HH:mm:ss.SSSZ</str>
<str>yyyy-MM-dd'T'HH:mm:ss,SSSZ</str>
<str>yyyy-MM-dd'T'HH:mm:ss.SSS</str>
<str>yyyy-MM-dd'T'HH:mm:ss,SSS</str>
<str>yyyy-MM-dd'T'HH:mm:ssZ</str>
<str>yyyy-MM-dd'T'HH:mm:ss</str>
<str>yyyy-MM-dd'T'HH:mmZ</str>
<str>yyyy-MM-dd'T'HH:mm</str>
<str>yyyy-MM-dd HH:mm:ss.SSSZ</str>
<str>yyyy-MM-dd HH:mm:ss,SSSZ</str>
<str>yyyy-MM-dd HH:mm:ss.SSS</str>
<str>yyyy-MM-dd HH:mm:ss,SSS</str>
<str>yyyy-MM-dd HH:mm:ssZ</str>
<str>yyyy-MM-dd HH:mm:ss</str>
<str>yyyy-MM-dd HH:mmZ</str>
<str>yyyy-MM-dd HH:mm</str>
<str>yyyy-MM-dd</str>
</arr>
</processor>
<processor class="solr.AddSchemaFieldsUpdateProcessorFactory">
<str name="defaultFieldType">strings</str>
<lst name="typeMapping">
<str name="valueClass">java.lang.Boolean</str>
<str name="fieldType">booleans</str>
</lst>
<lst name="typeMapping">
<str name="valueClass">java.util.Date</str>
<str name="fieldType">tdates</str>
</lst>
<lst name="typeMapping">
<str name="valueClass">java.lang.Long</str>
<str name="valueClass">java.lang.Integer</str>
<str name="fieldType">tlongs</str>
</lst>
<lst name="typeMapping">
<str name="valueClass">java.lang.Number</str>
<str name="fieldType">tdoubles</str>
</lst>
</processor>
<processor class="solr.LogUpdateProcessorFactory"/>
<processor class="solr.DistributedUpdateProcessorFactory"/>
<processor class="solr.RunUpdateProcessorFactory"/>
</updateRequestProcessorChain>
<!-- Deduplication
An example dedup update processor that creates the "id" field
on the fly based on the hash code of some other fields. This
example has overwriteDupes set to false since we are using the
id field as the signatureField and Solr will maintain
uniqueness based on that anyway.
-->
<!--
<updateRequestProcessorChain name="dedupe">
<processor class="solr.processor.SignatureUpdateProcessorFactory">
<bool name="enabled">true</bool>
<str name="signatureField">id</str>
<bool name="overwriteDupes">false</bool>
<str name="fields">name,features,cat</str>
<str name="signatureClass">solr.processor.Lookup3Signature</str>
</processor>
<processor class="solr.LogUpdateProcessorFactory" />
<processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>
-->
<!-- Language identification
This example update chain identifies the language of the incoming
documents using the langid contrib. The detected language is
written to field language_s. No field name mapping is done.
The fields used for detection are text, title, subject and description,
making this example suitable for detecting languages form full-text
rich documents injected via ExtractingRequestHandler.
See more about langId at http://wiki.apache.org/solr/LanguageDetection
-->
<!--
<updateRequestProcessorChain name="langid">
<processor
class="org.apache.solr.update.processor.TikaLanguageIdentifierUpdateProcessorFactory">
<str name="langid.fl">text,title,subject,description</str>
<str name="langid.langField">language_s</str>
<str name="langid.fallback">en</str>
</processor>
<processor class="solr.LogUpdateProcessorFactory" />
<processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>
-->
<!-- Script update processor
This example hooks in an update processor implemented using JavaScript.
See more about the script update processor at
http://wiki.apache.org/solr/ScriptUpdateProcessor
-->
<!--
<updateRequestProcessorChain name="script">
<processor class="solr.StatelessScriptUpdateProcessorFactory">
<str name="script">update-script.js</str>
<lst name="params">
<str name="config_param">example config parameter</str>
</lst>
</processor>
<processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>
