RE: Updating Expired Certificate

2020-10-03 Thread Jeremy Snyder 
Hello,

How do we update a certificate in a SOLR Cloud without downtime?

We have a SOLR cloud cluster with three nodes.  SSL is enabled for use with 
SiteCore.  We currently have a certificate in the "solr-ssl.keystore.jks".  
This certificate expired so we imported a new certificate into the keystore(the 
original is still there as well).  After restarting SOLR the certificate 
replacement is successful.  However, the node with the new cert cannot talk to 
the other nodes in the cluster.  How do services stay available as we update 
the certificate?

Thanks,
Jeremy

The information contained in this email is confidential and is intended solely 
for the use of the person identified and intended as the recipient. If you are 
not the intended recipient, any disclosure, copying, distribution, or taking of 
any action in reliance on the contents is prohibited. If you receive this 
message in error, contact the sender immediately and delete it from your 
computer. Personal e-mails are restricted by PSECU policy. As such, PSECU 
specifically disclaims any responsibility or liability for any personal 
information or opinions of the author expressed in this email

Solr Issue - Logger : UpdateLog Error Message : java.io.EOFException

2020-10-03 Thread Training By Coding 
*Events:*

   1. GC logs showing continuous Full GC events. Log report attached.
   2. *Core filling failed , showing less data( Num Docs)  than expected.*
   3. following warnings showing on dashboard before error.


Level Logger Message
WARN false ManagedIndexSchemaFactory The schema has been upgraded to
managed, but the non-managed schema schema.xml is still loadable. PLEASE
REMOVE THIS FILE.
WARN false ManagedIndexSchemaFactory The schema has been upgraded to
managed, but the non-managed schema schema.xml is still loadable. PLEASE
REMOVE THIS FILE.
WARN false SolrResourceLoader Solr loaded a deprecated plugin/analysis
class [solr.TrieDateField]. Please consult documentation how to replace it
accordingly.
WARN false ManagedIndexSchemaFactory The schema has been upgraded to
managed, but the non-managed schema schema.xml is still loadable. PLEASE
REMOVE THIS FILE.
WARN false UpdateLog Starting log replay
tlog{file=\data\tlog\tlog.0445482 refcount=2}
active=false starting pos=0 inSortedOrder=false

   - Total data in all cores around 8 GB
   - *Other Configurations:*
  - -XX:+UseG1GC
  - -XX:+UseStringDeduplication
  - -XX:MaxGCPauseMillis=500
  - -Xms15g
  - -Xmx15g
  - -Xss256k
   - *OS Environment :*
  - Windows 10,
  - Filling cores by calling SQL query using jtds-1.3.1 library.
  - Solr Version 7.5
  - Runtime: Oracle Corporation OpenJDK 64-Bit Server VM 11.0.2 11.0.2+9
  - Processors : 48
  - System Physical Memory : 128 GB
  - Swap Space : 256GB
   - solr-spec7.5.0
  - solr-impl7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi -
  2018-09-18 13:07:55
   - lucene-spec7.5.0
  - lucene-impl7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi
  - 2018-09-18 13:01:1

*Error Message :*

java.io.EOFException
at
org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
at org.apache.solr.common.util.JavaBinCodec.readStr(JavaBinCodec.java:863)
at org.apache.solr.common.util.JavaBinCodec.readStr(JavaBinCodec.java:857)
at
org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:266)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
at
org.apache.solr.common.util.JavaBinCodec.readSolrInputDocument(JavaBinCodec.java:603)
at
org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:315)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
at org.apache.solr.common.util.JavaBinCodec.readArray(JavaBinCodec.java:747)
at
org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:272)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
at
org.apache.solr.update.TransactionLog$LogReader.next(TransactionLog.java:673)
at
org.apache.solr.update.UpdateLog$LogReplayer.doReplay(UpdateLog.java:1832)
at org.apache.solr.update.UpdateLog$LogReplayer.run(UpdateLog.java:1747)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

Re: Solr Issue - Logger : UpdateLog Error Message : java.io.EOFException

2020-10-03 Thread Erick Erickson 
Very strange things start to happen when GC becomes unstable. The first and 
simplest thing to do would be to bump up your heap, say to 20g (note, try to 
stay under 32G or be prepared to jump significantly higher. At 32G long 
pointers have to be used and you actually have less memory available than you 
think).

The first three warnings indicate that you have both managed-schema and 
schema.xml in your configset _and_ are using the managed schema (enabled in 
solrconfig.xml). Which also suggests you’re upgrading from a previous version. 
This is printed out in as a courtesy notification that schema.xml is no longer 
being used so you should delete it to avoid confusion. NOTE: if you want to use 
schema.xml like you have before, see the reference guide.

The fourth warning suggests that you have killed Solr without committing and 
it’s replaying the transaction log. For instance, “kill -9” or other will do 
it. If you do something like that before a commit completes, updates are 
replayed from the tlog in order to preserve data.

Which leads to your second issue. I’d guess either you’re not committing after 
your updates (and, BTW, please just let your autocommit settings handle that), 
and forcefully killing Solr (e.g. kill -9). That can happen even if you use the 
“bin/solr stop” command if it takes too long (3 minutes by default last I 
knew). A “normal” shutdown that succeeds (i.e. bin/solr stop that doesn’t print 
a message about forcefully killing Solr” will commit on shutdown BTW. Taking 
over 3 minutes may be a symptom of GC going crazy.

You should to try to figure out why you have this kind of memory spike, 
returning a zillion documents is one possible cause (i.e. rows=100 or 
something). All the docs have to be assembled in memory, so if you need to 
return lots of rows, use streaming or cursorMark.

So what I’d do:
1> bump up your heap
2> insure that you shut Solr down gracefully
3> see if any particular query triggers this memory spike and if you’re using 
an anti-pattern.

Best,
Erick

> On Oct 2, 2020, at 7:10 PM, Training By Coding  
> wrote:
> 
> Events:
>   • GC logs showing continuous Full GC events. Log report attached.
>   • Core filling failed , showing less data( Num Docs)  than expected.
>   • following warnings showing on dashboard before error.
> 
> Level Logger  Message
> WARN falseManagedIndexSchemaFactory   The schema has been upgraded to 
> managed, but the non-managed schema schema.xml is still loadable. PLEASE 
> REMOVE THIS FILE.
> WARN falseManagedIndexSchemaFactory   The schema has been upgraded to 
> managed, but the non-managed schema schema.xml is still loadable. PLEASE 
> REMOVE THIS FILE.
> WARN falseSolrResourceLoader  Solr loaded a deprecated 
> plugin/analysis class [solr.TrieDateField]. Please consult documentation how 
> to replace it accordingly.
> WARN falseManagedIndexSchemaFactory   The schema has been upgraded to 
> managed, but the non-managed schema schema.xml is still loadable. PLEASE 
> REMOVE THIS FILE.
> WARN falseUpdateLog   Starting log replay 
> tlog{file=\data\tlog\tlog.0445482 refcount=2} 
> active=false starting pos=0 inSortedOrder=false
>   • Total data in all cores around 8 GB
>   • Other Configurations:
>   • -XX:+UseG1GC
>   • -XX:+UseStringDeduplication
>   • -XX:MaxGCPauseMillis=500
>   • -Xms15g
>   • -Xmx15g
>   • -Xss256k
>   • OS Environment :
>   • Windows 10,
>   • Filling cores by calling SQL query using jtds-1.3.1 library.
>   • Solr Version 7.5
>   • Runtime: Oracle Corporation OpenJDK 64-Bit Server VM 11.0.2 
> 11.0.2+9
>   • Processors : 48
>   • System Physical Memory : 128 GB
>   • Swap Space : 256GB
>   • solr-spec7.5.0
>   • solr-impl7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - 
> jimczi - 2018-09-18 13:07:55
>   • lucene-spec7.5.0
>   • lucene-impl7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - 
> jimczi - 2018-09-18 13:01:1
> Error Message : 
> java.io.EOFException
> at 
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at org.apache.solr.common.util.JavaBinCodec.readStr(JavaBinCodec.java:863)
> at org.apache.solr.common.util.JavaBinCodec.readStr(JavaBinCodec.java:857)
> at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:266)
> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
> at 
> org.apache.solr.common.util.JavaBinCodec.readSolrInputDocument(JavaBinCodec.java:603)
> at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:315)
> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
> at org.apache.solr.common.util.JavaBinCodec.readArray(JavaBinCodec.java:747)
> at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCo