also I am seeing the below error as a parent one from solr.log
at org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:366)
org.apache.solr.common.SolrException: javax.crypto.BadPaddingException: RSA
private key operation failed
at
org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:366)
~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 -
ishan - 2020-01-10 1$
at
org.apache.solr.security.PKIAuthenticationPlugin.generateToken(PKIAuthenticationPlugin.java:305)
~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d0$
at
org.apache.solr.security.PKIAuthenticationPlugin.access$200(PKIAuthenticationPlugin.java:61)
~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 -$
at
org.apache.solr.security.PKIAuthenticationPlugin$2.onQueued(PKIAuthenticationPlugin.java:239)
~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 $
at
org.apache.solr.client.solrj.impl.Http2SolrClient.decorateRequest(Http2SolrClient.java:469)
~[solr-solrj-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 -$
at
org.apache.solr.client.solrj.impl.Http2SolrClient.initOutStream(Http2SolrClient.java:324)
~[solr-solrj-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 - i$
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:227)
~[solr-solrj-8.4.1.jar:8.4.1 83$
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:181)
~[solr-solrj-8.4.1.jar:8.4.1 832bf13dd918709$
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
~[metrics-core-4.0.5.jar:4.0.5]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
~[solr-solrj-8.4.1.jar:8.4.1 832bf13dd9187095831caf6978$
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$142/0000000000000000.run(Unknown
Source) ~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: javax.crypto.BadPaddingException: RSA private key operation
failed
at
sun.security.rsa.NativeRSACore.crtCrypt_Native(NativeRSACore.java:149)
~[?:?]
at sun.security.rsa.NativeRSACore.rsa(NativeRSACore.java:91) ~[?:?]
at sun.security.rsa.RSACore.rsa(RSACore.java:149) ~[?:?]
at com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:355)
~[?:?]
at
com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:392) ~[?:?]
at javax.crypto.Cipher.doFinal(Cipher.java:2260) ~[?:?]
at
org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:364)
~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 -
ishan - 2020-01-10 1$
... 13 more
On Tue, Jun 2, 2020 at 6:37 PM yaswanth kumar <yaswanth...@gmail.com> wrote:
> Thanks Franke, but yes for all these questions I did configured it
> properly, I made sure to include
>
> <Set name="KeyStoreType"><Property name="solr.jetty.keystore.type"
> default="JKS"/></Set>
> <Set name="TrustStoreType"><Property name="solr.jetty.truststore.type"
> default="JKS"/></Set>
> in the jetty-ssl.xml along with the path keystore and truststore.
>
> Also I have made sure that trusstore exists on all nodes and also I am
> using the same file for both keystore and truststore as below
> <Set name="KeyStorePath"><Property name="solr.jetty.keystore"
> default="./etc/solr-keystore.jks"/></Set>
> <Set name="KeyStorePassword"><Property
> name="solr.jetty.keystore.password" default="xxxx"/></Set>
> <Set name="TrustStorePath"><Property name="solr.jetty.truststore"
> default="./etc/solr-keystore.jks"/></Set>
> <Set name="TrustStorePassword"><Property
> name="solr.jetty.truststore.password" default="xxxx"/></Set>
>
> also urlScheme for ZK is set to https
>
>
> Also the main error that I posted is the one that I am seeing as a return
> response where as the below one is what I see from solr logs
>
> 2020-06-02 22:32:04.472 ERROR (qtp984876512-93) [c:default s:shard1
> r:core_node3 x:default_shard1_replica_n1] o.a.s.s.HttpSolrCall
> null:org.apache.solr.update.processor.Distr$
> at
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)
> at
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
> at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>
>
> One strange observation is that when I hit update api on the leader node
> its working without any error, and now immediately if I hit non-leader its
> working fine (only once or twice), but if I keep on trying to hit this node
> again and again its then throwing the above error and once the error
> started happening , its consistent again.
>
> Please let me know if you need more information or if I am missing
> something else
>
> Thanks,
>
> On Tue, Jun 2, 2020 at 4:59 PM Jörn Franke <jornfra...@gmail.com> wrote:
>
>> Have you looked in the logfiles?
>>
>> Keystore Type correctly defined on all nodes?
>>
>> Have you configured the truststore on all nodes correctly?
>>
>> Have you set clusterprop urlScheme to htttps in ZK?
>>
>>
>> https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper
>>
>>
>>
>> > Am 02.06.2020 um 18:57 schrieb yaswanth kumar <yaswanth...@gmail.com>:
>> >
>> > team, can someone help me on the above topic?
>> >
>> >> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <yaswanth...@gmail.com>
>> >> wrote:
>> >>
>> >> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
>> >> configurations with all the certs in place, but the issue what I am
>> seeing
>> >> is when trying to hit /update api on non-leader solr node , its
>> throwing an
>> >> error
>> >>
>> >> configured 2 solr nodes with 1 zookeeper.
>> >>
>> >> metadata":[
>> >>
>> >>
>> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
>> >>
>> >>
>> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
>> >> "msg":"Async exception during distributed update:
>> >> javax.crypto.BadPaddingException: RSA private key operation failed",
>> >>
>> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>> >> Async exception during distributed update:
>> >> javax.crypto.BadPaddingException: RSA private key operation
>> failed\n\tat
>> >>
>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
>> >>
>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
>> >>
>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
>> >>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
>> >> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
>> >>
>> >> Strangely this is happening when we try to hit a non-leader node,
>> hitting
>> >> leader node its working fine without any issue and getting the data
>> indexed.
>> >>
>> >> Not able to track down where the exact issue is happening.
>> >>
>> >> Thanks,
>> >>
>> >> --
>> >> Thanks & Regards,
>> >> Yaswanth Kumar Konathala.
>> >> yaswanth...@gmail.com
>> >>
>> >
>> >
>> > --
>> > Thanks & Regards,
>> > Yaswanth Kumar Konathala.
>> > yaswanth...@gmail.com
>>
>
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> yaswanth...@gmail.com
>
--
Thanks & Regards,
Yaswanth Kumar Konathala.
yaswanth...@gmail.com
