BACKGROUND: I'm attempting to setup SolrCloud (Solr 6.6.6) with an external zookeeper ensemble on Azure. I have three dedicated vms for the zookeeper ensemble and two for solr all running Ubuntu 18.04 LTS. I'm new to Solr (and Linux) and have been heavily relying on the Solr Ref Guide 6.6, most recently the following section on enabling ssl:
https://lucene.apache.org/solr/guide/6_6/enabling-ssl.html So far I have: Installed and setup zookeeper Installed Solr (using install_solr_service.sh script) on both vms. Followed the steps under Basic SSL Setup, generating certificates on each of the nodes. Set the cluster-wide property to https per the Configure Zookeeper section of SolrCloud in the document Started both nodes and have been able to navigate to them in my browser with https If I do bin/solr status I get: Solr process 13106 running on port 8983 { "solr_home":"/opt/solr-6.6.6/cloud/test2", "version":"6.6.6 68fa249034ba8b273955f20097700dc2fbb7a800 - ishan - 2019-03-29 09:13:13", "startTime":"2020-09-03T18:15:34.092Z", "uptime":"0 days, 0 hours, 43 minutes, 29 seconds", "memory":"52.7 MB (%10.7) of 490.7 MB", "cloud":{ "ZooKeeper":"zk1:2181,zk2:2181,zk3:2181/solr", "liveNodes":"2", "collections":"0"}} THE ISSUE When I try to create a collection using the steps outlined in the above document, I get the following error: azureuser@solr-node-01-test:/opt/solr$ sudo bin/solr create -c mycollection -shards 2 -force Connecting to ZooKeeper at zk1:2181,zk2:2181,zk3:2181/solr ... INFO - 2020-09-03 18:21:26.784; org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider; Cluster at zk1:2181,zk2:2181,zk3:2181/solr ready Re-using existing configuration directory mycollection Creating new collection 'mycollection' using command: https://Solr1:8983/solr/admin/collections?action=CREATE&name=mycollection&numShards=2&replicationFactor=1&maxShardsPerNode=1&collection.configName=mycollection ERROR: Failed to create collection 'mycollection' due to: {Solr2:8983_solr=org.apache.solr.client.solrj.SolrServerException:IOException occured when talking to server at: https://Solr2:8983/solr} *I've attached logs at the bottom of this email. QUESTIONS: What am I doing wrong and how can I fix it? Was I right to create separate certificates on each of the nodes (one cert on vm1, another cert on vm 2)? Do I need to copy the certs for each node into the other (if so how)? CONCLUSION Thank you so much in advance and if there's any other information you need please let me know. Victor 2020-09-03 18:15:35.240 INFO (zkCallback-5-thread-1-processing-n:Solr1:8983_solr) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) 2020-09-03 18:15:40.124 INFO (qtp401424608-45) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores 2020-09-03 18:15:40.124 INFO (qtp401424608-45) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={indexInfo=false&wt=json&_=1599156956818} status=0 QTime=23 2020-09-03 18:15:40.134 INFO (qtp401424608-20) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json&_=1599156956818} status=0 QTime=29 2020-09-03 18:15:40.171 INFO (qtp401424608-13) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&wt=json&_=1599156956818 and sendToOCPQueue=true 2020-09-03 18:15:40.172 INFO (qtp401424608-13) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=json&_=1599156956818} status=0 QTime=1 2020-09-03 18:15:40.174 INFO (qtp401424608-16) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json&_=1599156956818} status=0 QTime=8 2020-09-03 18:15:58.225 INFO (qtp401424608-14) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={indexInfo=false&wt=json&_=1599156974989} status=0 QTime=0 2020-09-03 18:15:58.231 INFO (qtp401424608-13) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json&_=1599156974989} status=0 QTime=7 2020-09-03 18:15:58.258 INFO (qtp401424608-20) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&wt=json&_=1599156974989 and sendToOCPQueue=true 2020-09-03 18:15:58.258 INFO (qtp401424608-20) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=json&_=1599156974989} status=0 QTime=0 2020-09-03 18:15:58.263 INFO (qtp401424608-21) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json&_=1599156974989} status=0 QTime=7 2020-09-03 18:19:38.661 INFO (qtp401424608-16) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json} status=0 QTime=6 2020-09-03 18:19:38.687 INFO (qtp401424608-45) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=CLUSTERSTATUS&wt=json and sendToOCPQueue=true 2020-09-03 18:19:38.691 INFO (qtp401424608-45) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=json} status=0 QTime=4 2020-09-03 18:21:26.640 INFO (qtp401424608-13) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json} status=0 QTime=7 2020-09-03 18:21:26.677 INFO (qtp401424608-16) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json} status=0 QTime=5 2020-09-03 18:21:26.685 INFO (qtp401424608-45) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=CLUSTERSTATUS&wt=json and sendToOCPQueue=true 2020-09-03 18:21:26.688 INFO (qtp401424608-45) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=json} status=0 QTime=2 2020-09-03 18:21:26.799 INFO (qtp401424608-16) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=list&wt=json and sendToOCPQueue=true 2020-09-03 18:21:26.799 INFO (qtp401424608-16) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=list&wt=json} status=0 QTime=0 2020-09-03 18:21:26.809 INFO (qtp401424608-16) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&maxShardsPerNode=1&collection.configName=mycollection&name=mycollection&action=CREATE&numShards=2&wt=json and sendToOCPQueue=true 2020-09-03 18:21:26.835 INFO (OverseerThreadFactory-6-thread-1-processing-n:Solr1:8983_solr) [ ] o.a.s.c.CreateCollectionCmd Create collection mycollection 2020-09-03 18:21:27.068 INFO (qtp401424608-14) [ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=mycollection&newCollection=true&name=mycollection_shard2_replica1&action=CREATE&numShards=2&collection=mycollection&shard=shard2&wt=javabin&version=2 2020-09-03 18:21:27.104 ERROR (OverseerThreadFactory-6-thread-1-processing-n:Solr1:8983_solr) [ ] o.a.s.c.OverseerCollectionMessageHandler Error from shard: https://Solr2:8983/solr org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://Solr2:8983/solr at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:626) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268) at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219) at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:164) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed at sun.security.ssl.Alerts.getSSLException(Alerts.java:198) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1967) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:331) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:325) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1688) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:226) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1082) at sun.security.ssl.Handshaker.process_record(Handshaker.java:1010) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1079) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1388) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1416) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1400) at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543) at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177) at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515) ... 12 more Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:380) at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:285) at sun.security.validator.Validator.validate(Validator.java:262) at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237) at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1670) ... 29 more Caused by: java.security.cert.CertPathValidatorException: signature check failed at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135) at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:233) at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:141) at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:80) at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292) at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:375) ... 35 more Caused by: java.security.SignatureException: Signature does not match. at sun.security.x509.X509CertImpl.verify(X509CertImpl.java:457) at sun.security.provider.certpath.BasicChecker.verifySignature(BasicChecker.java:166) at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:147) at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125) ... 40 more
