OK, so I managed to 'fix' this issue, but I cannot explain why. The problem was that the SSL password was obfuscated on the server, where on my personal machine it was not.
When starting SOLR 6.4.1 in cloud mode with and obfuscated SLL password, it starts, but is not fully functioning. The first sign is that during starting, it does not respond by saying "happy searching" but rather "SOLR did not come up". At first I thought it was a time-out issue, as I could connect to SOLR via my browser. After switching on all debugging, during startup it already complains about "Keystore was tampered with, or password was incorrect" although it actually starts and the admin UI is fully functional. Using a clear-text password allowed us to start SOLR, use the admin UI and use the collections API. Can anybody replicate this issue and does anybody know why an OBF password results in a partially broken SOLR? The command we use to obfuscate the password: \solr-6.4.1>java -cp server\lib\jetty-util-9.3.14.v20161028.jar org.eclipse.jetty.util.security.Password xxxxxxxx (and yes, we did add the OBF: part to the password when copying to solr.in.cmd) Should I log an issue for this? Marcel From: Marcel Berteler Sent: 21 February 2017 08:12 AM To: 'solr-user@lucene.apache.org' Subject: SSL Problem solr 6.4.1 - Error from shard - ADDREPLICA failed to create replica We are trying to get SOLR 6.4.1 to run on a windows 10 server, but for some reason it just does not want to function properly. On my personal win7 machine, it works like a dream. We configured SOLR to use SSL and run on port 443 on localhost in cloud mode, using the build-in Zookeeper. All is working well and after starting the server for the 1st time, we added the urlScheme (https) as a cluster property. I also ensure the -Dsolr.ssl.checkPeerName=false is added to the SOLR properties in solr.in.cmd During starting, no SSL, password or keystore errors. I can surf to the admin UI and all seems fine. But as soon as I try and use the BACKUP or RESTORE functions, I get the weirdest errors indicating my SSL passwords are incorrect and the process fails. The passwords are correct otherwise SOLR would not start nor would I be able to use the admin UI. To me it indicates that whatever is doing the actual Backup and Restore is not reading the correct passwords. https://localhost/solr/admin/collections?action=RESTORE&name=coct20170214&location=/solr&collection=coct20170214&collection.configName=coct Is there anybody who has a clue why this happens? Here is an extract of the logs: 20 February 2017 09:00:13 AM ERROR true OverseerCollectionMessageHandler Error from shard: https://localhost:443/solr Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(Unknown Source) at sun.security.provider.JavaKeyStore$JKS.engineLoad(Unknown Source) at java.security.KeyStore.load(Unknown Source) at sun.security.ssl.SSLContextImpl$DefaultSSLContext.getDefaultKeyManager(Unknown Source) at sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(Unknown Source) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) ... 55 more Caused by: java.security.UnrecoverableKeyException: Password verification failed ... 64 more 20 February 2017 09:00:13 AM ERROR false OverseerCollectionMessageHandler Collection: coct20170214 operation: restore failed:org.apache.solr.common.SolrException: ADDREPLICA failed to create replica 20 February 2017 09:00:13 AM ERROR false HttpSolrCall null:org.apache.solr.common.SolrException: ADDREPLICA failed to create replica Kind regards, Marcel Berteler Disclaimer: This e-mail (including attachments) is subject to the disclaimer published at: http://www.capetown.gov.za/general/email-disclaimer Please read the disclaimer before opening any attachment or taking any other action in terms of this e-mail. If you cannot access the disclaimer, kindly send an email to disclai...@capetown.gov.za and a copy will be provided to you. By replying to this e-mail or opening any attachment you agree to be bound by the provisions of the disclaimer.