Hmmm so I looked closer - it looks like a side effect of the default passthrough of the keystore being passed to the client keystore.
https://github.com/apache/lucene-solr/blob/master/solr/bin/solr#L229 Can you remove or commout the entire SOLR_SSL_CLIENT_KEY_STORE section from bin/solr or bin/solr.cmd depending on which version you are using? The key being to make sure to not set "-Djavax.net.ssl.keyStore". This assumes that you aren't using Solr client auth (which based on your config you aren't) and you aren't trying to use Solr to connect to anything that is secured via clientAuth (most likely you aren't). If you can try this and report back that would be awesome. I think this will fix the issue and it would be possible to make client auth opt in instead of default fall back. Kevin Risden On Tue, Jul 14, 2020 at 1:46 AM Natarajan, Rajeswari < rajeswari.natara...@sap.com> wrote: > Thank you so much for the response. Below are the configs I have in > solr.in.sh and I followed > https://lucene.apache.org/solr/guide/8_5/enabling-ssl.html documentation > > # Enables HTTPS. It is implicitly true if you set SOLR_SSL_KEY_STORE. Use > this config > # to enable https module with custom jetty configuration. > SOLR_SSL_ENABLED=true > # Uncomment to set SSL-related system properties > # Be sure to update the paths to the correct keystore for your environment > SOLR_SSL_KEY_STORE=etc/solr-ssl.keystore.p12 > SOLR_SSL_KEY_STORE_PASSWORD=secret > SOLR_SSL_TRUST_STORE=etc/solr-ssl.keystore.p12 > SOLR_SSL_TRUST_STORE_PASSWORD=secret > # Require clients to authenticate > SOLR_SSL_NEED_CLIENT_AUTH=false > # Enable clients to authenticate (but not require) > SOLR_SSL_WANT_CLIENT_AUTH=false > # SSL Certificates contain host/ip "peer name" information that is > validated by default. Setting > # this to false can be useful to disable these checks when re-using a > certificate on many hosts > SOLR_SSL_CHECK_PEER_NAME=true > > In local , with the below certificate it works > --------------------------------------- > > keytool -list -keystore solr-ssl.keystore.p12 > Enter keystore password: > Keystore type: PKCS12 > Keystore provider: SUN > > Your keystore contains 1 entry > > solr-18, Jun 26, 2020, PrivateKeyEntry, > Certificate fingerprint (SHA1): > AB:F2:C8:84:E8:E7:A2:BF:2D:0D:2F:D3:95:4A:98:5B:2A:88:81:50 > C02W48C6HTD6:solr-8.5.1 i843100$ keytool -list -v -keystore > solr-ssl.keystore.p12 > Enter keystore password: > Keystore type: PKCS12 > Keystore provider: SUN > > Your keystore contains 1 entry > > Alias name: solr-18 > Creation date: Jun 26, 2020 > Entry type: PrivateKeyEntry > Certificate chain length: 1 > Certificate[1]: > Owner: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, > ST=State, C=Country > Issuer: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, > ST=State, C=Country > Serial number: 45a822c8 > Valid from: Fri Jun 26 00:13:03 PDT 2020 until: Sun Nov 10 23:13:03 PST > 2047 > Certificate fingerprints: > MD5: 0B:80:54:89:44:65:93:07:1F:81:88:8D:EC:BD:38:41 > SHA1: AB:F2:C8:84:E8:E7:A2:BF:2D:0D:2F:D3:95:4A:98:5B:2A:88:81:50 > SHA256: > 9D:65:A6:55:D7:22:B2:72:C2:20:55:66:F8:0C:9C:48:B1:F6:48:40:A4:FB:CB:26:77:DE:C4:97:34:69:25:42 > Signature algorithm name: SHA256withRSA > Subject Public Key Algorithm: 2048-bit RSA key > Version: 3 > > Extensions: > > #1: ObjectId: 2.5.29.17 Criticality=false > SubjectAlternativeName [ > DNSName: localhost > IPAddress: 172.20.10.4 > IPAddress: 127.0.0.1 > ] > > #2: ObjectId: 2.5.29.14 Criticality=false > SubjectKeyIdentifier [ > KeyIdentifier [ > 0000: 1B 6F BB 65 A4 3C 6A F4 C9 05 08 89 88 0E 9E 76 .o.e.<j........v > 0010: A1 B7 28 BE ..(. > ] > > ///////////////////////////////////////////////////////////////// > In a cluster env , where the deployment , keystore everything is > automated (used by multiple teams) keystore generated is as below. As you > can see the keystore has 2 certificates , in which case I get the > exception below. > > java.lang.UnsupportedOperationException: X509ExtendedKeyManager only > > supported on Server > > at > > > org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:223) > > > > In both cases , the config is same except the keystore certificates . In > the JIRA (https://issues.apache.org/jira/browse/SOLR-14105) , I see the > fix says it supports multiple DNS and multiple certificates. So I thought > it should be ok. Please let me know . > > keytool -list -keystore /etc/nginx/certs/sidecar.p12 > Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8 > Enter keystore password: > Keystore type: PKCS12 > Keystore provider: SUN > > Your keystore contains 1 entry > > 1, Jul 7, 2020, PrivateKeyEntry, > Certificate fingerprint (SHA1): > E2:3B:4B:4A:0E:05:CF:DA:59:09:55:8D:4E:6D:8A:1D:4E:DD:D4:62 > bash-5.0# > ————————- > > bash-5.0# keytool -list -v -keystore /etc/nginx/certs/sidecar.p12 > Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8 > Enter keystore password: > Keystore type: PKCS12 > Keystore provider: SUN > > Your keystore contains 1 entry > > Alias name: 1 > Creation date: Jul 7, 2020 > Entry type: PrivateKeyEntry > Certificate chain length: 2 > Certificate[1]: > Owner: OU=Cobalt, O=SAP, L=Walldorf, ST=Walldorf, C=DE > Issuer: CN=SAP Ariba Cobalt Sidecar Intermediate CA, OU=COBALT, O=SAP > Ariba, ST=CA, C=US > Serial number: 1000 > Valid from: Tue Jul 07 05:14:37 GMT 2020 until: Thu Jul 07 05:14:37 GMT > 2022 > Certificate fingerprints: > MD5: C0:13:87:37:96:C2:E2:DD:B9:D7:B4:E3:6B:73:A0:EC > SHA1: E2:3B:4B:4A:0E:05:CF:DA:59:09:55:8D:4E:6D:8A:1D:4E:DD:D4:62 > SHA256: > 89:AB:8E:3B:D4:EC:A6:D0:0E:D7:CB:65:8C:92:13:32:F2:FD:7E:41:C9:39:F5:66:D5:7D:F1:04:13:8A:4E:92 > Signature algorithm name: SHA256withRSA > Subject Public Key Algorithm: 2048-bit RSA key > Version: 3 > > Extensions: > > #1: ObjectId: 2.16.840.1.113730.1.13 Criticality=false > 0000: 16 24 4F 70 65 6E 53 53 4C 20 47 65 6E 65 72 61 .$OpenSSL Genera > 0010: 74 65 64 20 53 65 72 76 65 72 20 43 65 72 74 69 ted Server Certi > 0020: 66 69 63 61 74 65 ficate > > > #2: ObjectId: 2.5.29.35 Criticality=false > AuthorityKeyIdentifier [ > KeyIdentifier [ > 0000: E9 5C 42 72 5E 70 D9 02 05 AA 11 BA 0D 4D 8D 0D .\Br^p.......M.. > 0010: F3 37 2C 95 .7,. > ] > [CN=SAP Ariba Cobalt CA, OU=ES, O=SAP Ariba, L=Palo Alto, ST=CA, C=US] > SerialNumber: [ 1001] > ] > > #3: ObjectId: 2.5.29.19 Criticality=false > BasicConstraints:[ > CA:false > PathLen: undefined > ] > > #4: ObjectId: 2.5.29.37 Criticality=false > ExtendedKeyUsages [ > serverAuth > ] > > #5: ObjectId: 2.5.29.15 Criticality=true > KeyUsage [ > DigitalSignature > Key_Encipherment > ] > > #6: ObjectId: 2.16.840.1.113730.1.1 Criticality=false > NetscapeCertType [ > SSL server > ] > > #7: ObjectId: 2.5.29.17 Criticality=false > SubjectAlternativeName [ > DNSName: search-solrcloud-solrcloud.service > DNSName: search-solrcloud-solrcloud.service.mu.aws.ariba.com > DNSName: *.query.mu.aws.ariba.com > DNSName: *.query > DNSName: *.service > DNSName: > e046469b-1bb0-55f6-913f-bd6d52b238a8.search-solrcloud-solrcloud.service.mu.aws.ariba.com > DNSName: > e046469b-1bb0-55f6-913f-bd6d52b238a8.search-solrcloud-solrcloud.service > DNSName: *.service.mu.aws.ariba.com > DNSName: 1.search-solrcloud-solrcloud.service.mu.aws.ariba.com > DNSName: 1.search-solrcloud-solrcloud.service > DNSName: localhost > IPAddress: 10.1.56.9 > IPAddress: 10.169.50.16 > IPAddress: 127.0.0.1 > ] > > #8: ObjectId: 2.5.29.14 Criticality=false > SubjectKeyIdentifier [ > KeyIdentifier [ > 0000: 3F 9D 3D 24 48 1E 61 3C BD C0 A4 07 8B 64 51 0D ?.=$H.a<.....dQ. > 0010: A2 B2 FE 89 .... > ] > ] > > Certificate[2]: > Owner: CN=SAP Ariba Cobalt Sidecar Intermediate CA, OU=COBALT, O=SAP > Ariba, ST=CA, C=US > Issuer: CN=SAP Ariba Cobalt CA, OU=ES, O=SAP Ariba, L=Palo Alto, ST=CA, > C=US > Serial number: 1001 > Valid from: Thu Apr 16 07:18:55 GMT 2020 until: Sun Apr 14 07:18:55 GMT > 2030 > Certificate fingerprints: > MD5: FA:70:2F:DB:63:36:66:71:A6:7B:0F:46:F3:52:0B:3C > SHA1: 4F:27:D3:E3:12:24:64:18:B5:97:D0:BF:94:37:2D:5C:33:EA:1E:40 > SHA256: > 15:28:F4:DB:B3:D5:2E:21:6A:2E:56:47:E3:6B:D3:16:96:18:06:96:DA:5D:28:6B:34:CB:6D:FA:E8:FA:85:13 > Signature algorithm name: SHA256withRSA > Subject Public Key Algorithm: 4096-bit RSA key > Version: 3 > > Extensions: > > #1: ObjectId: 2.5.29.35 Criticality=false > AuthorityKeyIdentifier [ > KeyIdentifier [ > 0000: D8 A1 D1 11 50 8C 1C 2A 67 69 82 40 DF B5 68 6A ....P..*g...@..hj > 0010: E4 97 6E 32 ..n2 > ] > ] > > #2: ObjectId: 2.5.29.19 Criticality=true > BasicConstraints:[ > CA:true > PathLen:0 > ] > > #3: ObjectId: 2.5.29.15 Criticality=true > KeyUsage [ > DigitalSignature > Key_CertSign > Crl_Sign > ] > > #4: ObjectId: 2.5.29.14 Criticality=false > SubjectKeyIdentifier [ > KeyIdentifier [ > 0000: E9 5C 42 72 5E 70 D9 02 05 AA 11 BA 0D 4D 8D 0D .\Br^p.......M.. > 0010: F3 37 2C 95 .7,. > ] > ] > > > Thanks, > Rajeswari > > On 7/13/20, 2:16 PM, "Kevin Risden" <kris...@apache.org> wrote: > > > > > In local with just certificate and one domain name the SSL > communication > > worked. With multiple DNS and 2 certificates SSL fails with below > exception. > > > > A client keystore by definition can only have a single certificate. A > server keystore can have multiple certificates. The reason being is > that a > client can only be identified by a single certificate. > > Can you share more details about specifically what your solr.in.sh > configs > look like related to keystore/truststore and which files? Specifically > highlight which files have multiple certificates in them. > > It looks like for the Solr internal http client, the client keystore > has > more than one certificate in it and the error is correct. This is more > strict with recent versions of Jetty 9.4.x. Previously this would > silently > fail, but was still incorrect. Now the error is bubbled up so that > there is > no silent misconfigurations. > > Kevin Risden > > > On Mon, Jul 13, 2020 at 4:54 PM Natarajan, Rajeswari < > rajeswari.natara...@sap.com> wrote: > > > I looked at the patch mentioned in the JIRA > > https://issues.apache.org/jira/browse/SOLR-14105 reporting the > below > > issue. I looked at the solr 8.5.1 code base , I see the patch is > applied. > > But still seeing the same exception with different stack trace. The > > initial excsption stacktrace was at > > > > at > > > org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245) > > > > > > Now the exception we encounter is at httpsolrclient creation > > > > > > Caused by: java.lang.RuntimeException: > > java.lang.UnsupportedOperationException: X509ExtendedKeyManager only > > supported on Server > > at > > > org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:223) > > > > I commented the JIRA also. Let me know if this is still an issue. > > > > Thanks, > > Rajeswari > > > > On 7/13/20, 2:03 AM, "Natarajan, Rajeswari" < > rajeswari.natara...@sap.com> > > wrote: > > > > Re-sending to see if anyone encountered had this combination and > > encountered this issue. In local with just certificate and one > domain name > > the SSL communication worked. With multiple DNS and 2 certificates > SSL > > fails with below exception. Below JIRA says it is fixed for > > Http2SolrClient , wondering if this is fixed for http1 solr client > as we > > pass -Dsolr.http1=true . > > > > Thanks, > > Rajeswari > > > > https://issues.apache.org/jira/browse/SOLR-14105 > > > > On 7/6/20, 10:02 PM, "Natarajan, Rajeswari" < > > rajeswari.natara...@sap.com> wrote: > > > > Hi, > > > > We are using Solr 8.5.1 in cloud mode with Java 8. We are > > enabling TLS with http1 (as we get a warning java 8 + solr 8.5 > SSL can’t > > be enabled) and we get below exception > > > > > > > > 2020-07-07 03:58:53.078 ERROR (main) [ ] o.a.s.c.SolrCore > > null:org.apache.solr.common.SolrException: Error instantiating > > shardHandlerFactory class [HttpShardHandlerFactory]: > > java.lang.UnsupportedOperationException: X509ExtendedKeyManager only > > supported on Server > > at > > > org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:56) > > at > > org.apache.solr.core.CoreContainer.load(CoreContainer.java:647) > > at > > > org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:263) > > at > > > org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:183) > > at > > > org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:134) > > at > > > org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:751) > > at > > > java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) > > at > > > java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) > > at > > > java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) > > at > > > java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) > > at > > > org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:744) > > at > > > org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:360) > > at > > > org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445) > > at > > > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409) > > at > > > org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:822) > > at > > > org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275) > > at > > > org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) > > at > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) > > at > > > org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46) > > at > > > org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188) > > at > > > org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:513) > > at > > > org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:154) > > at > > > org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:173) > > at > > > org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:447) > > at > > > org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:66) > > at > > org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:784) > > at > > org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:753) > > at > org.eclipse.jetty.util.Scanner.scan(Scanner.java:641) > > at > org.eclipse.jetty.util.Scanner.doStart(Scanner.java:540) > > at > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) > > at > > > org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:146) > > at > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) > > at > > > org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:599) > > at > > > org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:249) > > at > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) > > at > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) > > at > org.eclipse.jetty.server.Server.start(Server.java:407) > > at > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) > > at > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:100) > > at > org.eclipse.jetty.server.Server.doStart(Server.java:371) > > at > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) > > at > > > org.eclipse.jetty.xml.XmlConfiguration.lambda$main$0(XmlConfiguration.java:1888) > > at java.security.AccessController.doPrivileged(Native > Method) > > at > > > org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1837) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at > org.eclipse.jetty.start.Main.invokeMain(Main.java:218) > > at org.eclipse.jetty.start.Main.start(Main.java:491) > > at org.eclipse.jetty.start.Main.main(Main.java:77) > > Caused by: java.lang.RuntimeException: > > java.lang.UnsupportedOperationException: X509ExtendedKeyManager only > > supported on Server > > at > > > org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:223) > > at > > > org.apache.solr.client.solrj.impl.Http2SolrClient.<init>(Http2SolrClient.java:153) > > at > > > org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:832) > > at > > > org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:321) > > at > > > org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:51) > > ... 50 more > > Caused by: java.lang.UnsupportedOperationException: > > X509ExtendedKeyManager only supported on Server > > at > > > org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1273) > > at > > > org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1255) > > at > > > org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:374) > > at > > > org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245) > > at > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) > > at > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) > > at > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) > > at > > org.eclipse.jetty.client.HttpClient.doStart(HttpClient.java:244) > > at > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) > > at > > > org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:221) > > ... 54 more > > > > > > I see that there is a below bug for this issue and is > resolved. > > So I am not sure what will the cause of the issue. > > > > https://issues.apache.org/jira/browse/SOLR-14105 > > > > > > Thanks, > > Rajeswari > > > > > > > >
