Hi Rob,
When we start tomcat with the date rolled back, we are not seeing any errors at
all. All of the ipa services start up without issue. The problem is in actually
renewing the certs, when we do so we have seen many different errors as we've
been troubleshooting -- mostly this one: `ca-error: Server at
https://<HOSTNAME>/ipa/xml failed request, will retry: 4035 (RPC failed at
server. Request failed with status 500: Non-2xx response from CA REST API:
500. String index out of range:
-1).[02/Aug/2023:00:00:31][ajp-bio-127.0.0.1-8009-exec-2]: EnrollProfile:
populate: begins`
When I restart certmonger after all services up, these are the errors that I am
seeing in the tomcat debug logs:
```
[02/Aug/2023:00:00:31][ajp-bio-127.0.0.1-8009-exec-2]: BasicProfile: populate:
policy setid =serverCertSet
[02/Aug/2023:00:00:31][ajp-bio-127.0.0.1-8009-exec-2]: EnrollDefault: populate:
SubjectNameDefault: start
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1967)
at com.netscape.certsrv.pattern.Pattern.substitute2(Pattern.java:132)
at
com.netscape.cms.profile.def.EnrollDefault.mapPattern(EnrollDefault.java:815)
at
com.netscape.cms.profile.def.SubjectNameDefault.populate(SubjectNameDefault.java:160)
at
com.netscape.cms.profile.def.EnrollDefault.populate(EnrollDefault.java:226)
at
com.netscape.cms.profile.common.BasicProfile.populate(BasicProfile.java:1114)
at
com.netscape.cms.profile.common.EnrollProfile.populate(EnrollProfile.java:2626)
at
com.netscape.cms.servlet.cert.CertProcessor.populateRequests(CertProcessor.java:379)
at
com.netscape.cms.servlet.cert.EnrollmentProcessor.processEnrollment(EnrollmentProcessor.java:188)
at
com.netscape.cms.servlet.cert.EnrollmentProcessor.processEnrollment(EnrollmentProcessor.java:96)
at
com.netscape.cms.servlet.cert.CertRequestDAO.submitRequest(CertRequestDAO.java:197)
at
org.dogtagpki.server.ca.rest.CertRequestService.enrollCert(CertRequestService.java:155)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:280)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:234)
at
org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:221)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
at
org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
at
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
at
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:297)
at
org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55)
at
org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191)
at
org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187)
at java.security.AccessController.doPrivileged(Native Method)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186)
at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
at
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
at
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:260)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237)
at
org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55)
at
org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191)
at
org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187)
at java.security.AccessController.doPrivileged(Native Method)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:498)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:750)
```
This is what we see when we run `getcert list` and `ipa-getcert list`
respectively:
```
Number of certificates and requests being tracked: 9.
Request ID '20190920201259':
status: CA_UNREACHABLE
ca-error: Server at https://<HOSTNAME>/ipa/xml failed request, will
retry: 4035 (RPC failed at server. Request failed with status 500: Non-2xx
response from CA REST API: 500. String index out of range: -1).
stuck: no
key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
CA: IPA
issuer: CN=Certificate Authority,O=<OU>
subject: CN=<HOSTNAME>,O=<OU>
expires: 2023-08-25 18:05:07 UTC
principal name: krbtgt/<OU>@<OU>
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-pkinit-KPKdc
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
track: yes
auto-renew: yes
Request ID '20210908000050':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS FIPS 140-2 Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS FIPS 140-2 Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<OU>
subject: CN=CA Audit,O=<OU>
expires: 2025-07-21 02:36:57 UTC
key usage: digitalSignature,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"auditSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20210908000051':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS FIPS 140-2 Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS FIPS 140-2 Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<OU>
subject: CN=OCSP Subsystem,O=<OU>
expires: 2025-07-21 02:36:17 UTC
key usage: digitalSignature,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20210908000052':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS FIPS 140-2 Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS FIPS 140-2 Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<OU>
subject: CN=CA Subsystem,O=<OU>
expires: 2025-07-21 02:37:17 UTC
key usage: digitalSignature,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"subsystemCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20210908000053':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS FIPS 140-2 Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS FIPS 140-2 Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<OU>
subject: CN=Certificate Authority,O=<OU>
expires: 2039-09-20 20:11:25 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"caSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20210908000054':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<OU>
subject: CN=IPA RA,O=<OU>
expires: 2025-06-26 02:36:15 UTC
key usage: digitalSignature,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20210908000055':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS FIPS 140-2 Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS FIPS 140-2 Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<OU>
subject: CN=<HOSTNAME>,O=<OU>
expires: 2025-07-21 02:36:37 UTC
dns: <HOSTNAME>
key usage: digitalSignature,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"Server-Cert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20210908000056':
status: CA_UNREACHABLE
ca-error: Server at https://<HOSTNAME>/ipa/xml failed request, will
retry: 4035 (RPC failed at server. Request failed with status 500: Non-2xx
response from CA REST API: 500. String index out of range: -1).
stuck: no
key pair storage:
type=NSSDB,location='/etc/dirsrv/slapd-<OU>',nickname='CN=<HOSTNAME>,O=<OU>',token='NSS
FIPS 140-2 Certificate DB',pinfile='/etc/dirsrv/slapd-<OU>/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/dirsrv/slapd-<OU>',nickname='CN=<HOSTNAME>,O=<OU>',token='NSS
FIPS 140-2 Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=<OU>
subject: CN=<HOSTNAME>,O=<OU>
expires: 2023-09-03 18:30:48 UTC
dns: <HOSTNAME>
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv <OU>
track: yes
auto-renew: yes
Request ID '20210908000057':
status: CA_UNREACHABLE
ca-error: Server at https://<HOSTNAME>/ipa/xml failed request, will
retry: 4035 (RPC failed at server. Request failed with status 500: Non-2xx
response from CA REST API: 500. String index out of range: -1).
stuck: no
key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='CN=<HOSTNAME>,O=<OU>',token='NSS
FIPS 140-2 Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='CN=<HOSTNAME>,O=<OU>',token='NSS
FIPS 140-2 Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=<OU>
subject: CN=<HOSTNAME>,O=<OU>
expires: 2023-09-03 18:30:48 UTC
dns: <HOSTNAME>
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
```
```
Number of certificates and requests being tracked: 9.
Request ID '20190920201259':
status: CA_UNREACHABLE
ca-error: Server at https://<HOSTNAME>/ipa/xml failed request, will
retry: 4035 (RPC failed at server. Request failed with status 500: Non-2xx
response from CA REST API: 500. String index out of range: -1).
stuck: no
key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
CA: IPA
issuer: CN=Certificate Authority,O=<OU>
subject: CN=<HOSTNAME>,O=<OU>
expires: 2023-08-25 18:05:07 UTC
principal name: krbtgt/<OU>@<OU>
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-pkinit-KPKdc
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
track: yes
auto-renew: yes
Request ID '20210908000056':
status: CA_UNREACHABLE
ca-error: Server at https://<HOSTNAME>/ipa/xml failed request, will
retry: 4035 (RPC failed at server. Request failed with status 500: Non-2xx
response from CA REST API: 500. String index out of range: -1).
stuck: no
key pair storage:
type=NSSDB,location='/etc/dirsrv/slapd-<OU>',nickname='CN=<HOSTNAME>,O=<OU>',token='NSS
FIPS 140-2 Certificate DB',pinfile='/etc/dirsrv/slapd-<OU>/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/dirsrv/slapd-<OU>',nickname='CN=<HOSTNAME>,O=<OU>',token='NSS
FIPS 140-2 Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=<OU>
subject: CN=<HOSTNAME>,O=<OU>
expires: 2023-09-03 18:30:48 UTC
dns: <HOSTNAME>
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv <OU>
track: yes
auto-renew: yes
Request ID '20210908000057':
status: CA_UNREACHABLE
ca-error: Server at https://<HOSTNAME>/ipa/xml failed request, will
retry: 4035 (RPC failed at server. Request failed with status 500: Non-2xx
response from CA REST API: 500. String index out of range: -1).
stuck: no
key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='CN=<HOSTNAME>,O=<OU>',token='NSS
FIPS 140-2 Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='CN=<HOSTNAME>,O=<OU>',token='NSS
FIPS 140-2 Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=<OU>
subject: CN=<HOSTNAME>,O=<OU>
expires: 2023-09-03 18:30:48 UTC
dns: <HOSTNAME>
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
```
_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/[email protected]
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
