WebSocket session is never closed

2025-05-21 Thread Harri Pesonen 
Hello,

We have a random problem with Apache Tomcat/9.0.100 in Windows, JDK 11.0.13.
We have seen this problem only once so far.
Problem is that WebSocket connection is apparently closed but there is no 
callback to @OnClose handler, which is implemented in @ServerEndpoint class.
Then this WebSocket session lives forever, until Tomcat is restarted.
When our application tries to send async message to this WebSocket, then it 
always fails with this error:

java.io.IOException: An established connection was aborted by the software in 
your host machine
  at java.base/sun.nio.ch.SocketDispatcher.write0(Native Method)
  at java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:51)
  at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113)
  at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:79)
  at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:50)
  at 
java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:462)
  at 
org.apache.tomcat.util.net.SecureNioChannel.flush(SecureNioChannel.java:125)
  at 
org.apache.tomcat.util.net.SecureNioChannel.write(SecureNioChannel.java:883)
  at 
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1684)
  at 
org.apache.tomcat.util.net.SocketWrapperBase$OperationState.start(SocketWrapperBase.java:1077)
  at 
org.apache.tomcat.util.net.SocketWrapperBase.vectoredOperation(SocketWrapperBase.java:1496)
  at 
org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:1422)
  at 
org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:1393)
  at 
org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.doWrite(WsRemoteEndpointImplServer.java:171)
  at 
org.apache.tomcat.websocket.WsRemoteEndpointImplBase.writeMessagePart(WsRemoteEndpointImplBase.java:520)
  at 
org.apache.tomcat.websocket.WsRemoteEndpointImplBase.startMessage(WsRemoteEndpointImplBase.java:411)
  at 
org.apache.tomcat.websocket.WsRemoteEndpointImplBase$TextMessageSendHandler.write(WsRemoteEndpointImplBase.java:844)
  at 
org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendStringByCompletion(WsRemoteEndpointImplBase.java:212)
  at 
org.apache.tomcat.websocket.WsRemoteEndpointAsync.sendText(WsRemoteEndpointAsync.java:46)

I just implemented possible workaround for this problem by calling close() on 
the WebSocket when this IOException happens in the sendText() callback.
Now I am asking, do you think that this workaround will fix the problem, if it 
ever happens again?

javax.websocket.Session#close()

/**
 * Close the current conversation with a normal status code and no reason 
phrase.
 *
 * @throws IOException if there was a connection error closing the connection.
 */
@Override
void close() throws IOException;


Another problem with this WebSocket is that isOpen() always returns true.

javax.websocket.Session#isOpen

/**
 * Return true if and only if the underlying socket is open.
 *
 * @return whether the session is active.
 */
boolean isOpen();

Thanks,
-Harri

VS: WebSocket session is never closed

2025-05-21 Thread Harri Pesonen 
One more thing, we are using Tomcat native.dll:

19:39:09.263 INF> (main) 
(org.apache.catalina.core.AprLifecycleListener#lifecycleEvent) Loaded Apache 
Tomcat Native library [1.3.1] using APR version [1.7.4].
19:39:09.263 INF> (main) 
(org.apache.catalina.core.AprLifecycleListener#lifecycleEvent) APR 
capabilities: IPv6 [true], sendfile [true], accept filters [false], random 
[true], UDS [true].
19:39:09.263 INF> (main) 
(org.apache.catalina.core.AprLifecycleListener#lifecycleEvent) APR/OpenSSL 
configuration: useAprConnector [false], useOpenSSL [true]
19:39:09.274 INF> (main) 
(org.apache.catalina.core.AprLifecycleListener#initializeSSL) OpenSSL 
successfully initialized [OpenSSL 3.0.14 4 Jun 2024]

-Harri

Lähettäjä: Harri Pesonen 
Lähetetty: keskiviikko 21. toukokuuta 2025 12.37
Vastaanottaja: users@tomcat.apache.org 
Aihe: WebSocket session is never closed

Hello,

We have a random problem with Apache Tomcat/9.0.100 in Windows, JDK 11.0.13.
We have seen this problem only once so far.
Problem is that WebSocket connection is apparently closed but there is no 
callback to @OnClose handler, which is implemented in @ServerEndpoint class.
Then this WebSocket session lives forever, until Tomcat is restarted.
When our application tries to send async message to this WebSocket, then it 
always fails with this error:

java.io.IOException: An established connection was aborted by the software in 
your host machine
  at java.base/sun.nio.ch.SocketDispatcher.write0(Native Method)
  at java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:51)
  at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113)
  at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:79)
  at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:50)
  at 
java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:462)
  at 
org.apache.tomcat.util.net.SecureNioChannel.flush(SecureNioChannel.java:125)
  at 
org.apache.tomcat.util.net.SecureNioChannel.write(SecureNioChannel.java:883)
  at 
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1684)
  at 
org.apache.tomcat.util.net.SocketWrapperBase$OperationState.start(SocketWrapperBase.java:1077)
  at 
org.apache.tomcat.util.net.SocketWrapperBase.vectoredOperation(SocketWrapperBase.java:1496)
  at 
org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:1422)
  at 
org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:1393)
  at 
org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.doWrite(WsRemoteEndpointImplServer.java:171)
  at 
org.apache.tomcat.websocket.WsRemoteEndpointImplBase.writeMessagePart(WsRemoteEndpointImplBase.java:520)
  at 
org.apache.tomcat.websocket.WsRemoteEndpointImplBase.startMessage(WsRemoteEndpointImplBase.java:411)
  at 
org.apache.tomcat.websocket.WsRemoteEndpointImplBase$TextMessageSendHandler.write(WsRemoteEndpointImplBase.java:844)
  at 
org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendStringByCompletion(WsRemoteEndpointImplBase.java:212)
  at 
org.apache.tomcat.websocket.WsRemoteEndpointAsync.sendText(WsRemoteEndpointAsync.java:46)

I just implemented possible workaround for this problem by calling close() on 
the WebSocket when this IOException happens in the sendText() callback.
Now I am asking, do you think that this workaround will fix the problem, if it 
ever happens again?

javax.websocket.Session#close()

/**
 * Close the current conversation with a normal status code and no reason 
phrase.
 *
 * @throws IOException if there was a connection error closing the connection.
 */
@Override
void close() throws IOException;


Another problem with this WebSocket is that isOpen() always returns true.

javax.websocket.Session#isOpen

/**
 * Return true if and only if the underlying socket is open.
 *
 * @return whether the session is active.
 */
boolean isOpen();

Thanks,
-Harri

Re: Tcnative-2 PQC support

2025-05-21 Thread Christopher Schultz 

Federico,

On 5/21/25 2:22 PM, federico bustamante wrote:

I've only tried with OpenSSL 3.5, but I suspect it's the same story with
older versions (which, to be honest, wouldn't bother me as much as it being
a specific problem with 3.5).


I'll try reproducing the build process on Windows. We know the build CAN 
be produced on Windows, because, well... we produce convenience binaries 
for Windows with each tcnative release. But there may be some 
environmental nuances that aren't completely documented in our build 
process. We should fix that if it's the case.



I'll try building the DLL on Ubuntu.


Strictly speaking, this is not possible. You may be able to build a 
Linux binary, but creating a Windows DLL on Linux would require that you 
set up a toolchain that is much more complicated than just doing it on 
Windows in the first place.


-chris


On Wed, May 21, 2025, 13:35 Christopher Schultz <
ch...@christopherschultz.net> wrote:


Federico,

On 5/21/25 12:27 PM, Christopher Schultz wrote:

Federico,

On 5/21/25 10:58 AM, federico bustamante wrote:

Hi!
Did anyone have any luck building tcnative-2.dll with Openssl 3.5 to add
Post Quantum Cryptography support?
I tried following the steps in the wiki, but didn't have any luck (tried
with cmake and nmake).


Have you been able to compile tcnative-2.dll with earlier versions of
OpenSSL? I'm asking to see if you are having trouble with building
tcnative in general, or if OpenSSL 3.5 is what is causing the problem.


FYI I just tried building libtcnative on MacOS against OpenSSL 3.5 and
the build was successful. I believe the build process for Windows is ...
complex compared to *NIX-style builds.

But I wanted to confirm that tcnative didn't have any issues with
building against OpenSSL 3.5 in general, and that seems not to be the case.

So the problem must be with the Windows build process itself.

-chris


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org







-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: WebSocket session is never closed

2025-05-21 Thread Mark Thomas 

On 21/05/2025 10:37, Harri Pesonen wrote:

Hello,

We have a random problem with Apache Tomcat/9.0.100 in Windows, JDK 11.0.13.
We have seen this problem only once so far.
Problem is that WebSocket connection is apparently closed but there is no 
callback to @OnClose handler, which is implemented in @ServerEndpoint class.


That shouldn't happen. If you ever figure out what triggers this issue, 
do let us know. There might be an edge case we need to fix in the error 
handling.





I just implemented possible workaround for this problem by calling close() on 
the WebSocket when this IOException happens in the sendText() callback.
Now I am asking, do you think that this workaround will fix the problem, if it 
ever happens again?


It should do yes. You might want to call:

Session.close(new CloseReason(CLOSED_ABNORMALLY, "IOE on write"))

rather than

Session.close()

to signal to Tomcat that the session is being closed due to an error 
condition. It should step additional error events being triggered. But 
both methods should work.


Mark



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tcnative-2 PQC support

2025-05-21 Thread Mark Thomas 

On 21/05/2025 23:04, federico bustamante wrote:

Yes, I don't have high hopes on make in it work on Ubuntu, but I thought of
giving it a try using mingw-64.
I'll report back.


I've been building the Tomcat Native binaries for Windows for a while. 
I'll try with 3.5 and report back.


I'll also start a discussion on dev@ about switching the convenience 
builds to use OpenSSL 3.5 since that is the new LTS version.


Mark



Fede

On Wed, May 21, 2025, 18:20 Christopher Schultz <
ch...@christopherschultz.net> wrote:


Federico,

On 5/21/25 2:22 PM, federico bustamante wrote:

I've only tried with OpenSSL 3.5, but I suspect it's the same story with
older versions (which, to be honest, wouldn't bother me as much as it

being

a specific problem with 3.5).


I'll try reproducing the build process on Windows. We know the build CAN
be produced on Windows, because, well... we produce convenience binaries
for Windows with each tcnative release. But there may be some
environmental nuances that aren't completely documented in our build
process. We should fix that if it's the case.


I'll try building the DLL on Ubuntu.


Strictly speaking, this is not possible. You may be able to build a
Linux binary, but creating a Windows DLL on Linux would require that you
set up a toolchain that is much more complicated than just doing it on
Windows in the first place.

-chris


On Wed, May 21, 2025, 13:35 Christopher Schultz <
ch...@christopherschultz.net> wrote:


Federico,

On 5/21/25 12:27 PM, Christopher Schultz wrote:

Federico,

On 5/21/25 10:58 AM, federico bustamante wrote:

Hi!
Did anyone have any luck building tcnative-2.dll with Openssl 3.5 to

add

Post Quantum Cryptography support?
I tried following the steps in the wiki, but didn't have any luck

(tried

with cmake and nmake).


Have you been able to compile tcnative-2.dll with earlier versions of
OpenSSL? I'm asking to see if you are having trouble with building
tcnative in general, or if OpenSSL 3.5 is what is causing the problem.


FYI I just tried building libtcnative on MacOS against OpenSSL 3.5 and
the build was successful. I believe the build process for Windows is ...
complex compared to *NIX-style builds.

But I wanted to confirm that tcnative didn't have any issues with
building against OpenSSL 3.5 in general, and that seems not to be the

case.


So the problem must be with the Windows build process itself.

-chris


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org







-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org







-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Adding a CRL to certificate client authentication causes connector to stop responding to all requests having a client certificate.

2025-05-21 Thread Christopher Schultz 

Alex,

On 5/21/25 1:26 PM, My Subs wrote:

To test this, I set
certificateRevocationListPath to the directory having the CRL file;
changed to certificateVerification="optional"; and
downgraded to HTTP 1.1 (as mentioned, "optional" does not
work with HTTP 2).  The result is that requests without a client
certificate get a 200 OK response.


Good.


However, *all* requests having a client certificate (valid or
revoked) are rejected.


Okay.


If the client certificate is revoked, it is correctly identified as
such.  Firefox displays message "SSL peer rejected your
certificate as revoked.  Error code: SSL_ERROR_REVOKED_CERT_ALERT".


Good!


If the client certificate is not revoked, the connection fails with
Firefox displaying an unexpected message: "Peer does not
recognize and trust the CA that issued your certificate.  Error code:
SSL_ERROR_UNKNOWN_CA_ALERT".


Obviously, not good.

But I wonder if this is a problem with adding the CRL (which looks like 
it's working) versus the trust store. But...


But that same issuing CA is perfectly recognized when the 
certificateRevocationListPath attribute is removed from the 
connector.

Yeah, I was afraid you'd say that.


Requests with a non-revoked client certificate get a 200 OK
response, and that certificate is made visible to servlets as a
java.security.cert.X509Certificate object. This holds both for
"optional" and> "required" verification.

> > It's CRL checking what triggers the error.


I'm
not sure whether the message "No client certificate
CA names sent" means a problem with the trust store.  As you can
see from my connector, caCertificatePath is set to the trust store directory. It
contains the certificate of the root CA, which signed the certificate
of the subordinate CA.  The client certificates used in the above
test are signed by that subordinate CA.  And as
shown, those are correctly verified when no CRL is configured.


Is the client sending just their own cert, or also the chain? It would 
be unusual for the client to send a chain, but instead just sends their 
own leaf certificate. I'm assuming that both the root and the 
intermediate / subordinate cert are both in the trust store. Is that 
correct?



  > If you do not configure the CRL, are any CA certs listed in
this output?

No.  Message "No client certificate CA names sent" is still there.
The full output is:

> > [snip]
>

---

No client certificate CA names sent


Okay.

Sounds like it's time for me to try to reproduce this.

Can you please try one more thing for me?

Can you create a self-signed certificate, and use ONLY that self-signed 
certificate in your trust store, and have the client send that to the 
server?


It will be easier for me if I just have to deal with a single cert on 
either side of the connection rather than set up a whole signing 
authority, etc. before I'm able to reproduce it.


-chris


 On Wed, 21 May 2025 07:19:06 -0500 Christopher Schultz 
 wrote ---



Alex,
  
On 5/19/25 5:37 PM, My Subs wrote:

  > I'm using Ubuntu 20.04 with OpenSSL 1.1.1f.
  
Okay.
  

In your earlier message, you had a different configuration. This time you haven't 
specified the class name in the "protocol" attribute. Which one are you 
actually using?




I did change the connector configuration because when updating to Tomcat 10.1.40, I could 
no longer use class Http11AprProtocol on the protocol attribute (I learned then that the 
APR connector had been deprecated).  So, I set it to "HTTP/1.1" to get 
automatic selection of the JSSE OpenSSL implementation.



I also set certificateVerification="required" because I found in the logs an error 
message saying that "optional" does not work with HTTP/2.



Besides, I'm now testing on Tomcat 11.0.6.
  
Okay.
  

So the above configuration works for all requests that do not try to send a 
client certificate during the handshake? It's only when you try to send a 
client certificate that things stop working?




Actually, the configuration works seamlessly with client certificate 
verification for as long as no CRL verification is set up.  Client certificates 
are accepted and made visible to servlets.



However, as soon as I set on the , attribute 
certificateRevocationListFile to the CRL file, or certificateRevocationListPath to 
the directory containing the CRL file (properly c_rehashed), all client certificates 
are rejected.
  
When you have configured a CRL, are *all* requests rejected, or only

those which include a client certificate during the handshake? I see you
have configured certificateVerification="required" so maybe there are no
modes of operation where client certificates are used.
  
I'm trying to understand whether this is a problem with the whole setup

when CRLs are added, or only a problem when a client certificate is
actively being checked against the CRL.
  

When they stop working, does that mean that no more requests are accepted and 
processed, or is it that handshakes fail with client certs but handshakes 
w

Re: Tcnative-2 PQC support

2025-05-21 Thread federico bustamante 
Yes, I don't have high hopes on make in it work on Ubuntu, but I thought of
giving it a try using mingw-64.
I'll report back.

Fede

On Wed, May 21, 2025, 18:20 Christopher Schultz <
ch...@christopherschultz.net> wrote:

> Federico,
>
> On 5/21/25 2:22 PM, federico bustamante wrote:
> > I've only tried with OpenSSL 3.5, but I suspect it's the same story with
> > older versions (which, to be honest, wouldn't bother me as much as it
> being
> > a specific problem with 3.5).
>
> I'll try reproducing the build process on Windows. We know the build CAN
> be produced on Windows, because, well... we produce convenience binaries
> for Windows with each tcnative release. But there may be some
> environmental nuances that aren't completely documented in our build
> process. We should fix that if it's the case.
>
> > I'll try building the DLL on Ubuntu.
>
> Strictly speaking, this is not possible. You may be able to build a
> Linux binary, but creating a Windows DLL on Linux would require that you
> set up a toolchain that is much more complicated than just doing it on
> Windows in the first place.
>
> -chris
>
> > On Wed, May 21, 2025, 13:35 Christopher Schultz <
> > ch...@christopherschultz.net> wrote:
> >
> >> Federico,
> >>
> >> On 5/21/25 12:27 PM, Christopher Schultz wrote:
> >>> Federico,
> >>>
> >>> On 5/21/25 10:58 AM, federico bustamante wrote:
>  Hi!
>  Did anyone have any luck building tcnative-2.dll with Openssl 3.5 to
> add
>  Post Quantum Cryptography support?
>  I tried following the steps in the wiki, but didn't have any luck
> (tried
>  with cmake and nmake).
> >>>
> >>> Have you been able to compile tcnative-2.dll with earlier versions of
> >>> OpenSSL? I'm asking to see if you are having trouble with building
> >>> tcnative in general, or if OpenSSL 3.5 is what is causing the problem.
> >>
> >> FYI I just tried building libtcnative on MacOS against OpenSSL 3.5 and
> >> the build was successful. I believe the build process for Windows is ...
> >> complex compared to *NIX-style builds.
> >>
> >> But I wanted to confirm that tcnative didn't have any issues with
> >> building against OpenSSL 3.5 in general, and that seems not to be the
> case.
> >>
> >> So the problem must be with the Windows build process itself.
> >>
> >> -chris
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>
> >>
> >
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Adding a CRL to certificate client authentication causes connector to stop responding to all requests having a client certificate.

2025-05-21 Thread My Subs 
Chris,



> Is the client sending just their own cert, or also the chain?
It would 
 be unusual for the client to send a chain, but instead
just sends their 
 own leaf certificate. I'm assuming that both
the root and the 
 intermediate / subordinate cert are both in
the trust store. Is that 
 correct?


The client is
sending both the leaf and the intermediate certificates (both are in
the PKCS12 file imported into Firefox).  That's the only way for the
chain of trust to work, because only the root certificate is in the
trust store (well, that's how it's supposed to be, as far as I
understand).  And it works fine until the CRL comes into play.



I also tested adding
the subordinate CA's certificate to the trust store.  No change. 
With no CRL configured, works alright; but when it is configured, I
get the same errors.  Revoked client cert is rejected as such, and
valid client cert is rejected with SSL_ERROR_UNKNOWN_CA_ALERT.




> Can
you please try one more thing for me? 
 
 > Can
you create a self-signed certificate, and use ONLY that self-signed 
 certificate in your trust store, and have the client send that
to the 
 server? 
 
 > It will be easier
for me if I just have to deal with a single cert on 
 either side
of the connection rather than set up a whole signing 
 authority,
etc. before I'm able to reproduce it. 
 


Done.  Removed root CA certificate from trust store, and left only a
self-signed certificate.  The result is exactly the same.  With no
CRL checking, it works fine.  Add CRL checking, and I get: "Peer
does not recognize and trust the CA that issued your certificate. 
Error code: SSL_ERROR_UNKNOWN_CA_ALERT"



Guess the finger
increasingly points to the CRL checking code.



Alex





Sent using Zoho Mail








 On Wed, 21 May 2025 16:29:10 -0500 Christopher Schultz 
 wrote ---



Alex, 
 
On 5/21/25 1:26 PM, My Subs wrote: 
> To test this, I set 
> certificateRevocationListPath to the directory having the CRL file; 
> changed to certificateVerification="optional"; and 
> downgraded to HTTP 1.1 (as mentioned, "optional" does not 
> work with HTTP 2).  The result is that requests without a client 
> certificate get a 200 OK response. 
 
Good. 
 
> However, *all* requests having a client certificate (valid or 
> revoked) are rejected. 
 
Okay. 
 
> If the client certificate is revoked, it is correctly identified as 
> such.  Firefox displays message "SSL peer rejected your 
> certificate as revoked.  Error code: SSL_ERROR_REVOKED_CERT_ALERT". 
 
Good! 
 
> If the client certificate is not revoked, the connection fails with 
> Firefox displaying an unexpected message: "Peer does not 
> recognize and trust the CA that issued your certificate.  Error code: 
> SSL_ERROR_UNKNOWN_CA_ALERT". 
 
Obviously, not good. 
 
But I wonder if this is a problem with adding the CRL (which looks like 
it's working) versus the trust store. But... 
 
> But that same issuing CA is perfectly recognized when the 
> certificateRevocationListPath attribute is removed from the 
> connector. 
Yeah, I was afraid you'd say that. 
 
> Requests with a non-revoked client certificate get a 200 OK 
> response, and that certificate is made visible to servlets as a 
> java.security.cert.X509Certificate object. This holds both for 
> "optional" and> "required" verification. 
 > > It's CRL checking what triggers the error. 
> 
> I'm 
> not sure whether the message "No client certificate 
> CA names sent" means a problem with the trust store.  As you can 
> see from my connector, caCertificatePath is set to the trust store directory. 
> It 
> contains the certificate of the root CA, which signed the certificate 
> of the subordinate CA.  The client certificates used in the above 
> test are signed by that subordinate CA.  And as 
> shown, those are correctly verified when no CRL is configured. 
 
Is the client sending just their own cert, or also the chain? It would 
be unusual for the client to send a chain, but instead just sends their 
own leaf certificate. I'm assuming that both the root and the 
intermediate / subordinate cert are both in the trust store. Is that 
correct? 
 
>   > If you do not configure the CRL, are any CA certs listed in 
> this output? 
> 
> No.  Message "No client certificate CA names sent" is still there. 
> The full output is: 
 > > [snip] 
 > 
> --- 
> 
> No client certificate CA names sent 
 
Okay. 
 
Sounds like it's time for me to try to reproduce this. 
 
Can you please try one more thing for me? 
 
Can you create a self-signed certificate, and use ONLY that self-signed 
certificate in your trust store, and have the client send that to the 
server? 
 
It will be easier for me if I just have to deal with a single cert on 
either side of the connection rather than set up a whole signing 
authority, etc. before I'm able to reproduce it. 
 
-chris 
 
>  On Wed, 21 May 2025 07:19:06 -0500 Christopher Schultz 
>  wrote --- 
> 
> 
> 
> Alex, 
> 
> On 5/19/

Tomcat 9, ClassCast exception

2025-05-21 Thread Zdeněk Henek 
Hello,

I am getting these errors in one of our systems:

java.lang.ClassCastException: class com.sun.mail.handlers.text_html cannot
be cast to class javax.activation.DataContentHandler
(com.sun.mail.handlers.text_html is in unnamed module of loader
org.apache.catalina.loader.ParallelWebappClassLoader @6c5398e5;
javax.activation.DataContentHandler is in unnamed module of loader
org.apache.catalina.loader.ParallelWebappClassLoader @719f9efd)
at
javax.activation.MailcapCommandMap.getDataContentHandler(MailcapCommandMap.java:596)
at
javax.activation.MailcapCommandMap.createDataContentHandler(MailcapCommandMap.java:550)
at
javax.activation.DataHandler.getDataContentHandler(DataHandler.java:611)
at javax.activation.DataHandler.writeTo(DataHandler.java:315)
at javax.mail.internet.MimeUtility.getEncoding(MimeUtility.java:340)
at
javax.mail.internet.MimeBodyPart.updateHeaders(MimeBodyPart.java:1575)
at
javax.mail.internet.MimeMessage.updateHeaders(MimeMessage.java:2271)
at
javax.mail.internet.MimeMessage.saveChanges(MimeMessage.java:2231)
at javax.mail.Transport.send(Transport.java:123)


The system runs on Apache Tomcat 9.0.102, OpenJDK Runtime Environment
Temurin-21.0.6+7 (build 21.0.6+7-LTS), RHEL 9.5 Linux

The functionality has a number of threads in the thread pool and only one
of the threads is causing this issue. Other threads are working as expected.

I am aware of duplicated jar files (even the same jar file version) in
different classloaders or even in the same classloader. I understand the
classes could be in jars with different names. I have tool to review all
jar files and report
jar files having the same classes.

Classloaders are configured this way:

current Tomcat classloaders configuration conf/catalina.properties

common.loader="${catalina.base}/lib","${catalina.base}/lib/.jar","${catalina.home}/lib","${catalina.home}/lib/.jar"

server an shared loaders are empty
server.loader=
shared.loader=

The class com.sun.mail.handlers.text_html is in jar file with name
 javax.mail-1.6.2.jar and the  javax.activation.DataContentHandler
activation-1.1.jar
Both jars are in webapplication in WEB-INF/lib directory
There is multiple web applications with these jar files, but these jar
files are not in /lib directory

The jars are in same war application but in the exception they are in
different ParallelWebappClassLoaders
ParallelWebappClassLoader @6c5398e5 and ParallelWebappClassLoader @719f9efd

I have run a heap dump of the application and listed all
org.apache.catalina.loader.ParallelWebappClassLoader instances.
There are 8 objects but none has object id 6c5398e5 or 719f9efd . I tried
to search objects using these object ids in MAT, but they are not available
even when I keep unreachable objects. The object ids are the same every
time in the exceptions.

I don't see any errors in catalina.out or in localhost.log
files, and the above is the only error I get. System runs stable. We don't
use hot redeployment in this server.

The system CLASSPATH variable is not pointing to any jar files.

Any idea what is going on here? Why does this happen?
The system with the error is still running. I could collect more
information if needed.

Thank you.

Regards,
Zdenek Henek

Re: Tomcat 9, ClassCast exception

2025-05-21 Thread Zdeněk Henek 
I log classes when loaded into a separate log file.
I am sure classes are loaded from expected jar files
The files are 4 times because I have four web applications in the system.

$ grep javax.activation.DataContentHandler classloaded.log
[5436.540s][info][class,load] javax.activation.DataContentHandler source:
file://webapps/app1/WEB-INF/lib/activation-1.1.jar
[8903.600s][info][class,load] javax.activation.DataContentHandler source:
file://webapps/app2/WEB-INF/lib/activation-1.1.jar
[36283.268s][info][class,load] javax.activation.DataContentHandler source:
file://webapps/app3/WEB-INF/lib/activation-1.1.jar
[558992.846s][info][class,load] javax.activation.DataContentHandler source:
file:/mnt/app/tomcat/webapps/app4/WEB-INF/lib/activation-1.1.jar


  (0)
$ grep com.sun.mail.handlers.text_html classloaded.log
[5436.558s][info][class,load] com.sun.mail.handlers.text_html source:
file://webapps/app1/WEB-INF/lib/javax.mail-1.6.2.jar
[8903.600s][info][class,load] com.sun.mail.handlers.text_html source:
file://webapps/app2/WEB-INF/lib/javax.mail-1.6.2.jar
[36283.283s][info][class,load] com.sun.mail.handlers.text_html source:
file://webapps/app3/WEB-INF/lib/javax.mail-1.6.2.jar
[558992.864s][info][class,load] com.sun.mail.handlers.text_html source:
file://webapps/app4/WEB-INF/lib/javax.mail-1.6.2.jar

Regards,
Zdenek Henek

On Wed, May 21, 2025 at 12:44 PM Zdeněk Henek  wrote:

> Hello,
>
> I am getting these errors in one of our systems:
>
> java.lang.ClassCastException: class com.sun.mail.handlers.text_html cannot
> be cast to class javax.activation.DataContentHandler
> (com.sun.mail.handlers.text_html is in unnamed module of loader
> org.apache.catalina.loader.ParallelWebappClassLoader @6c5398e5;
> javax.activation.DataContentHandler is in unnamed module of loader
> org.apache.catalina.loader.ParallelWebappClassLoader @719f9efd)
> at
> javax.activation.MailcapCommandMap.getDataContentHandler(MailcapCommandMap.java:596)
> at
> javax.activation.MailcapCommandMap.createDataContentHandler(MailcapCommandMap.java:550)
> at
> javax.activation.DataHandler.getDataContentHandler(DataHandler.java:611)
> at javax.activation.DataHandler.writeTo(DataHandler.java:315)
> at
> javax.mail.internet.MimeUtility.getEncoding(MimeUtility.java:340)
> at
> javax.mail.internet.MimeBodyPart.updateHeaders(MimeBodyPart.java:1575)
> at
> javax.mail.internet.MimeMessage.updateHeaders(MimeMessage.java:2271)
> at
> javax.mail.internet.MimeMessage.saveChanges(MimeMessage.java:2231)
> at javax.mail.Transport.send(Transport.java:123)
>
>
> The system runs on Apache Tomcat 9.0.102, OpenJDK Runtime Environment
> Temurin-21.0.6+7 (build 21.0.6+7-LTS), RHEL 9.5 Linux
>
> The functionality has a number of threads in the thread pool and only one
> of the threads is causing this issue. Other threads are working as expected.
>
> I am aware of duplicated jar files (even the same jar file version) in
> different classloaders or even in the same classloader. I understand the
> classes could be in jars with different names. I have tool to review all
> jar files and report
> jar files having the same classes.
>
> Classloaders are configured this way:
>
> current Tomcat classloaders configuration conf/catalina.properties
>
>
> common.loader="${catalina.base}/lib","${catalina.base}/lib/.jar","${catalina.home}/lib","${catalina.home}/lib/.jar"
>
> server an shared loaders are empty
> server.loader=
> shared.loader=
>
> The class com.sun.mail.handlers.text_html is in jar file with name
>  javax.mail-1.6.2.jar and the  javax.activation.DataContentHandler
> activation-1.1.jar
> Both jars are in webapplication in WEB-INF/lib directory
> There is multiple web applications with these jar files, but these jar
> files are not in /lib directory
>
> The jars are in same war application but in the exception they are in
> different ParallelWebappClassLoaders
> ParallelWebappClassLoader @6c5398e5 and ParallelWebappClassLoader @719f9efd
>
> I have run a heap dump of the application and listed all
> org.apache.catalina.loader.ParallelWebappClassLoader instances.
> There are 8 objects but none has object id 6c5398e5 or 719f9efd . I tried
> to search objects using these object ids in MAT, but they are not available
> even when I keep unreachable objects. The object ids are the same every
> time in the exceptions.
>
> I don't see any errors in catalina.out or in localhost.log
> files, and the above is the only error I get. System runs stable. We don't
> use hot redeployment in this server.
>
> The system CLASSPATH variable is not pointing to any jar files.
>
> Any idea what is going on here? Why does this happen?
> The system with the error is still running. I could collect more
> information if needed.
>
> Thank you.
>
> Regards,
> Zdenek Henek
>

Re: Tcnative-2 PQC support

2025-05-21 Thread Christopher Schultz 

Federico,

On 5/21/25 10:58 AM, federico bustamante wrote:

Hi!
Did anyone have any luck building tcnative-2.dll with Openssl 3.5 to add
Post Quantum Cryptography support?
I tried following the steps in the wiki, but didn't have any luck (tried
with cmake and nmake).


Have you been able to compile tcnative-2.dll with earlier versions of 
OpenSSL? I'm asking to see if you are having trouble with building 
tcnative in general, or if OpenSSL 3.5 is what is causing the problem.


-chris


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tcnative-2 PQC support

2025-05-21 Thread Christopher Schultz 

Federico,

On 5/21/25 12:27 PM, Christopher Schultz wrote:

Federico,

On 5/21/25 10:58 AM, federico bustamante wrote:

Hi!
Did anyone have any luck building tcnative-2.dll with Openssl 3.5 to add
Post Quantum Cryptography support?
I tried following the steps in the wiki, but didn't have any luck (tried
with cmake and nmake).


Have you been able to compile tcnative-2.dll with earlier versions of 
OpenSSL? I'm asking to see if you are having trouble with building 
tcnative in general, or if OpenSSL 3.5 is what is causing the problem.


FYI I just tried building libtcnative on MacOS against OpenSSL 3.5 and 
the build was successful. I believe the build process for Windows is ... 
complex compared to *NIX-style builds.


But I wanted to confirm that tcnative didn't have any issues with 
building against OpenSSL 3.5 in general, and that seems not to be the case.


So the problem must be with the Windows build process itself.

-chris


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 9, ClassCast exception

2025-05-21 Thread Mark Thomas 

On 21/05/2025 13:44, Zdeněk Henek wrote:

Hello,

I am getting these errors in one of our systems:

java.lang.ClassCastException: class com.sun.mail.handlers.text_html cannot
be cast to class javax.activation.DataContentHandler
(com.sun.mail.handlers.text_html is in unnamed module of loader
org.apache.catalina.loader.ParallelWebappClassLoader @6c5398e5;
javax.activation.DataContentHandler is in unnamed module of loader
org.apache.catalina.loader.ParallelWebappClassLoader @719f9efd)





Any idea what is going on here?


Looks like javax.activation.DataContentHandler has been loaded by a web 
application and then registered with something that has wider visibility 
- probably JVM wide.


That causes problems when another web application tries to use it.


Why does this happen?


If is a form of memory leak.


The system with the error is still running. I could collect more
information if needed.


Remove  javax.mail-1.6.2.jar and activation-1.1.jar from all web 
applications.


Add those JARs to $CATALINA_BASE/lib

That way when they get loaded they'll be loaded by Tomcat's common 
loader and visible to all web applications.


Mark


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Adding a CRL to certificate client authentication causes connector to stop responding to all requests having a client certificate.

2025-05-21 Thread My Subs 
Hello Christopher,



I'm answering below.



>When you have configured a CRL, are *all* requests rejected,
or only 
 those which include a client certificate during the
handshake? I see you 
 have configured
certificateVerification="required" so maybe there are no 
 modes of operation where client certificates are used. 
 
 >I'm
trying to understand whether this is a problem with the whole setup 
 when CRLs are added, or only a problem when a client
certificate is 
 actively being checked against the CRL.



I believe you are
right in suspecting that it is "only a problem when a client
certificate is 

 actively being checked against the CRL".




To test this, I set
certificateRevocationListPath to the directory having the CRL file;
changed to certificateVerification="optional"; and
downgraded to HTTP 1.1 (as mentioned, "optional" does not
work with HTTP 2).  The result is that requests without a client
certificate get a 200 OK response.



However, *all*
requests having a client certificate (valid or revoked) are rejected.



If the client certificate is revoked, it is correctly identified as
such.  Firefox displays message "SSL peer rejected your
certificate as revoked.  Error code: SSL_ERROR_REVOKED_CERT_ALERT".



If the client certificate is not revoked, the connection fails with
Firefox displaying an unexpected message: "Peer does not
recognize and trust the CA that issued your certificate.  Error code:
SSL_ERROR_UNKNOWN_CA_ALERT".



But that same
issuing CA is perfectly recognized when the
certificateRevocationListPath attribute is removed from the
connector.  Requests with a non-revoked client certificate get a 200
OK response, and that certificate is made visible to servlets as a 
java.security.cert.X509Certificate object.  This holds both for "optional" and
"required" verification.



It's CRL checking
what triggers the error.



> Does the
connection actually hang, or do you simply get a failed (but 
 complete) handshake?


From
the above, I'd say that the handshake is failed, but complete, since
FF displays those error messages.



> This
looks like a problem. Do you have the trusted certificates 
 configured correctly? I would have expected Tomcat to send a
list of 
 acceptable certificates back to the client. That's not
strictly required 
 by the TLS spec, but it's handy for debugging
like this.



I'm
not sure whether the message "No client certificate
CA names sent" means a problem with the trust store.  As you can
see from my connector, caCertificatePath is set to the trust store directory. It
contains the certificate of the root CA, which signed the certificate
of the subordinate CA.  The client certificates used in the above
test are signed by that subordinate CA.  And as
shown, those are correctly verified when no CRL is configured.


 
 > If you do not configure the CRL, are any CA certs listed in
this output? 
 

 > I haven't used TLS with
Tomcat much, so I'm not entirely sure what to 
 expect. 
 


No.  Message "No
client certificate CA names sent" is still there.   The full
output is:



CONNECTED(0003)

Can't use
SSL_get_servername

depth=0 CN =
localhost

verify
error:num=18:self signed certificate

verify return:1

depth=0 CN =
localhost

verify return:1

---

Certificate chain

0 s:CN = localhost

i:CN = localhost

-BEGIN
CERTIFICATE-

MIIBfzCCASWgAwIBAgIUBtGtuw6cAZvC560e0RVjnI/+tnwwCgYIKoZIzj0EAwIw

FDESMBAGA1UEAwwJbG9jYWxob3N0MCAXDTI1MDQxNjIyNTg0OFoYDzIxMjUwMzIz

MjI1ODQ4WjAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwWTATBgcqhkjOPQIBBggqhkjO

PQMBBwNCAASWvdHleExdyTqn+wXgNY3XueCLjkkpbrtVhw8lB4DsmkTJDUCdszZX

9ElKT01bP10cX+mrinNNEtgKFPBwcTCXo1MwUTAdBgNVHQ4EFgQUJIQfq2nU9T2J

uexYvw1bqosji6cwHwYDVR0jBBgwFoAUJIQfq2nU9T2JuexYvw1bqosji6cwDwYD

VR0TAQH/BAUwAwEB/zAKBggqhkjOPQQDAgNIADBFAiAUmPONFAU4ThvidgLnlXiu

7XElAkAGfmlXKkN0DgJWwAIhAPkF8ngCXY4G7Y2obrGUS2u80p06O2ZYFtXyrM3+

UuRN

-END
CERTIFICATE-

---

Server certificate

subject=CN =
localhost



issuer=CN =
localhost



---

No client
certificate CA names sent

Requested Signature
Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:E

d448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:

RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224

Shared Requested
Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed

25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+

SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512

Peer signing digest:
SHA256

Peer signature type:
ECDSA

Server Temp Key:
X25519, 253 bits

---

SSL handshake has
read 824 bytes and written 393 bytes

Verification error:
self signed certificate

---

New, TLSv1.3, Cipher
is TLS_AES_256_GCM_SHA384

Server public key is
256 bit

Secure Renegotiation
IS NOT supported

Compression: NONE

Expansion: NONE

No ALPN negotiated

Early data was not
sent

Verify return code:
18 (self signed certificate)

---

---

Post-Handshake New
Session Ticket arrived:

SSL-Session:

Proto

Tcnative-2 PQC support

2025-05-21 Thread federico bustamante 
Hi!
Did anyone have any luck building tcnative-2.dll with Openssl 3.5 to add
Post Quantum Cryptography support?
I tried following the steps in the wiki, but didn't have any luck (tried
with cmake and nmake).
Thanks!

Re: Adding a CRL to certificate client authentication causes connector to stop responding to all requests having a client certificate.

2025-05-21 Thread Christopher Schultz 

Alex,

On 5/19/25 5:37 PM, My Subs wrote:
> I'm using Ubuntu 20.04 with OpenSSL 1.1.1f.

Okay.


In your earlier message, you had a different configuration. This time you haven't 
specified the class name in the "protocol" attribute. Which one are you 
actually using?




I did change the connector configuration because when updating to Tomcat 10.1.40, I could 
no longer use class Http11AprProtocol on the protocol attribute (I learned then that the 
APR connector had been deprecated).  So, I set it to "HTTP/1.1" to get 
automatic selection of the JSSE OpenSSL implementation.



I also set certificateVerification="required" because I found in the logs an error 
message saying that "optional" does not work with HTTP/2.



Besides, I'm now testing on Tomcat 11.0.6.


Okay.


So the above configuration works for all requests that do not try to send a 
client certificate during the handshake? It's only when you try to send a 
client certificate that things stop working?




Actually, the configuration works seamlessly with client certificate 
verification for as long as no CRL verification is set up.  Client certificates 
are accepted and made visible to servlets.



However, as soon as I set on the , attribute 
certificateRevocationListFile to the CRL file, or certificateRevocationListPath to 
the directory containing the CRL file (properly c_rehashed), all client certificates 
are rejected.


When you have configured a CRL, are *all* requests rejected, or only 
those which include a client certificate during the handshake? I see you 
have configured certificateVerification="required" so maybe there are no 
modes of operation where client certificates are used.


I'm trying to understand whether this is a problem with the whole setup 
when CRLs are added, or only a problem when a client certificate is 
actively being checked against the CRL.



When they stop working, does that mean that no more requests are accepted and 
processed, or is it that handshakes fail with client certs but handshakes 
without client certs work okay?




Since certificateVerification is now set to "required", it means that 
handshakes fail with client certs, and therefore, there is no access at all.


Does the connection actually hang, or do you simply get a failed (but 
complete) handshake?



If you connect to your server like this, what does the output look like:


$ openssl s_client -showcerts -connect https://host/whatever



I get the following:

   


     CONNECTED(0003)

     Can't use SSL_get_servername

     depth=0 CN = localhost

     verify error:num=18:self signed certificate

     verify return:1

     depth=0 CN = localhost

     verify return:1

     ---

     Certificate chain

  0 s:CN = localhost

     i:CN = localhost

     -BEGIN CERTIFICATE-

     MIIBfzCCASWgAwIBAgIUBtGtuw6cAZvC560e0RVjnI/+tnwwCgYIKoZIzj0EAwIw

     FDESMBAGA1UEAwwJbG9jYWxob3N0MCAXDTI1MDQxNjIyNTg0OFoYDzIxMjUwMzIz

     MjI1ODQ4WjAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwWTATBgcqhkjOPQIBBggqhkjO

     PQMBBwNCAASWvdHleExdyTqn+wXgNY3XueCLjkkpbrtVhw8lB4DsmkTJDUCdszZX

     9ElKT01bP10cX+mrinNNEtgKFPBwcTCXo1MwUTAdBgNVHQ4EFgQUJIQfq2nU9T2J

     uexYvw1bqosji6cwHwYDVR0jBBgwFoAUJIQfq2nU9T2JuexYvw1bqosji6cwDwYD

     VR0TAQH/BAUwAwEB/zAKBggqhkjOPQQDAgNIADBFAiAUmPONFAU4ThvidgLnlXiu

     7XElAkAGfmlXKkN0DgJWwAIhAPkF8ngCXY4G7Y2obrGUS2u80p06O2ZYFtXyrM3+

     UuRN

     -END CERTIFICATE-

     ---

     Server certificate

     subject=CN = localhost




     issuer=CN = localhost




     ---

     No client certificate CA names sent


This looks like a problem. Do you have the trusted certificates 
configured correctly? I would have expected Tomcat to send a list of 
acceptable certificates back to the client. That's not strictly required 
by the TLS spec, but it's handy for debugging like this.


If you do not configure the CRL, are any CA certs listed in this output?

I haven't used TLS with Tomcat much, so I'm not entirely sure what to 
expect.



Just to be clear, this is my current :



     

     

     

     

     

     



Thanks for posting the whole thing.

-chris


 On Fri, 09 May 2025 13:46:35 -0500 Christopher Schultz 
 wrote ---



Alex,
  
On 5/9/25 2:11 PM, My Subs wrote:

I have tested on Tomcat 10.1.40 with Native
Library 1.3.1 running on JDK 21.0.7+6.  The result is exactly the
same as described before.  The connector below works well with client
authentication, until I add the caCertificatePath attribute.  There
are no error messages in the logs.
  
Thanks for confirming that.
  
It probably does not matter, but what OS are you using, and what version

of OpenSSL are you using?
  

  
  
In your earlier message, you had a different configuration. This time

you haven't specified the class name in the "protocol" attribute. Which
one are you actually using?

Re: Tcnative-2 PQC support

2025-05-21 Thread federico bustamante 
I've only tried with OpenSSL 3.5, but I suspect it's the same story with
older versions (which, to be honest, wouldn't bother me as much as it being
a specific problem with 3.5).
I'll try building the DLL on Ubuntu.

Thanks,
Fede.


On Wed, May 21, 2025, 13:35 Christopher Schultz <
ch...@christopherschultz.net> wrote:

> Federico,
>
> On 5/21/25 12:27 PM, Christopher Schultz wrote:
> > Federico,
> >
> > On 5/21/25 10:58 AM, federico bustamante wrote:
> >> Hi!
> >> Did anyone have any luck building tcnative-2.dll with Openssl 3.5 to add
> >> Post Quantum Cryptography support?
> >> I tried following the steps in the wiki, but didn't have any luck (tried
> >> with cmake and nmake).
> >
> > Have you been able to compile tcnative-2.dll with earlier versions of
> > OpenSSL? I'm asking to see if you are having trouble with building
> > tcnative in general, or if OpenSSL 3.5 is what is causing the problem.
>
> FYI I just tried building libtcnative on MacOS against OpenSSL 3.5 and
> the build was successful. I believe the build process for Windows is ...
> complex compared to *NIX-style builds.
>
> But I wanted to confirm that tcnative didn't have any issues with
> building against OpenSSL 3.5 in general, and that seems not to be the case.
>
> So the problem must be with the Windows build process itself.
>
> -chris
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>