Re: Errors after upgrading to Tomcat 9.0.90

[Christopher Schultz]

Rainer,

On 6/21/24 07:55, Rainer Jung wrote:

Am 20.06.24 um 17:52 schrieb Christopher Schultz:

Francesco,

On 6/20/24 09:03, Francesco Chicchiriccò wrote:

On 2024/06/20 12:18:15 Konstantin Kolinko wrote:
чт, 20 июн. 2024 г. в 13:25, Francesco Chicchiriccò 
:


Hi there,
at Syncope we usually use the latest Tomcat versions to run a large 
chunk of our integration tests.


In our master branch we relay on Tomcat 10.1.x, and upgrading to 
10.1.25 from 10.1.24 went smooth as usual.


In our 3_0_X branch we relay on Tomcat 9.0.x; with 9.0.89 
everything goes as expected, but with 9.0.90 we are getting the 
exception [1].


Any idea of what could be changed in 9.0.90 within this regard? 
Thank you.


[1] https://gist.github.com/ilgrosso/be1fb1f2ea205eef60fb221973f87b34



The "java.lang.IllegalStateException: The request object has been
recycled and is no longer associated with this facade" message means
that a Request object has been (illegally) stored and is accessed
outside of its lifecycle - when the underlying structures and buffers
have already been recycled and may have been reused to process
subsequent requests.

Mark wrote:

You could try explicitly setting discardFacades to false.


In general, not recommended.
https://tomcat.apache.org/tomcat-9.0-doc/security-howto.html#Connectors

Calling getScheme() might be safe (as IIRC the info will come from a
connector configuration), but anything beyond that may lead to either
false results or to security issues. It would be better to identify
and fix the underlying cause.


Are you suggesting there is something to fix on Syncope's or CXF's 
code? Or rather on Tomcat?


This is a protection that Tomcat is providing to the application. 
There is no known bug in Tomcat's code.


It's not clear to me whether Syncope's or CXF's code is retaining such 
a reference, but, given the stack trace I would suspect that 
UserServiceImpl in Syncope's code is where you should start looking.


As I am writing above, we have no issues with Tomcat 9.0.89 nor with 
Tomcat 10.1.25.


It looks like this change was made in 9.0.90 *only* and was not ported 
to the other June releases. Honestly, this change should have been 
made to all active branches... I'm not sure why Rémy only applied it 
to 9.0.x.


The default value for discardFacades is already "true" in 10.1 and 11 
and the old system property 
org.apache.catalina.connector.RECYCLE_FACADES no longer exists for 
these. So setting discardFacades to "true" in 9.0.x makes the default 
behavior consistent.


+1 thanks for pointing this out. This commit was Rémy aligning 9.0.x 
with the other active branches, not making it /out/ of alignment.



No idea, why the application now shows different behavior for 9.0.x
and 10.1.x.
Agreed. But wrapping in HttpServletRequestWrapper, using Async, and 
performing what the OP described as "multiplex[ing] the incoming request 
onto atomic requests, injected to CXF service processing" sounds like a 
recipe for retaining references to these objects after they go out of scope.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Possible penetration attempt or DOS attack: any suggestions on what can be done?

[Christopher Schultz]

James,

On 6/24/24 17:57, James H. H. Lampert wrote:

On 6/24/24 12:03 PM, Tim Funk wrote:

Conversely, this is a good time for the developers to review
their server logging and tune it to be less verbose for these
normal exceptions. As well as implementing logging frameworks
and logging at the appropriate level (fatal through debug)


Thanks for your thoughts on the subject.

The primary problem isn't that the catalina.out file is enormous, nor 
that the log messages are too verbose (if anything, they're not verbose 
enough: a timestamp would help) ; it's that while the random filenames 
were being tried, in quick succession, the CPU usage of the JVM job 
skyrocketed, and legitimate traffic was probably having trouble getting in.


Hmm. Are you getting requests for the same (missing) resource over and 
over again, or many different ones? I suspect different, since attackers 
have a huge dictionary of "telltale resources" to identify a service as 
running one of many known-vulnerable services (e.g. Wordpress, JIRA, etc.).


Why aren't you seeing the source-IP in your own logs? If you are using a 
reverse-proxy and always getting the IP of the proxy, you should be able 
to (a) configure the reverse-proxy to provide its own IP to you and 
configure Tomcat to use that for logging and/or (b) look-up the same 
request in the proxy logs. Once you have that, you can ban.


Banning can be done in many ways.

One way would be just by-hand. If you see a huge number of requests from 
one IP, just ban it. If it's more than one IP, then you get to play 
whack-a-mole and automated solutions would be better.


Be careful with fail2ban (which I like a lot, but I've never used it 
with HTTP logs): you can easily ban people quickly who aren't actually 
attacking you. If you use fail2ban (or similar), I would be very 
conservative about its configuration. Basically, you choose the number 
of flags that have to be thrown per-IP per-time-interval. So something 
like "100 404s in a minute per-IP" would be conservative.


fail2ban is definitely available on Linux and probably also BSD. I think 
you are on AS/400 or something like that, right? And of course you'd 
want to ban as far up the chain as possible, at the reverse-proxy, TLS 
terminator, or load-balancer. I'm not sure exactly what options you have 
in that environment. fail2ban works directly with the OS's firewall, so 
on Linux that's iptables and on BSD that's generally ipfilter. If AS/400 
has a built-in firewall then it's possible fail2ban has a build for it.


Tim's suggestion of a WAF is a good one. I've tried to configure 
mod_security for Apache httpd and it's got like 100M rules and it 
immediately banned every one of our developers when we used it 
out-of-the-box. It is SUPER PARANOID about everything. If you are 
considering using mod_security, I would (a) run it only in a development 
/ testing environment and (b) run it in "audit mode" for a week or so 
and look at all the complaints it makes. I have had to specifically 
disable a LARGE number of checks to make it even partially usable.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Possible penetration attempt or DOS attack: any suggestions on what can be done?

[Stephen Tenberg]

Not a tomcat issue but I can't say enough good things about cloudflare
which we use.  Easy to configure WAF so the site is locked down to
cloudflare only and the rest of the world has to come through them where
each keystroke is inspected. They have a number of preconfigured rules that
mostly just work right out of the box. Various plans ranging from free and
up. We did this years ago on every site.

On Thu, Jun 27, 2024 at 11:03 AM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> James,
>
> On 6/24/24 17:57, James H. H. Lampert wrote:
> > On 6/24/24 12:03 PM, Tim Funk wrote:
> >> Conversely, this is a good time for the developers to review
> >> their server logging and tune it to be less verbose for these
> >> normal exceptions. As well as implementing logging frameworks
> >> and logging at the appropriate level (fatal through debug)
> >
> > Thanks for your thoughts on the subject.
> >
> > The primary problem isn't that the catalina.out file is enormous, nor
> > that the log messages are too verbose (if anything, they're not verbose
> > enough: a timestamp would help) ; it's that while the random filenames
> > were being tried, in quick succession, the CPU usage of the JVM job
> > skyrocketed, and legitimate traffic was probably having trouble getting
> in.
>
> Hmm. Are you getting requests for the same (missing) resource over and
> over again, or many different ones? I suspect different, since attackers
> have a huge dictionary of "telltale resources" to identify a service as
> running one of many known-vulnerable services (e.g. Wordpress, JIRA, etc.).
>
> Why aren't you seeing the source-IP in your own logs? If you are using a
> reverse-proxy and always getting the IP of the proxy, you should be able
> to (a) configure the reverse-proxy to provide its own IP to you and
> configure Tomcat to use that for logging and/or (b) look-up the same
> request in the proxy logs. Once you have that, you can ban.
>
> Banning can be done in many ways.
>
> One way would be just by-hand. If you see a huge number of requests from
> one IP, just ban it. If it's more than one IP, then you get to play
> whack-a-mole and automated solutions would be better.
>
> Be careful with fail2ban (which I like a lot, but I've never used it
> with HTTP logs): you can easily ban people quickly who aren't actually
> attacking you. If you use fail2ban (or similar), I would be very
> conservative about its configuration. Basically, you choose the number
> of flags that have to be thrown per-IP per-time-interval. So something
> like "100 404s in a minute per-IP" would be conservative.
>
> fail2ban is definitely available on Linux and probably also BSD. I think
> you are on AS/400 or something like that, right? And of course you'd
> want to ban as far up the chain as possible, at the reverse-proxy, TLS
> terminator, or load-balancer. I'm not sure exactly what options you have
> in that environment. fail2ban works directly with the OS's firewall, so
> on Linux that's iptables and on BSD that's generally ipfilter. If AS/400
> has a built-in firewall then it's possible fail2ban has a build for it.
>
> Tim's suggestion of a WAF is a good one. I've tried to configure
> mod_security for Apache httpd and it's got like 100M rules and it
> immediately banned every one of our developers when we used it
> out-of-the-box. It is SUPER PARANOID about everything. If you are
> considering using mod_security, I would (a) run it only in a development
> / testing environment and (b) run it in "audit mode" for a week or so
> and look at all the complaints it makes. I have had to specifically
> disable a LARGE number of checks to make it even partially usable.
>
> -chris
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: [OT] ldap administration tool and error LDAP: error code 49 - Invalid Credentials

[Christopher Schultz]

Shekhar,

On 6/25/24 01:40, Shekhar Dhotre wrote:

Is there any DLap admin tool available ? I want to manage IBM and openldap with 
it . trying to reset users password in IBM ldap but it fails with


"[root@camttvpws002 app]# ldapsearch -x -h //102.85.9.23 -D 
"ldap@seth.local" -b "dc=seth,dc=local" -w * 
"sAMAcountName=shekhdho"

ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)


My ldapsearch command does not take a -h argument for "host". It uses -H 
(capitalized), and it takes a proper URI like ldap://102.85.9.23 or 
ldaps://102.85.9.23


You should check the protocol (ldap or ldaps) and use the right one. 
Then check to see if your server can contact the LDAP server on that 
port (389 for ldap, 636 for ldaps). You can use telnet, nc, etc. to test 
connectivity.



[root@camttvpws002 ~]# tail -f /app/web/logs/RMA/RMA.log

2024-06-22 06:34:31,696{ INFO [http-bio-8443-exec-3] (RMAdao.java:5047) - after 
preparing the statement

2024-06-22 06:34:31,697{ INFO [http-bio-8443-exec-3] (RMAdao.java:5050) - 
application name is:RMA

2024-06-22 06:34:31,737{ERROR [http-bio-8443-exec-3] (RMAdao.java:5116) - The 
exception occurred is:ORA-00942: table or view does not exist


This looks like JDBC stuff, not LDAP.


2024-06-22 06:34:31,738{ERROR [http-bio-8443-exec-3] (CheckDownTime.java:60) - 
Exception Occurred : java.lang.NullPointerException

2024-06-22 06:34:31,739{ INFO [http-bio-8443-exec-3] (LoginAction.java:64) - 
Inside Action:- Method:fetchUserInfo

2024-06-22 06:34:31,739{ INFO [http-bio-8443-exec-3] (LoginAction.java:67) - 
The username is : shekhdho

2024-06-22 06:34:31,777{ERROR [http-bio-8443-exec-3] (Ldap.java:85) - Exception 
occurred :javax.naming.AuthenticationException: [LDAP: error code 49 - Invalid 
Credentials]


That's pretty readable, but the cause could be one of several things.

Can you post your  configuration, minus any secrets?

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: PKCS#8 encryption algorithm unrecognized

[Christopher Schultz]

Timothy,

On 6/25/24 10:39, Timothy Resh wrote:

In the
  SSLPassword="${KSENC(6qkaMErQ==; C:\Certificate\Keystore\Vessel.p12)}"

we defined a Class to convert the Encrypted password  and set the following
properties:
public class MyPropertySource implements
org.apache.tomcat.util.IntrospectionUtils.PropertySource
...
  public String getProperty(String arg0) {
 if (arg0.contains("KSENC(")) {

 System.setProperty("javax.net.ssl.keyStore",
keyStorePath);
 System.setProperty("javax.net.ssl.keyStorePassword",
clearText);
 System.setProperty("javax.net.ssl.trustStore",
trustStorePath);
 System.setProperty("javax.net.ssl.trustStorePassword",
clearText);
  }
...
}
This class will set the following properties at the beginning of Tomcat init

In my debugging I found that it could not determine the store properly.
So, I tried putting in the keystoreType="PKCS12 and now it works.

I hope this helps.  I'm still set up for debugging if you need something
looked at.


FYI I /think/ that all the javax.net.ssl.* properties are read only once 
during JSSE initialization. It's entirely possible that JSSE is 
configured *before* this code runs and it doesn't have any effect. For 
JVM-wide properties like these, it's best to set those system properties 
on the command-line instead of waiting for Java code to set them.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Possible penetration attempt or DOS attack: any suggestions on what can be done?

[James H. H. Lampert]

On 6/27/24 8:01 AM, Christopher Schultz wrote:


Why aren't you seeing the source-IP in your own logs?


Because our webapp developer hadn't thought to put them into the log 
messages we generate. He did, however, direct us to the 
localhost_access_log files (where I quite frankly hadn't thought to 
look). And they turned out to mostly be from an IP address that doesn't 
return anything in a reverse DNS lookup.


The filenames in the requests seem to run in alphabetical order, and 
cover a gamut from nonsense combinations of random letters to clinical 
terms for sexual anatomy(!).


--
JHHL

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Possible penetration attempt or DOS attack: any suggestions on what can be done?

[James H. H. Lampert]

On 6/27/24 8:01 AM, Christopher Schultz wrote:

"100 404s in a minute per-IP"


Actually, what I was seeing, once the webapp developer pointed me in the 
right direction, was several dozen 404s per *second* from a single IP.


Not sure if Fail2ban would even work in this situation: like the 
overwhelming majority of the customer Tomcat installations we've done, 
this one is running on an IBM Midrange box (AS/400, iSeries, or whatever 
IBM is calling it this week). Which is probably how the huge amount of 
CPU usage was so easy to spot: WRKACTJOB is more user-friendly than any 
Linux-equivalent (or Mac-equivalent or WinDoze-equivalent) I've seen.


--
JHHL

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: How to configure Tomcat with a Managed Service Account when using LocalMachine certificates for TLS

[Christopher Schultz]

Gavioto,

On 6/25/24 13:51, Gavioto 🕵 wrote:
Finally, I got a configuration that works with Certificate Storage. It 
is very specific, and I couldn't find any other until date.
It works, but in our environment there is a required manual step yet. I 
think it should be configured in Windows and is common for all programs 
requesting private key not special for Tomcat.
A dialog of Windows Security is shown asking to allow access, similar to 
UAC but for certificates.


Your image was stripped from your mailing-list post. Can you please 
*describe* this manual-step? Without the image, I fear your post has no 
usable content.



The server.xml configuration used is

|SSLEnabled="true">              certificateKeystoreType="Windows-MY-LOCALMACHINE" 
certificateKeystoreFile=""             certificateKeyAlias="tomcat"/>   
    |


This is the same as before, yes?

I continue my research trying to use different ciphers and properties, 
because I won't be able to use the standard without the . It is clear 
for me, that different internals libraries are used in Tomcat, so when I 
use the only the it doesn't work and it won't find the key in the storage.


Do you need to specify the Provider to use? Your sample code was 
manually-loading a specific JSSE Provider.


*For the purpose of the initial question, I consider this solutions as 
valid, as Tomcat is able to work with the Windows Local Machine 
Certificate Storage.*

Solution versions:

  * JDK 11.0.20+ or JDK version with fixed bug
_https://bugs.openjdk.org/browse/JDK-8286790
_
  * Tested with Tomcat 9.0.63
  * Tested in Windows 2019


*Thank you for your support. Hope this information is useful to crate a 
Use Case in the wiki.*


Some doubts are in the air yet, but it is essentially possible for 
Tomcat to work with this.


Two questions thats remaining for the future:

  *
How to disable or bypass the Windows Security dialog? Is it possible
to disable for specific Users/Services?


Does it show every time, or only once and after that it works without a 
dialog?



  *
Why if I use keyStoreType="windows-my-localmachine" directly in
 it doesn't find the certificate?


Do you mean certificateKeystoreType? Speling matters.

https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#Special%20Features_SSL%20Support%20-%20Certificate_certificateKeystoreType

There is a note about key store types:

"
Key store types

In addition to the standard key store types (JKS and PKCS12), most Java 
runtimes support additional key store types such as Windows-ROOT, 
Windows-My, DKS as well as hardware security modules. Generally, to use 
these additional keystore types with a TLS Connector in Tomcat:


*Set the certificateKeystoreType and/or truststoreType Connector 
attribute (as appropriate) to the necessary type


*If a configuration file is required, set the 
certificateKeystoreFile and/or truststoreFile Connector attribute (as 
appropriate) to point to the file


*If no configuration file is required then you will almost certainly 
need to explicitly set the certificateKeystoreFile and/or truststoreFile 
Connector attribute (as appropriate) to the empty string ("")


*If a password is required, set the certificateKeystorePassword 
and/or truststorePassword Connector attribute (as appropriate) to the 
required password


*If no password is required then you will almost certainly need to 
explicitly set the certificateKeystorePassword and/or truststorePassword 
Connector attribute (as appropriate) to the empty string ("")

"

Note this last item. I'm not sure if it is required in your particular case.

-chris



*De:* Gavioto 🕵 
*Enviado:* martes, 25 de junio de 2024 15:27
*Para:* users@tomcat.apache.org 
*Asunto:* RE: How to configure Tomcat with a Managed Service Account 
when using LocalMachine certificates for TLS

- how are are starting Tomcat?
  Tomcat is starting as a service with "Domain\account1$" (Managed 
Service Account)


- is Tomcat installed as a Windows service?
  Yes

- which account is Tomcat running under?
  "Domain\account1$" (Managed Service Account) It is not a normal 
domain account used for the service.  Secure group managed service 
accounts - Microsoft Entra | Microsoft 
Learn>


Regarding the mean of "user" I'm referring to the user who is running 
the Tomcat Service. In this case, the Managed Service Account.



De: Mark Thomas 
Enviado: martes, 25 de junio de 2024 12:51
Para: users@tomcat.apache.org 
Asunto: Re: How to configure Tomcat with a Managed Service Account when 
using LocalMachine certificates for T

Re: Isolating the Root Cause of "Connection Refused"

[Christopher Schultz]

Eric,

On 6/25/24 14:22, Chuck Caldarale wrote:



On Jun 25, 2024, at 13:12, Eric Robinson  wrote:

I'm obviously not getting something. If I understand correctly, the purpose of 
the acceptCount param is to allow tomcat to reject connections even after the 
TCP stack has passed them to the java process. At that point, the request has 
made it through the OS all the way to tomcat, so if tomcat rejects it, then it 
should log a message... no?



No - Tomcat passes the acceptCount value to the TCP/IP stack of the OS as part of 
listener socket initialization. The accept queue is maintained entirely by the OS; 
Tomcat is not involved in the management thereof, other than taking requests off the 
accept queue with the listener thread for the . If connection 
requests arrive at the NIC fast enough to fill up the accept queue, excess ones are 
simply rejected by the OS.


+1

If the OS's TCP stack rejects a request because the backlog queue is 
full, the application is never notified. I've never even tried to see if 
this situation results in a syslog message or similar. You'd have to do 
some Linux kernel research and/or Rocky config research to see if you 
can coax that information out of the kernel. It would be one more attack 
vector: jut slam a server with connections and fill the disk with logs.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Isolating the Root Cause of "Connection Refused"

[Christopher Schultz]

Eric,

On 6/25/24 20:10, Eric Robinson wrote:


No - Tomcat passes the acceptCount value to the TCP/IP stack of the
OS as part of listener socket initialization.


I thought of that after I sent my previous message.


the OS won't log this, since it's considered to be an application
error.


Assuming the problem is the acceptCount value, then it's technically
the app's fault for improperly initializing the socket listener.
Nevertheless, it would make sense for the OS to log a neighborly
message (perhaps optionally) along the lines of, "I rejected a
connection attempt because of the acceptCount setting you told me to
use." It would sure help with troubleshooting. As you pointed out,
the app can't log errors it does not know about, so nothing is logged
anywhere.

What is impact on memory utilization if we increase the acceptCount
value? There are 100 tomcat instances on the server. And would
maxThreads have to be increased to accommodate the extra
connections?


I suspect that your problem lies in the neighborhood of the acceptCount
value, so I think this is worthwhile pursuing.

You said that your clients have very different usage patterns. Perhaps
you can identify those that get the biggest bursts of traffic and
reconfigure only those to have higher acceptCount values? There is no
particular reason that you have to raise them all to 5000 or whatever.

The memory impact is likely negligible, especially with that insane
amount of RAM. I'm no Linux kernel expert, but I think there are
essentially no limits (other than physical RAM and avoiding swapping) on
the amount of memory that the kernel can use for this kind of thing. A
TCP backlog entry can't be THAT big. A few dozens of bytes or maybe even
a whole kilobyte and you have scads of RAM to spare.

maxThreads does not need to be increased. Think of acceptCount and 
maxThreads as sympathetic settings. If you never want any client to be 
turned-away, then set acceptCount/backlog to something "high". If you 
never want any client to wait a long time, set maxThreads to something 
"high". But of course your CPU(s) can only do so much work at once, so 
at some point setting maxThreads too high will just make things worse.


Do you have any reverse-proxy or similar out in front of any/all of 
these services? If so, you could turn "connection refused" into a HTTP 
503 response to clients. Of course, then you have the problem of tuning 
the reverse-proxy so that it doesn't turn anyone away...


"Connection refused" is one of TCP's ways of telling a client that the 
service isn't available. That can be because it's down completely or 
because it's being heavily-used. IMHO, "under heavy use" is an 
acceptable reason to return "connection refused" but your (paying) 
clients may disagree. How often are you getting reports of "connection 
refusals"? You should be able to monitor your processes to see the 
number of concurrent connections being processed, and see if they are 
reaching the various limits you have.


Any time your server's concurrent-request-count and max-thread-count are 
the same, it means your acceptCount/backlog is being used. Look for 
instances of those situations and re-tune those services to have a 
higher thread count.


Can you post an example  configuration? Are most of your 
servers configured with similar  configurations?


There are a lot of variables that go into these calculations, including 
connector type (BIO, NIO/2, APR, etc.), usage pattern (short requests 
versus long-running ones), use of async/websocket, etc. So more 
information would certainly help us help you.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Isolating the Root Cause of "Connection Refused"

[Christopher Schultz]

Eric,

On 6/26/24 01:43, Eric Robinson wrote:

What is impact on memory utilization if we increase the acceptCount
value? There are 100 tomcat instances on the server. And would
maxThreads have to be increased to accommodate the extra
connections?


After reading more, I guess that's a dumb question.


:)


I'm trying to prevent connections from being rejected. As I
understand it, when the server hits the maxThreads limit, then
additional connections queue up, until the acceptCount value is
finally reached, and then new connections are rejected.


This is correct according to all documentation. But. Java's 
documentation is clear that acceptCount/backlog is a "hint" to the OS 
about what that value can be. The OS is under no obligation to (a) set 
the accept-queue size to anything specific (such as your desired value), 
(b) keep the accept-queue size consistent. The documentation for 
listen(2) on my Debian Linux system suggests that the backlog is 
actually respected but silently capped to the OS's max count which is 
4096 is anything remotely recent.


My guess is that Java didn't want to over-promise with the acceptCount 
since it's supposed to be cross-platform. While Linux will generally 
respect the acceptCount, it CAN be capped silently. Other OSs may always 
give a process the same acceptCount or so something else somewhat 
unexpected.


Since you are using Linux, I think you can bet on acceptCount working 
mostly as you expect, with the caveat that it might be capped at an 
environment-specific value (which is likely changeable if necessary).



Since I am seeing connection refusals, then it is possible that the
acceptCount limit is being reached.

I would say "probable".


If so, then it seems there are two ways to address this. I can either
increase acceptCount, which gives the server more time to make
threads are available, or I can increase maxThreads, in which case
more requests can be handled concurrently and the acceptCount may not
be reached.

+1


Both of these settings improve performance, but in different ways.
I wouldn't say that raising acceptCount "improves performance" but maybe 
"improves experience". It may effectively /decrease/ performance because 
connections accepted (into the accept queue) but that have to wait a few 
seconds to be serviced may result in clients hanging up the phone before 
they are serviced. Depending upon the details of the situation, your 
server may serve a request from a client who has already given-up on the 
response... and even re-submitted the same request. So you get to do 
double the work in that situation.


Unfortunately, there is no "timeout" for the accept-queue. Once a 
connection has been accepted (bad verb, since accept(2) is the call that 
the application makes to accept the connection) into the 
accept-queue/backlog, it will remain there until the application calls 
accept(2) or terminates, even if the client disconnects or 15 minutes 
goes by. There is no way to say "put them in the queue but after 5 
seconds go ahead and refuse the connection". It's "connect or refuse" at 
that point.



maxThreads improves performance by allowing more requests to be
processed concurrently, while acceptCount improves performance by
preventing connections from being refused, which requires requests to
be reissued by the client, and that is an expensive process.
I would argue the opposite: clients receiving "connection refused" go 
away quickly and cheaply. It's the ones who wait in the accept queue who 
are "expensive".



Which of these approaches is correct? I tend to think increasing
maxThreads is better if the server has enough memory and CPU.
However, since the impact on memory and CPU depend on the
characteristics of the app, that's not a question you can be expected
to answer.


Again, it all depends upon your clients, usage patterns, and the "price" 
of a refused connection. If you have (human) clients who get VERY upset 
if their connections are refused than you may need to either change 
client expectations or change your service configuration with the 
understanding that "waiting in line is better than being turned away". 
Nobody can answer this question for you: you will need to decide for 
yourself which is better.


Hope that helps,
-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Many CLOSE_WAIT connections causing the app not available

[Christopher Schultz]

Stephen,

On 6/26/24 09:31, Chuck Caldarale wrote:



On Jun 26, 2024, at 00:18, Stephen Stevie
 wrote:

We are using Apache Tomcat 8.5.49



Stop right there. Tomcat 8.5.x reached end-of-life earlier this year
and is no longer supported. The particular version you’re running is
over 4.5 years old, and many, many security and correctness fixes
have been implemented since. You really need to upgrade to a
supported version ASAP.


+1

I would say using Tomcat 8.5.100 should be okay for "a little while" as 
long as you have a plan to upgrade to something past 8.5.


I haven't heard a single complaint from anyone who upgraded from 8.5 to 
9.0 unless they were doing something out of the ordinary.


But you should really have a plan in-place NOW for moving to a supported 
version of Tomcat.


And using 8.5.49 is a really *really* bad idea for exactly the reasons 
Chuck has described above.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Many CLOSE_WAIT connections causing the app not available

[Christopher Schultz]

Stephen,

On 6/26/24 01:18, Stephen Stevie wrote:

We are using Apache Tomcat 8.5.49 and sometimes in a day, we see the
application is going unresponsive though the service is up and running and
giving 503 (service unavailable error). When netstat for the port is run,
we see many CLOSE_WAIT connections. Once we restart the service, the
application is coming back and sometimes this comes back on its own.

We have not made any changes to Tomcat in recent times and it was working
fine until the start of this year. After that, it was once a month and now
every alternate day, we are seeing this. We're using 16GB RAM Web Server (2
servers) and can see 1k-1.5k users connected to the application in a day.


Please post your  configuration, minus any secrets.

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Possible penetration attempt or DOS attack: any suggestions on what can be done?

[Christopher Schultz]

James,

On 6/27/24 11:36, James H. H. Lampert wrote:

On 6/27/24 8:01 AM, Christopher Schultz wrote:


Why aren't you seeing the source-IP in your own logs?


Because our webapp developer hadn't thought to put them into the log 
messages we generate. He did, however, direct us to the 
localhost_access_log files (where I quite frankly hadn't thought to 
look). And they turned out to mostly be from an IP address that doesn't 
return anything in a reverse DNS lookup.


That's fine. IPs are IPs. Just bad the IP.

You can also perform IP reverse-lookup which is hit-or-miss. I did the 
same on an IP that was obviously scanning us last week in the way you 
described, and I got many results each claiming that the IP was (a) 
Russian (b) Ukrainian (c) Belgian (d) Polish and a bunch of other things 
as well. So obviously it's not foolproof. But sometimes it says 
"Sandusky, OH" and you say "oops that's our data center" and have to 
figure out what that means for you.


Don't ban your own data center, it probably won't go well for you. :)

The filenames in the requests seem to run in alphabetical order, and 
cover a gamut from nonsense combinations of random letters to clinical 
terms for sexual anatomy(!).


Yep, it's just a stupid scanner.

Anyone who was *really* trying to get it would be doing it much more 
covertly. fail2ban would be a fine solution in this case.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Possible penetration attempt or DOS attack: any suggestions on what can be done?

[Christopher Schultz]

James,

On 6/27/24 11:47, James H. H. Lampert wrote:

On 6/27/24 8:01 AM, Christopher Schultz wrote:

"100 404s in a minute per-IP"


Actually, what I was seeing, once the webapp developer pointed me in the 
right direction, was several dozen 404s per *second* from a single IP.


Not sure if Fail2ban would even work in this situation: like the 
overwhelming majority of the customer Tomcat installations we've done, 
this one is running on an IBM Midrange box (AS/400, iSeries, or whatever 
IBM is calling it this week). Which is probably how the huge amount of 
CPU usage was so easy to spot: WRKACTJOB is more user-friendly than any 
Linux-equivalent (or Mac-equivalent or WinDoze-equivalent) I've seen.


Yeah... some lazy Google searches for "fail2ban" along with various 
synonyms for os/as/400 and ibm midrange don't come up with much.


I might reach-out to your IBM contacts and/or OS/400 "community" to see 
if anyone has any suggestions.


In the meantime, use whatever tools are at your disposal to just bin 
every connection from that IP.


I found an article from 1998 on getting rhe OS/400 firewall up and 
running. Evidently, there is a web interface for configuration. Maybe it 
won't be a huge headache to simply firewall that IP out of your environment.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Compatibility Check Tomcat 10.1.x with IBM MQ 9.3.0.20

[Chaudhary, Mohit]

Hi Team,

We have installed Tomcat 10.1.1 in RHEL 8 server so we needed to check if 
tomcat 10.1.1 is compatible with IBM MQ 9.3.0.20.


Thanks & Regards,
Mohit Chaudhary

Re: Possible penetration attempt or DOS attack: any suggestions on what can be done?

[Greg Huber]

If they are serious they use multiple IP addresses.  If you are self 
hosting, for a DOS case their is not much you can do other than blocking 
the ip address(s) on the router.


Usually after a couple of days they get bored and try some where else.

It is worth having some internal filter mechanism in your app that 
checks for page abuse, so they get 403's.


On 27/06/2024 16:47, James H. H. Lampert wrote:

On 6/27/24 8:01 AM, Christopher Schultz wrote:

"100 404s in a minute per-IP"


Actually, what I was seeing, once the webapp developer pointed me in 
the right direction, was several dozen 404s per *second* from a single 
IP.


Not sure if Fail2ban would even work in this situation: like the 
overwhelming majority of the customer Tomcat installations we've done, 
this one is running on an IBM Midrange box (AS/400, iSeries, or 
whatever IBM is calling it this week). Which is probably how the huge 
amount of CPU usage was so easy to spot: WRKACTJOB is more 
user-friendly than any Linux-equivalent (or Mac-equivalent or 
WinDoze-equivalent) I've seen.


--
JHHL

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org