https://bz.apache.org/bugzilla/show_bug.cgi?id=69700
Bug ID: 69700
Summary: Thread pool stops working when memory resources are
exceed and new thread is needed
Product: Tomcat 10
Version: 10.1.40
Hardware: All
OS: All
Status: NEW
Severity: normal
Priority: P2
Component: Connectors
Assignee: dev@tomcat.apache.org
Reporter: pavel.ja...@broadcom.com
Target Milestone: ------
Testing environment:
- Spring Boot 3.4.5
- Embedded Tomcat 10.1.40
- OS: z/OS 3.1
This bug is related to a system with low resources (memory, see MEMLIMIT=100M),
and could appear in any system. The problem is in ThreadPoolExecutor. The
default configuration is to have a minimum of 20 threads and a maximum of 200.
When a new thread is required, it is created on demand. When there is not
enough available memory to create a new thread, the OutOfMemoryError error is
thrown. The issue is that this error is not handled properly.
The error is handled by org.apache.tomcat.util.ExceptionUtils#handleThrowable
and always re-thrown as an implementation of VirtualMachineError. It leads to
stopping the thread pool, and Tomcat itself stops accepting new requests.
```
Exception in thread "https-jsse-nio-0.0.0.0-10660-Poller"
java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830,
errno 132 (0x84), errno2 0xc112001e
.at java.base/java.lang.Thread.startImpl(Native Method)
.at java.base/java.lang.Thread.start(Thread.java:1041)
.at
org.apache.tomcat.util.threads.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:983)
.at
org.apache.tomcat.util.threads.ThreadPoolExecutor.executeInternal(ThreadPoolExecutor.java:1449)
.at
org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1376)
.at
org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1264)
.at
org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:82
.at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:793)
.at java.base/java.lang.Thread.run(Thread.java:857)
```
The expected behaviour in this case is to reject (skip) the new request and
wait till any thread in the pool is ready to process it. It should not stop the
whole system. Probably a good approach could be once a new thread is not
created, Tomcat starts blocking creating new till another is one is recycled
(to avoid infinite loop, etc.)
The known workarround is to set the initial number of threads to the maximum.
In this case, all threads are created on the start-up, and no other thread is
needed at runtime, ie.:
```
-Dserver.tomcat.threads.min-spare=200 -Dserver.tomcat.threads.max=200
```
Test case:
- Prerequisites: a system with limited memory, threads.minSpare < threads.max
1. Start the Tomcat
2. Prepare a big load in memory
- For example, start downloading a huge file and stop receiving data on the
client side
3. Once memory is exceeded (in our testing, buffer bytes are fulfilled the
memory), make any request to Tomcat
- Note: In case there are still available threads in the pool, it is necessary
to make more requests than available threads (also possible to start opening
requests till Tomcat fails)
4. Tomcat pool asks for a new worker (starting a new thread)
5. OS reject creating a new thread and JDK throws `java.lang.OutOfMemoryError`
6. Any other request is not accepted, even if all previous connections are
closed
If an attacker has access to a big resource that can exceed the memory on the
system, there is a possibility of making a DOS. Just opening new requests is
enough to stop the server.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
