ArrayIndexOutOfBoundsException in ResourceCache.java
I'm pretty regularly seeing the following error in my embedded Tomcat
(5.0.28) connector. It appears to occur as a result of one thread
entering the ResourceCache.lookup(String name) method and getting the
index of an element in the cache using the find method. Before this
thread can access the element at that index the cache array is modified
by a second thread and the index is now outside the bounds of the array.
A basic race condition. I've noticed that I can configure my Context
(StandardContext) to increase the max cache size which seems to help
quite a bit with this problem. Is this the recommended solution or
should a code change also be considered to alleviate this problem? I
would suggest that the ResourceCache.lookup method should create a
reference to the current cache which it should use rather than working
on the global cache reference which may change. Something like this:
/* current code, ResourceCache.java line 286 */
CacheEntry cacheEntry = null;
...
int pos = find(cache, name);
/* ArrayIndexOutOfBoundsException occurs when backing array modified
between these lines */
if ((pos != -1) && (name.equals(cache[pos].name))) {
cacheEntry = cache[pos];
}
/* new code */
CacheEntry cacheEntry = null;
CacheEntry[] currentCache = cache;
...
int pos = find(currentCache, name);
if ((pos != -1) && (name.equals(currentCache[pos].name))) {
cacheEntry = currentCache[pos];
}
Thanks in advance for your help!
James Courtney
**
** The original stack trace **
**
2005-11-28 15:09:01: ERROR An exception or error occurred in the
container
during the request processing
java.lang.ArrayIndexOutOfBoundsException: 4868
at
org.apache.naming.resources.ResourceCache.lookup(ResourceCache.java:288)
at
org.apache.naming.resources.ProxyDirContext.cacheLookup(ProxyDirContext.
java:1393)
at
org.apache.naming.resources.ProxyDirContext.lookup(ProxyDirContext.java:
279)
at
org.apache.tomcat.util.http.mapper.Mapper.internalMapWrapper(Mapper.java
:775)
at
org.apache.tomcat.util.http.mapper.Mapper.internalMap(Mapper.java:621)
at
org.apache.tomcat.util.http.mapper.Mapper.map(Mapper.java:511)
at
org.apache.coyote.tomcat5.CoyoteAdapter.postParseRequest(CoyoteAdapter.j
ava:279)
at
org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:158)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:79
9)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processC
onnection(Http11Protocol.java:705)
at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:57
7)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool
.java:683)
at java.lang.Thread.run(Thread.java:534)
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: ArrayIndexOutOfBoundsException in ResourceCache.java
Thanks Filip,
I believe the threads causing the condition are the connector threads
as we have 200 or so of them configured for handling requests of which
most are for static content which apparently comes from the
ResourceCache via ProxyDirContext and Mapper. We do not "own" these
threads as they are created and managed by the connector for handling
requests. The exception is occuring when looking up a static piece of
content. ProxyDirContext manages the ResourceCache and synchonizes on
it whenever it modifies it using allocate and load and also unload. It
does not synchronize on it when it uses the lookup method which has the
problem I mentioned below. An alternative solution would be for
ProxyDirContext to synchronized on the cache when performing the lookup
but that might introduce more overhead than the read-copy method I
suggested initially.
I will attempt to file a bug on this and suggest a test case. We're
seeing this in our environment because we have a static content base of
30+ MB consisting of 8000-1 static files all being accessed at
regular intervals by 8000+ clients. This causes a lot of "churn" in the
ResourceCache I suspect and aggrivates the exception we see. One might
be able to reproduce this by explicitly reducing the max size of the
cache on the Context and using a smaller number of files whose total
size still far exceeds the cache maximum size set. A sufficient number
of clients would still be required to trigger the race condition. In
short it's probably a PITA to reproduce but I'll suggest what I can.
How explicit a configuration do you guys need from me for a test case?
I'll try and reproduce it on vanilla Tomcat 4.0.28 (non-embedded) as I'd
think that'd be just as possible to do and more generally relevant. If
I can then I'll try and do it using as simple/minimal a config as
possible but we'll see:)
Thanks again.
Jamey
-Original Message-
From: Filip Hanik - Dev Lists [mailto:[EMAIL PROTECTED]
Sent: Tuesday, August 01, 2006 3:21 PM
To: Tomcat Developers List
Subject: Re: ArrayIndexOutOfBoundsException in ResourceCache.java
if you are running it embedded, what threads are causing the race
condition, if it is one of your threads, you can avoid it.
otherwise, if you have a test case for us, submit it to bugzilla, and we
will be happy to consider it for the next 5.0.x release
Filip
James Courtney wrote:
> I'm pretty regularly seeing the following error in my embedded Tomcat
> (5.0.28) connector. It appears to occur as a result of one thread
> entering the ResourceCache.lookup(String name) method and getting the
> index of an element in the cache using the find method. Before this
> thread can access the element at that index the cache array is
> modified by a second thread and the index is now outside the bounds of
the array.
> A basic race condition. I've noticed that I can configure my Context
> (StandardContext) to increase the max cache size which seems to help
> quite a bit with this problem. Is this the recommended solution or
> should a code change also be considered to alleviate this problem? I
> would suggest that the ResourceCache.lookup method should create a
> reference to the current cache which it should use rather than working
> on the global cache reference which may change. Something like this:
>
> /* current code, ResourceCache.java line 286 */ CacheEntry cacheEntry
> = null; ...
> int pos = find(cache, name);
> /* ArrayIndexOutOfBoundsException occurs when backing array modified
> between these lines */ if ((pos != -1) &&
> (name.equals(cache[pos].name))) {
> cacheEntry = cache[pos];
> }
>
> /* new code */
> CacheEntry cacheEntry = null;
> CacheEntry[] currentCache = cache;
> ...
> int pos = find(currentCache, name);
> if ((pos != -1) && (name.equals(currentCache[pos].name))) {
> cacheEntry = currentCache[pos];
> }
>
> Thanks in advance for your help!
>
> James Courtney
>
>
>
> **
> ** The original stack trace **
> **
>
> 2005-11-28 15:09:01: ERROR An exception or error occurred in the
> container during the request processing
> java.lang.ArrayIndexOutOfBoundsException: 4868
> at
>
org.apache.naming.resources.ResourceCache.lookup(ResourceCache.java:288)
> at
>
org.apache.naming.resources.ProxyDirContext.cacheLookup(ProxyDirContext.
> java:1393)
> at
>
org.apache.naming.resources.ProxyDirContext.lookup(ProxyDirContext.java:
> 279)
> at
> org.apache.tomcat.util.http.mapper.Mapper.internalMapWrapper(Mapper.ja
> va
> :775)
> at
> org.apache.tomcat.util.http.mapper.Mapper.internalMap(Mapper.java:621)
> at
> org.apache.tomcat.util
RE: ArrayIndexOutOfBoundsException in ResourceCache.java
Yep, 5.0.28, sorry:)
Jamey
-Original Message-
From: Filip Hanik - Dev Lists [mailto:[EMAIL PROTECTED]
Sent: Tuesday, August 01, 2006 4:48 PM
To: Tomcat Developers List
Subject: Re: ArrayIndexOutOfBoundsException in ResourceCache.java
if you can create a test case, great, if not, a patch would be nice with
your bug submission.
I believe you meant version 5.0.28, not 4.xx.x.
Filip
James Courtney wrote:
> Thanks Filip,
> I believe the threads causing the condition are the connector
> threads as we have 200 or so of them configured for handling requests
> of which most are for static content which apparently comes from the
> ResourceCache via ProxyDirContext and Mapper. We do not "own" these
> threads as they are created and managed by the connector for handling
> requests. The exception is occuring when looking up a static piece of
> content. ProxyDirContext manages the ResourceCache and synchonizes on
> it whenever it modifies it using allocate and load and also unload.
> It does not synchronize on it when it uses the lookup method which has
> the problem I mentioned below. An alternative solution would be for
> ProxyDirContext to synchronized on the cache when performing the
> lookup but that might introduce more overhead than the read-copy
> method I suggested initially.
>
> I will attempt to file a bug on this and suggest a test case. We're
> seeing this in our environment because we have a static content base
> of
> 30+ MB consisting of 8000-1 static files all being accessed at
> regular intervals by 8000+ clients. This causes a lot of "churn" in
> the ResourceCache I suspect and aggrivates the exception we see. One
> might be able to reproduce this by explicitly reducing the max size of
> the cache on the Context and using a smaller number of files whose
> total size still far exceeds the cache maximum size set. A sufficient
> number of clients would still be required to trigger the race
> condition. In short it's probably a PITA to reproduce but I'll
suggest what I can.
>
> How explicit a configuration do you guys need from me for a test case?
> I'll try and reproduce it on vanilla Tomcat 4.0.28 (non-embedded) as
> I'd think that'd be just as possible to do and more generally
> relevant. If I can then I'll try and do it using as simple/minimal a
> config as possible but we'll see:)
>
> Thanks again.
>
> Jamey
>
>
>
> -Original Message-
> From: Filip Hanik - Dev Lists [mailto:[EMAIL PROTECTED]
> Sent: Tuesday, August 01, 2006 3:21 PM
> To: Tomcat Developers List
> Subject: Re: ArrayIndexOutOfBoundsException in ResourceCache.java
>
> if you are running it embedded, what threads are causing the race
> condition, if it is one of your threads, you can avoid it.
> otherwise, if you have a test case for us, submit it to bugzilla, and
> we will be happy to consider it for the next 5.0.x release
>
> Filip
>
> James Courtney wrote:
>
>> I'm pretty regularly seeing the following error in my embedded Tomcat
>> (5.0.28) connector. It appears to occur as a result of one thread
>> entering the ResourceCache.lookup(String name) method and getting the
>> index of an element in the cache using the find method. Before this
>> thread can access the element at that index the cache array is
>> modified by a second thread and the index is now outside the bounds
>> of
>>
> the array.
>
>> A basic race condition. I've noticed that I can configure my Context
>> (StandardContext) to increase the max cache size which seems to help
>> quite a bit with this problem. Is this the recommended solution or
>> should a code change also be considered to alleviate this problem? I
>> would suggest that the ResourceCache.lookup method should create a
>> reference to the current cache which it should use rather than
>> working
>>
>
>
>> on the global cache reference which may change. Something like this:
>>
>> /* current code, ResourceCache.java line 286 */ CacheEntry cacheEntry
>> = null; ...
>> int pos = find(cache, name);
>> /* ArrayIndexOutOfBoundsException occurs when backing array modified
>> between these lines */ if ((pos != -1) &&
>> (name.equals(cache[pos].name))) {
>> cacheEntry = cache[pos];
>> }
>>
>> /* new code */
>> CacheEntry cacheEntry = null;
>> CacheEntry[] currentCache = cache;
>> ...
>> int pos = find(currentCache, name);
>> if ((pos != -1) && (name.equals(currentCache[pos].name))) {
>> cacheEntry = currentCache[pos]
RE: ArrayIndexOutOfBoundsException in ResourceCache.java
You coded the modification methods that way but the read method (lookup) is not.
I'll take a few minutes to try and create a test case but my other work will
prevent me spending too much time on it. I'll certainly submit a patch with
the bug.
Thanks to all!
Jamey
-Original Message-
From: Remy Maucherat [mailto:[EMAIL PROTECTED]
Sent: Tuesday, August 01, 2006 5:16 PM
To: Tomcat Developers List
Subject: Re: ArrayIndexOutOfBoundsException in ResourceCache.java
James Courtney wrote:
> /* new code */
> CacheEntry cacheEntry = null;
> CacheEntry[] currentCache = cache;
> ...
> int pos = find(currentCache, name);
> if ((pos != -1) && (name.equals(currentCache[pos].name))) {
> cacheEntry = currentCache[pos];
> }
I think I should have coded it that way.
Rémy
-
To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail:
[EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: svn commit: r427821 - /tomcat/tc6.0.x/trunk/java/org/apache/naming/resources/ResourceCache.java
Fantastic, thanks Remy!
Do you guys still want a bug filed for that?
Jamey
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
Sent: Tuesday, August 01, 2006 5:39 PM
To: tomcat-dev@jakarta.apache.org
Subject: svn commit: r427821 -
/tomcat/tc6.0.x/trunk/java/org/apache/naming/resources/ResourceCache.jav
a
Author: remm
Date: Tue Aug 1 17:39:28 2006
New Revision: 427821
URL: http://svn.apache.org/viewvc?rev=427821&view=rev
Log:
- Use a single reference to the cache during lookup (the cache array
could be concurrently replaced).
Modified:
tomcat/tc6.0.x/trunk/java/org/apache/naming/resources/ResourceCache.java
Modified:
tomcat/tc6.0.x/trunk/java/org/apache/naming/resources/ResourceCache.java
URL:
http://svn.apache.org/viewvc/tomcat/tc6.0.x/trunk/java/org/apache/naming
/resources/ResourceCache.java?rev=427821&r1=427820&r2=427821&view=diff
==
---
tomcat/tc6.0.x/trunk/java/org/apache/naming/resources/ResourceCache.java
(original)
+++ tomcat/tc6.0.x/trunk/java/org/apache/naming/resources/ResourceCache.
+++ java Tue Aug 1 17:39:28 2006
@@ -289,10 +289,11 @@
public CacheEntry lookup(String name) {
CacheEntry cacheEntry = null;
+CacheEntry[] currentCache = cache;
accessCount++;
-int pos = find(cache, name);
-if ((pos != -1) && (name.equals(cache[pos].name))) {
-cacheEntry = cache[pos];
+int pos = find(currentCache, name);
+if ((pos != -1) && (name.equals(currentCache[pos].name))) {
+cacheEntry = currentCache[pos];
}
if (cacheEntry == null) {
try {
-
To unsubscribe, e-mail: [EMAIL PROTECTED] For additional
commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: svn commit: r427821 - /tomcat/tc6.0.x/trunk/java/org/apache/naming/resources/ResourceCache.java
BTW, is my increasing cacheMaxSize on the Context a sensible workaround
until this code is released?
Thanks!
Jamey
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
Sent: Tuesday, August 01, 2006 5:39 PM
To: tomcat-dev@jakarta.apache.org
Subject: svn commit: r427821 -
/tomcat/tc6.0.x/trunk/java/org/apache/naming/resources/ResourceCache.jav
a
Author: remm
Date: Tue Aug 1 17:39:28 2006
New Revision: 427821
URL: http://svn.apache.org/viewvc?rev=427821&view=rev
Log:
- Use a single reference to the cache during lookup (the cache array
could be concurrently replaced).
Modified:
tomcat/tc6.0.x/trunk/java/org/apache/naming/resources/ResourceCache.java
Modified:
tomcat/tc6.0.x/trunk/java/org/apache/naming/resources/ResourceCache.java
URL:
http://svn.apache.org/viewvc/tomcat/tc6.0.x/trunk/java/org/apache/naming
/resources/ResourceCache.java?rev=427821&r1=427820&r2=427821&view=diff
==
---
tomcat/tc6.0.x/trunk/java/org/apache/naming/resources/ResourceCache.java
(original)
+++ tomcat/tc6.0.x/trunk/java/org/apache/naming/resources/ResourceCache.
+++ java Tue Aug 1 17:39:28 2006
@@ -289,10 +289,11 @@
public CacheEntry lookup(String name) {
CacheEntry cacheEntry = null;
+CacheEntry[] currentCache = cache;
accessCount++;
-int pos = find(cache, name);
-if ((pos != -1) && (name.equals(cache[pos].name))) {
-cacheEntry = cache[pos];
+int pos = find(currentCache, name);
+if ((pos != -1) && (name.equals(currentCache[pos].name))) {
+cacheEntry = currentCache[pos];
}
if (cacheEntry == null) {
try {
-
To unsubscribe, e-mail: [EMAIL PROTECTED] For additional
commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Out of Threads error
I'm getting this message from my Tomcat 5.0.28 on Linux. 2006-08-02 21:01:19: ERROR All threads (400) are currently busy, waiting. Increase maxThreads (400) or check the servlet status Originally I had 200 threads configured and now 400 in an attempt to avoid it. Either number seems larger than what should be necessary. It seems quite likely that there's an application issue causing all threads to block and accumulate or something so I need to do a kill -3 next time it happens to try and see what everyone is up to. That said, I was attempting to grep the Tomcat source code for where this message is logged and thus where I hoped to find and understand better the thread manager. I grepped for "Increase maxThreads", "Increase", and "are currently busy" all to no avail in finding the code doing the logging. Is the code which logs this message part of a dependency library not in the tomcat source tree? If so, which one please. I just want to understand the application behavior when the limit is reached. Many thanks! Jamey - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Out of Threads error
Thanks Tim! I'd been grepping only *.java files. Doh! Jamey -Original Message- From: Tim Funk [mailto:[EMAIL PROTECTED] Sent: Friday, August 04, 2006 11:34 AM To: Tomcat Developers List Subject: Re: Out of Threads error For tomcat 5: find . -type f |xargs grep "Increase maxThreads" jakarta-tomcat-connectors/util/java/org/apache/tomcat/util/threads/res/L ocalStrings.properties -Tim James Courtney wrote: > I'm getting this message from my Tomcat 5.0.28 on Linux. > > 2006-08-02 21:01:19: ERROR All threads (400) are currently busy, > waiting. Increase maxThreads (400) or check the servlet status > > Originally I had 200 threads configured and now 400 in an attempt to > avoid it. Either number seems larger than what should be necessary. > It seems quite likely that there's an application issue causing all > threads to block and accumulate or something so I need to do a kill -3 > next time it happens to try and see what everyone is up to. That > said, I was attempting to grep the Tomcat source code for where this > message is logged and thus where I hoped to find and understand better > the thread manager. I grepped for "Increase maxThreads", "Increase", > and "are currently busy" all to no avail in finding the code doing the logging. > Is the code which logs this message part of a dependency library not > in the tomcat source tree? If so, which one please. I just want to > understand the application behavior when the limit is reached. > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
