There's no intent to handle this case?  Really?

Tomcat should throw an "IllegalArgumentException" whenever 'acceptCount' is set to anything other than 0 explaining this if this is the case!

If so, this is very unfortunate. We use "maxThreads" as a throttle to limit the concurrency at this level under the (silly?) assumption that "acceptCount" behaves as documented in the documentation. [Yes, one could argue that a separate throttle should be used behind this layer, which we might have been inclined to do if the documentation said "acceptCount does not work".]

Further, I've been able to reproduce such issues using a concurrency of exactly the "maxThreads". At *best* there is an off-by-one error here, but I've still seen problems in some testing using a concurrency level 2 or 3 less than "maxThreads" -- which leads me to believe there is something a bit more nefarious going on here. Another of my colleagues claims that he's encountered issues in simple ab tests quite a ways below "maxThreads". I've not managed to reproduce this case myself, but I could certainly focus on this case if the Tomcat contributors have no intent to fix "acceptCount" behavior for the Java AJP connector.

Is there a firm intent to handle this case in the _/native/_ AJP connector? If so, then this distinction should be clearly stated (and an IllegalArgumentException thrown as per above when the native connector cannot be found/initialized).

Personally I see utility in having "acceptCount" working in the AJP case. If one exceeds the mod_proxy_ajp hard maximum the client immediately gets a 503. There are cases when just queuing a few requests makes more sense than either returning 503's or letting them rip -- which is where acceptCount comes in. [Granted the acceptCount numbers below are excessive, but that's another matter...]

--
Jess Holle
[EMAIL PROTECTED]

Bill Barker wrote:
Yes, since you know in advance how many connections you are going to get to
the AJP connector, you can configure it so that it has enough threads to
handle all of those connections.  That is why it doesn't attempt to handle
the case when the concurrency goes above maxThreads
-----Original Message-----
From: Jess Holle [mailto:[EMAIL PROTECTED] Sent: Thursday, June 14, 2007 2:58 PM
To: Tomcat Developers List
Cc: Dobson, Simon; Wang, Andy; Fenlason, Josh
Subject: Serious non-native AJP connector issue

We're facing a /serious /issue with the non-native AJP connector.

To put it most simply, some requests seem to "get lost" in Tomcat in various cases involving concurrent requests -- and not egregious numbers of concurrent requests, either.

For instance,

   1. Use a Tomcat 5.5.23 with a configuration like:

        <Connector port="8010"
                   minSpareThreads="4" maxSpareThreads="12"
    maxThreads="18" acceptCount="282"
                   tomcatAuthentication="false"
    useBodyEncodingForURI="true" URIEncoding="UTF-8"
                   enableLookups="false" redirectPort="8443"
    protocol="AJP/1.3" />

(which are intended solely for making it easier to test concurrency
    issues that to overrun a realistic 'maxThreads' setting) and as a
control, similar thread pool settings on the direct HTTP connector:

        <Connector port="8080" maxHttpHeaderSize="8192"
                   minSpareThreads="4" maxSpareThreads="12"
    maxThreads="18" acceptCount="282"
                   enableLookups="false" redirectPort="8443"
connectionTimeout="20000" disableUploadTimeout="true" />

2. Use an Apache 2.2.4 with mod_proxy_ajp with a configuration like:

        <Proxy balancer://ajpWorker>
        BalancerMember ajp://localhost:8010 min=16 max=300 smax=40
        ttl=900 keepalive=Off timeout=900
        </Proxy>

        RewriteEngine on
        RewriteRule ^(/TestApp/(.*\.jsp(.*)|servlet/.*|.*\.jar))$
        balancer://ajpWorker$1 [P]

    (on Windows in this case; similar results can be obtained on Linux
    at least)

3. Use a simple test JSP page (placed in a web app containing nothing
      else):

        <[EMAIL PROTECTED] session="false"
        %><[EMAIL PROTECTED] contentType="text/html" pageEncoding="UTF-8"
        %><%!
private static final String titleString = "Sleepy Test JSP Page";
        %><html>
        <head>
        <%
          String  sleepSeconds = request.getParameter( "secs" );
          if ( sleepSeconds == null )
            sleepSeconds = "1";
          long  secsToSleep = Long.parseLong( sleepSeconds );
          Thread.sleep( 1000L * secsToSleep );
        %>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
        <title><%=titleString%></title>
        </head>
        <body>
        <center>
        <h2><%=titleString%>: SUCCESS!</h2>
        [Slept <%= secsToSleep %> seconds.]
        </center>
        </body>
        </html>

   4. Hit the page with ab
          * First, test direct Tomcat connections:
                o ab -n 24 -c 24 -A wcadmin:wcadmin
                  http://hostname:*8080*/TestApp/test.jsp?secs=3
                      + Result: All requests complete sucessfully in
                        6118 ms.
          * Second, test connections via Apache:
                o ab -n 24 -c 24 -A wcadmin:wcadmin
                  http://hostname/TestApp/test.jsp?secs=3
                      + Result: Only 17 requests complete before ab
                        times out.
          * Third, test connections via Apache again soon (under the
            BalancerMember 'timeout' seconds) after the last test
                o ab -n 24 -c 24 -A wcadmin:wcadmin
                  http://hostname/TestApp/test.jsp?secs=3
+ Result: Only 9 requests complete before ab times
                        out.

Something is clearly /horribly/ wrong with the handling of any concurrency over 'maxThreads' in this case. Even so, there seems to be some sort of "off-by-one" error in that only 17 requests complete, not 18. Worse, this has a lingering effect that decreases Tomcat's ability to concurrent requests thereafter (with this impact seemingly being much worse the longer the BalancerMember timeout is set to -- and we have some very long running requests and thus need this timeout to be /very/ large).

This is not the only ill effect we've seen when hitting Tomcat 5.5.24 with concurrent requests in this manner, but it is a good place to start.

As for the native connector, it just works here. So why don't I just use it? Well, first off, we have to support Tomcat on Windows (32 and 64-bit), Linux, Solaris, HPUX (PA-RISC and Itanium), and AIX. We've been unable to get the connector built on some of these platforms and on some we can't get the resulting binary to function (specifically on AIX). Further, we had some stability issues with the native connector in the past and had considered the Java connector the safest, if not fastest, option -- and to a degree given that everything is Java I still feel that's the case. Finally, however, this connector should just plain work. Tomcat shouldn't be a cripple unless/until you manage to build a native connector for your platform.

Any troubleshooting and/or debugging ideas (e.g. where exactly to place breakpoints, what logs to turn on, etc, etc) would be /greatly/ appreciated.

--
Jess Holle
[EMAIL PROTECTED]

P.S. If this should go to the user's mailing list instead that's fine, but this really seemed like a developer-oriented issue to me.





This message is intended only for the use of the person(s) listed above as the 
intended recipient(s), and may contain information that is PRIVILEGED and 
CONFIDENTIAL.  If you are not an intended recipient, you may not read, copy, or 
distribute this message or any attachment. If you received this communication 
in error, please notify us immediately by e-mail and then delete all copies of 
this message and any attachments.

In addition you should be aware that ordinary (unencrypted) e-mail sent through 
the Internet is not secure. Do not send confidential or sensitive information, 
such as social security numbers, account numbers, personal identification 
numbers and passwords, to us via ordinary (unencrypted) e-mail.


---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]


Reply via email to