Agreed, the root cause is a bug in the Java part of Tomcat. However, but it points to a weakness of mod_jk, that if mod_jk hits the connection timeout on a socket and the server does not shut down the socket in a timely manner, all AJP requests in the httpd process grind to a halt for at least two seconds.
Larry -----Original Message----- From: jean-frederic clere [mailto:[EMAIL PROTECTED] Sent: Wednesday, November 28, 2007 10:21 AM To: Tomcat Developers List Cc: Rainer Jung; Larry Reisler Subject: Re: FW: Delays in mod_jk Larry Reisler wrote: > Rainer -- > > I recently changed replaced the version of JBOSS Web we were using to > JBOSSWEB_2_0_0_GA_CP04. It included several patches to the AJP code. That > appears to have solved the problem. The FIN packets from the back end come > back immediately now. I'm guessing that the fix to JBPAPP-366 > (http://jira.jboss.org/jira/browse/JBPAPP-366) fixed this too. > > I still see the mod_jk architecture as problematic JBPAPP-366 is http://svn.apache.org/viewvc?rev=589062&view=rev that was a bug in the JAVA part not in mod_jk. Cheers Jean-Frederic > -- it would be better if cleaning up the sockets would occur on a different > thread and without the critical section locked. > > Larry > > -----Original Message----- > From: Larry Reisler [mailto:[EMAIL PROTECTED] > Sent: Monday, November 26, 2007 10:08 AM > To: Rainer Jung; Tomcat Developers List > Subject: RE: Delays in mod_jk > > Rainer -- > > I am using out of the box JBOSS 4.21 -- no special connector, and no firewall > between the httpd tier and the other tier. > > Indeed, I agree that the FIN packet from the back end is missing. It seems > to come much later in the dump (exactly 4 minutes later). I can't help but > think this is an issue on the JBOSS side, but I'm not sure where to go to try > to debug that. > > I will send you the full capture file separately by private Mail. > > Larry > > > -----Original Message----- > From: Rainer Jung [mailto:[EMAIL PROTECTED] > Sent: Sunday, November 25, 2007 4:35 PM > To: Tomcat Developers List > Subject: Re: Delays in mod_jk > > Hi Larry, > > I'm again investigating your problem report concerning 2 second pauses > during socket shutdown in JK maintenance. Sorry for the long pause, but > I want to see, if there is something we need to fix before 1.2.26 > related to this case. > > I couldn't reproduce the behaviour on Linux. Do you use a special > connector for AJP in JBoss, like Tomcats APR connector, or is it just > the plain Coyote connector? Is there a firewall in between httpd and > JBoss? If so, would it be possible to sniff again on both sides? > > Larry Reisler wrote: >> I got a trace using some of the settings you have below. I'm not >> quite sure how to get the whole thing to you, as it is fairly large, >> and I don't wish to post it to the mailing list. > > If you like, you can send it by private Mail. > >> In any event the relevant lines from one example of an interaction I >> am attaching below. 10.45.3.22 is the apache 2.2 server and >> 10.45.3.21 and 10.45.3.24 are the tomcat servers. The trace was >> taken from the apache server. There are numerous other examples of >> this type of interaction. >> >> 02:49:04.329952 IP 10.45.3.22.34977 > 10.45.3.21.8009: F >> 2991069804:2991069804(0) ack 1909717451 win 8244 <nop,nop,timestamp >> 865633671 2700468480> 02:49:04.370343 IP 10.45.3.21.8009 > >> 10.45.3.22.34977: . ack 1 win 1448 <nop,nop,timestamp 2700815038 >> 865633671> 02:49:06.329558 IP 10.45.3.22.34972 > 10.45.3.24.8009: F >> 2991428814:2991428814(0) ack 330342931 win 4624 <nop,nop,timestamp >> 865635671 4202573488> 02:49:06.369533 IP 10.45.3.24.8009 > >> 10.45.3.22.34972: . ack 1 win 2269 <nop,nop,timestamp 4202873970 >> 865635671> 02:49:08.329843 IP 10.45.3.22.35008 > 10.45.3.24.8009: S >> 3372523679:3372523679(0) win 5840 <mss 1460,sackOK,timestamp >> 865637671 0,nop,wscale 2> 02:49:08.329961 IP 10.45.3.24.8009 > >> 10.45.3.22.35008: S 707449532:707449532(0) ack 3372523680 win 5792 >> <mss 1460,sackOK,timestamp 4202875930 865637671,nop,wscale 2> >> 02:49:08.329972 IP 10.45.3.22.35008 > 10.45.3.24.8009: . ack 1 win >> 1460 <nop,nop,timestamp 865637671 4202875930> 02:49:08.330001 IP >> 10.45.3.22.35008 > 10.45.3.24.8009: P 1:821(820) ack 1 win 1460 >> <nop,nop,timestamp 865637671 4202875930> 02:49:08.330023 IP >> 10.45.3.22.35008 > 10.45.3.24.8009: P 821:921(100) ack 1 win 1460 >> <nop,nop,timestamp 865637671 4202875930> >> >> My analysis of this is as follows: 1) A request comes in to the >> apache server, which has two timed out socket connections to the >> tomcat servers on ports 34977 and 34972. At 02:49:04.329952 it sends >> a FIN packet to the tomcat server and receives a response at >> 02:49:04.370343. It then waits two seconds from the time it sent the >> FIN packet. 2) At 02:49:06.329558 it sends a FIN packet to the other >> tomcat server and receives a response at 02:49:06.369533. It then >> waits two seconds from the time it sent the FIN packet, and then >> creates a new socket (35008) on which it successfully sends the >> transaction. > > The tcpdump excerpt seems to be incomplete. After the FIN and the > accompanying ACK, the final FIN/ACK from the backend is missing. Does it > occur later in the dump? If not, there should be a RST or the connection > should still be in CLOSE_WAIT on the JBoss machine. > > Regards, > > Rainer >
