Hello, My application server running Apache HTTP Server 2.2.3 as a front end with mod_jk 1.20 to transfer request in AJP to a cluster of two Tomcat 6.0.10 instances is getting "Connecting to tomcat failed." errors within the mod_jk logs on a random basis. The whole cluster runs on a Windows Server 2003 Release 2, which is itself a virtual machine running under VMWare ESX 3.2.
The cluster runs applications for an intranet and some specific users accesses it via the extranet. The whole thing is not public on the internet. The whole cluster can run properly for a week and then going down as much as three times in the same day. I assume this is not traffic related since the issue occured at times of day where no one is at the office (very early in the morning). When this happens to the first instance of Tomcat, the second one goes down seconds after. Restarting Apache HTTP server nor Tomcat services fixes the issue. I need a complete reboot of the server. I cannot see anything relevant in the Tomcat logs. However, in JConsole, when it occurs I can see the total threads of both tomcat instances rising up to 1000 (it normally floats aroung 200 / 250 total threads) even if the number of active threads stays the same (150). As if threads were created/deleted/created... and so on very quickly. There is a strange behaviour: When it occurs, obviously I cannot access the applications via Apache HTTP server since it says "Service Temporarily Unavailable" but if I try to access directly both Tomcat on their HTTP connector, pages shows but with incomplete parts, sometimes pictures are missing, sometimes HTML is incomplete. I assume this is a Tomcat issue since the threads are going berserk, but I am unsure at this point. No need to say that my goal is to prevent all of this from happening... Here is what's dumped in mod_jk.log: [Tue Nov 04 09:40:09 2008] TC1 applications.labvolt.ca 2.135721 [Tue Nov 04 09:40:09 2008] [1304:2444] [info] mod_jk.c (2142): Service error=0 for worker=TC1 [Tue Nov 04 09:40:27 2008] [1304:2444] [info] jk_connect.c (451): connect to 127.0.0.1:8013 failed with errno=61 [Tue Nov 04 09:40:27 2008] [1304:2444] [info] jk_ajp_common.c (873): Failed opening socket to (127.0.0.1:8013) with (errno=61) [Tue Nov 04 09:40:27 2008] [1304:2444] [info] jk_ajp_common.c (1259): (TC1) error connecting to the backend server (errno=61) [Tue Nov 04 09:40:27 2008] [1304:2444] [info] jk_ajp_common.c (1916): (TC1) sending request to tomcat failed, recoverable operation attempt=1 [Tue Nov 04 09:40:28 2008] [1304:2444] [info] jk_connect.c (451): connect to 127.0.0.1:8013 failed with errno=61 [Tue Nov 04 09:40:28 2008] [1304:2444] [info] jk_ajp_common.c (873): Failed opening socket to (127.0.0.1:8013) with (errno=61) [Tue Nov 04 09:40:28 2008] [1304:2444] [info] jk_ajp_common.c (1259): (TC1) error connecting to the backend server (errno=61) [Tue Nov 04 09:40:28 2008] [1304:2444] [info] jk_ajp_common.c (1916): (TC1) sending request to tomcat failed, recoverable operation attempt=2 [Tue Nov 04 09:40:28 2008] [1304:2444] [error] jk_ajp_common.c (1928): (TC1) Connecting to tomcat failed. Tomcat is probably not started or is listening on the wrong port [Tue Nov 04 09:41:26 2008] TC1 applications.labvolt.ca 0.061460 [Tue Nov 04 09:41:33 2008] [1304:2444] [info] jk_connect.c (451): connect to 127.0.0.1:8014 failed with errno=61 [Tue Nov 04 09:41:33 2008] [1304:2444] [info] jk_ajp_common.c (873): Failed opening socket to (127.0.0.1:8014) with (errno=61) [Tue Nov 04 09:41:33 2008] [1304:2444] [info] jk_ajp_common.c (1259): (TC2) error connecting to the backend server (errno=61) [Tue Nov 04 09:41:33 2008] [1304:2444] [info] jk_ajp_common.c (1916): (TC2) sending request to tomcat failed, recoverable operation attempt=1 [Tue Nov 04 09:41:34 2008] [1304:2444] [info] jk_connect.c (451): connect to 127.0.0.1:8014 failed with errno=61 [Tue Nov 04 09:41:34 2008] [1304:2444] [info] jk_ajp_common.c (873): Failed opening socket to (127.0.0.1:8014) with (errno=61) [Tue Nov 04 09:41:34 2008] [1304:2444] [info] jk_ajp_common.c (1259): (TC2) error connecting to the backend server (errno=61) [Tue Nov 04 09:41:34 2008] [1304:2444] [info] jk_ajp_common.c (1916): (TC2) sending request to tomcat failed, recoverable operation attempt=2 [Tue Nov 04 09:41:34 2008] [1304:2444] [error] jk_ajp_common.c (1928): (TC2) Connecting to tomcat failed. Tomcat is probably not started or is listening on the wrong port Here is my workers used in Apache HTTP server: worker.TC1.port=8013 worker.TC1.host=localhost worker.TC1.type=ajp13 worker.TC1.lbfactor=1 worker.TC2.port=8014 worker.TC2.host=localhost worker.TC2.type=ajp13 worker.TC2.lbfactor=0 Here is one of my AJP connectors (both are same) in Tomcat: <Connector port="8013" protocol="AJP/1.3" redirectPort="8443" minSpareThreads="100" maxSpareThreads="300"/> I am new to mailing lists so feel free to ask me whatever I may have omitted. Best regards, and thanks in advance! -Eric Gauthier
