Hi all,
My webapp can somewhat regularly crash the JVM on multiple machines. Most of the hotspot dump files are remarkably consistence with where the error is occurring...then again, I do know what is important and what is not. Running tomcat without tcnative-1.dll resolves the problem but I'd like to help find the solution. While researching this issue, it has become clear that I should submit debugging info that would be generated by having a specially compiled tcnative-1.dll what would provide a stack trace. C programming (and compiling) is not exactly my thing, so... Where can I get myself a tcnative-1.dll binary compiled with debugging symbol? Once I get that, I should be able to reproduce the error and submit a bug report with the debugging data. Tia, Stu Particulars: - OS: Windows XP and Server 2003 - Java: 1.5.0_08-b03, 1.5.0_07-b03, ... - Tomcat: 6.0.10 HotSpot dump: # # An unexpected error has been detected by HotSpot Virtual Machine: # # EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x10003ede, pid=2040, tid=2444 # # Java VM: Java HotSpot(TM) Client VM (1.5.0_08-b03 mixed mode) # Problematic frame: # C [tcnative-1.dll+0x3ede] # --------------- T H R E A D --------------- Current thread (0x0b4da4a8): JavaThread "Thread-2" [_thread_in_native, id=2444] siginfo: ExceptionCode=0xc0000005, reading address 0x00001225 Registers: EAX=0x00001221, EBX=0x00000000, ECX=0x0b720000, EDX=0x0b484f38 ESP=0x0ba2f7ec, EBP=0x0ba2f7f0, ESI=0x0b484f38, EDI=0x0b71ff38 EIP=0x10003ede, EFLAGS=0x00010206 Top of Stack: (sp=0x0ba2f7ec) 0x0ba2f7ec: 0b717f90 0ba2f810 10014394 0b484f38 0x0ba2f7fc: 00000000 06ed64e8 0b717f90 0ba2f824 0x0ba2f80c: 10004048 0ba2f824 10004055 0b71ff38 0x0ba2f81c: 0b4da4a8 06ed64e8 0ba2f858 008d832f 0x0ba2f82c: 0b4da568 0ba2f860 0b717f90 00000000 0x0ba2f83c: 0ba2f83c 06ed64e8 0ba2f86c 06ed7410 0x0ba2f84c: 00000000 06ed64e8 0ba2f868 0ba2f88c 0x0ba2f85c: 008d2a8f 06ed73b0 008d6509 0b717f90 Instructions: (pc=0x10003ede) 0x10003ece: 90 90 55 8b ec 56 8b 75 08 8b 46 18 85 c0 74 10 0x10003ede: 8b 40 04 85 c0 74 09 8b 4e 0c 51 ff d0 83 c4 04 Stack: [0x0b9f0000,0x0ba30000), sp=0x0ba2f7ec, free space=253k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C [tcnative-1.dll+0x3ede] C [tcnative-1.dll+0x14394] C [tcnative-1.dll+0x4055] j org.apache.tomcat.jni.Socket.close(J)I+0 j org.apache.tomcat.util.net.AprEndpoint.destroy()V+27 j org.apache.coyote.http11.Http11AprProtocol.destroy()V+35 j org.apache.catalina.connector.Connector.stop()V+109 j org.apache.catalina.core.StandardService.stop()V+201 j org.apache.catalina.core.StandardServer.stop()V+65 j org.apache.catalina.startup.Catalina.stop()V+39 j org.apache.catalina.startup.Catalina.start()V+154 v ~StubRoutines::call_stub V [jvm.dll+0x86e84] V [jvm.dll+0xddead] V [jvm.dll+0x86d55] V [jvm.dll+0xf1f59] V [jvm.dll+0xa4d04] C [java.dll+0x6d11] j sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lan g/Object;)Ljava/lang/Object;+87 J sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava /lang/Object;)Ljava/lang/Object; J java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Lj ava/lang/Object; v ~RuntimeStub::alignment_frame_return Runtime1 stub j org.apache.catalina.startup.Bootstrap.start()V+37 j org.apache.catalina.startup.Bootstrap.main([Ljava/lang/String;)V+125 v ~StubRoutines::call_stub V [jvm.dll+0x86e84] V [jvm.dll+0xddead] V [jvm.dll+0x86d55] V [jvm.dll+0x8dda7] C [tomcat6.exe+0x625a] Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j org.apache.tomcat.jni.Socket.close(J)I+0 j org.apache.tomcat.util.net.AprEndpoint.destroy()V+27 j org.apache.coyote.http11.Http11AprProtocol.destroy()V+35 j org.apache.catalina.connector.Connector.stop()V+109 j org.apache.catalina.core.StandardService.stop()V+201 j org.apache.catalina.core.StandardServer.stop()V+65 j org.apache.catalina.startup.Catalina.stop()V+39 j org.apache.catalina.startup.Catalina.start()V+154 v ~StubRoutines::call_stub j sun.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;L java/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+0 j sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lan g/Object;)Ljava/lang/Object;+87 J sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava /lang/Object;)Ljava/lang/Object; J java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Lj ava/lang/Object; v ~RuntimeStub::alignment_frame_return Runtime1 stub j org.apache.catalina.startup.Bootstrap.start()V+37 j org.apache.catalina.startup.Bootstrap.main([Ljava/lang/String;)V+125 v ~StubRoutines::call_stub --------------- P R O C E S S --------------- Java Threads: ( => current thread ) 0x0bcf3e18 JavaThread "ajp-8009-1" daemon [_thread_blocked, id=1556] 0x0b56bc38 JavaThread "Timer-1" daemon [_thread_blocked, id=3256] 0x0b8cc2b0 JavaThread "http-80-3" daemon [_thread_blocked, id=2956] 0x0b8dde18 JavaThread "http-80-2" daemon [_thread_blocked, id=3400] 0x0b415d70 JavaThread "ajp-8009-CometPoller-7" daemon [_thread_blocked, id=2780] 0x0b8544b0 JavaThread "ajp-8009-CometPoller-6" daemon [_thread_blocked, id=3868] 0x0b773828 JavaThread "ajp-8009-CometPoller-5" daemon [_thread_blocked, id=2156] 0x0b553c48 JavaThread "ajp-8009-CometPoller-4" daemon [_thread_blocked, id=2584] 0x0b514ce8 JavaThread "ajp-8009-CometPoller-3" daemon [_thread_blocked, id=924] 0x0b56cbf8 JavaThread "ajp-8009-CometPoller-2" daemon [_thread_blocked, id=1516] 0x0b56c9e0 JavaThread "ajp-8009-CometPoller-1" daemon [_thread_blocked, id=3912] 0x0b56c058 JavaThread "ajp-8009-CometPoller-0" daemon [_thread_blocked, id=3352] 0x0b854b40 JavaThread "ajp-8009-Poller-7" daemon [_thread_blocked, id=3568] 0x0b8549b8 JavaThread "ajp-8009-Poller-6" daemon [_thread_blocked, id=2828] 0x0b8375c0 JavaThread "ajp-8009-Poller-5" daemon [_thread_blocked, id=3416] 0x0b837438 JavaThread "ajp-8009-Poller-4" daemon [_thread_blocked, id=3304] 0x0b4163d8 JavaThread "ajp-8009-Poller-3" daemon [_thread_blocked, id=1452] 0x0b88c218 JavaThread "ajp-8009-Poller-2" daemon [_thread_blocked, id=3768] 0x0b88bc00 JavaThread "ajp-8009-Poller-1" daemon [_thread_blocked, id=1540] 0x0b8ba708 JavaThread "ajp-8009-Poller-0" daemon [_thread_blocked, id=3920] 0x0b8ba580 JavaThread "ajp-8009-Acceptor-0" daemon [_thread_blocked, id=3988] 0x0b7e9678 JavaThread "http-80-Sendfile-0" daemon [_thread_blocked, id=3296] 0x0b814c98 JavaThread "http-80-CometPoller-7" daemon [_thread_blocked, id=3156] 0x0b78c5d0 JavaThread "http-80-CometPoller-6" daemon [_thread_blocked, id=1796] 0x0b7aa678 JavaThread "http-80-CometPoller-5" daemon [_thread_blocked, id=3532] 0x0b8b7618 JavaThread "http-80-CometPoller-4" daemon [_thread_blocked, id=1756] 0x0b7f6810 JavaThread "http-80-CometPoller-3" daemon [_thread_blocked, id=3180] 0x0b8de808 JavaThread "http-80-CometPoller-2" daemon [_thread_blocked, id=300] 0x0b45de80 JavaThread "http-80-CometPoller-1" daemon [_thread_blocked, id=3588] 0x0b81fd58 JavaThread "http-80-CometPoller-0" daemon [_thread_blocked, id=3636] 0x0b7b2840 JavaThread "http-80-Poller-7" daemon [_thread_blocked, id=2364] 0x0b46cd78 JavaThread "http-80-Poller-6" daemon [_thread_blocked, id=1788] 0x0b7fcb30 JavaThread "http-80-Poller-5" daemon [_thread_blocked, id=352] 0x0b7aada8 JavaThread "http-80-Poller-4" daemon [_thread_blocked, id=164] 0x0b81f408 JavaThread "http-80-Poller-3" daemon [_thread_blocked, id=1268] 0x0b46d408 JavaThread "http-80-Poller-2" daemon [_thread_blocked, id=2316] 0x0b808008 JavaThread "http-80-Poller-1" daemon [_thread_blocked, id=2052] 0x0b470e50 JavaThread "http-80-1" daemon [_thread_blocked, id=3696] 0x0b78c900 JavaThread "http-80-Poller-0" daemon [_thread_blocked, id=2692] 0x0b808210 JavaThread "http-80-Acceptor-0" daemon [_thread_blocked, id=916] 0x0b874350 JavaThread "simplex-daemon-historgram" [_thread_blocked, id=856] =>0x0b4da4a8 JavaThread "Thread-2" [_thread_in_native, id=2444] 0x0b5a90c0 JavaThread "RMI TCP Accept-1291" daemon [_thread_in_native, id=3244] 0x008ceef8 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=3284] 0x0b529be0 JavaThread "Timer-0" daemon [_thread_blocked, id=3288] 0x0089f1d8 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=2244] 0x0089dda8 JavaThread "CompilerThread0" daemon [_thread_blocked, id=4076] 0x0089d068 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=2972] 0x0089c4f8 JavaThread "Surrogate Locker Thread (CMS)" daemon [_thread_blocked, id=3272] 0x008936b0 JavaThread "Finalizer" daemon [_thread_blocked, id=1952] 0x00892248 JavaThread "Reference Handler" daemon [_thread_blocked, id=3036] 0x003d6de0 JavaThread "main" [_thread_in_native, id=3584] Other Threads: 0x0088e098 VMThread [id=1156] 0x008c4c20 WatcherThread [id=2760] VM state:not at safepoint (normal execution) VM Mutex/Monitor currently owned by a thread: None Heap def new generation total 4032K, used 464K [0x028d0000, 0x02cd0000, 0x02cd0000) eden space 3968K, 11% used [0x028d0000, 0x02944188, 0x02cb0000) from space 64K, 0% used [0x02cb0000, 0x02cb0000, 0x02cc0000) to space 64K, 0% used [0x02cc0000, 0x02cc0000, 0x02cd0000) concurrent mark-sweep generation total 12740K, used 6444K [0x02cd0000, 0x03941000, 0x068d0000) concurrent-mark-sweep perm gen total 33068K, used 19862K [0x068d0000, 0x0891b000, 0x0a8d0000) Dynamic libraries: 0x00400000 - 0x0040e000 d:\Tomcat6.0\bin\tomcat6.exe 0x7c900000 - 0x7c9b0000 C:\WINDOWS\system32\ntdll.dll 0x7c800000 - 0x7c8f4000 C:\WINDOWS\system32\kernel32.dll 0x7e410000 - 0x7e4a0000 C:\WINDOWS\system32\USER32.dll 0x77f10000 - 0x77f57000 C:\WINDOWS\system32\GDI32.dll 0x77dd0000 - 0x77e6b000 C:\WINDOWS\system32\ADVAPI32.dll 0x77e70000 - 0x77f01000 C:\WINDOWS\system32\RPCRT4.dll 0x77c10000 - 0x77c68000 C:\WINDOWS\system32\MSVCRT.dll 0x7c9c0000 - 0x7d1d5000 C:\WINDOWS\system32\SHELL32.dll 0x77f60000 - 0x77fd6000 C:\WINDOWS\system32\SHLWAPI.dll 0x76390000 - 0x763ad000 C:\WINDOWS\system32\IMM32.DLL 0x773d0000 - 0x774d3000 C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df _6.0.2600.2982_x-ww_ac3f9c03\comctl32.dll 0x5d090000 - 0x5d12a000 C:\WINDOWS\system32\comctl32.dll 0x6d6c0000 - 0x6d85b000 C:\Program Files\Java\jre1.5.0_08\bin\client\jvm.dll 0x76b40000 - 0x76b6d000 C:\WINDOWS\system32\WINMM.dll 0x6d280000 - 0x6d288000 C:\Program Files\Java\jre1.5.0_08\bin\hpi.dll 0x76bf0000 - 0x76bfb000 C:\WINDOWS\system32\PSAPI.DLL 0x77fe0000 - 0x77ff1000 C:\WINDOWS\system32\Secur32.dll 0x6d690000 - 0x6d69c000 C:\Program Files\Java\jre1.5.0_08\bin\verify.dll 0x6d300000 - 0x6d31d000 C:\Program Files\Java\jre1.5.0_08\bin\java.dll 0x6d6b0000 - 0x6d6bf000 C:\Program Files\Java\jre1.5.0_08\bin\zip.dll 0x6d4b0000 - 0x6d4b8000 C:\Program Files\Java\jre1.5.0_08\bin\management.dll 0x6d4c0000 - 0x6d4d3000 C:\Program Files\Java\jre1.5.0_08\bin\net.dll 0x71ab0000 - 0x71ac7000 C:\WINDOWS\system32\WS2_32.dll 0x71aa0000 - 0x71aa8000 C:\WINDOWS\system32\WS2HELP.dll 0x71a50000 - 0x71a8f000 C:\WINDOWS\System32\mswsock.dll 0x76f20000 - 0x76f47000 C:\WINDOWS\system32\DNSAPI.dll 0x76fb0000 - 0x76fb8000 C:\WINDOWS\System32\winrnr.dll 0x76f60000 - 0x76f8c000 C:\WINDOWS\system32\WLDAP32.dll 0x76fc0000 - 0x76fc6000 C:\WINDOWS\system32\rasadhlp.dll 0x0ffd0000 - 0x0fff8000 C:\WINDOWS\system32\rsaenh.dll 0x662b0000 - 0x66308000 C:\WINDOWS\system32\hnetcfg.dll 0x71a90000 - 0x71a98000 C:\WINDOWS\System32\wshtcpip.dll 0x10000000 - 0x100d0000 D:\Tomcat6.0\bin\tcnative-1.dll 0x6d4e0000 - 0x6d4e9000 C:\Program Files\Java\jre1.5.0_08\bin\nio.dll VM Arguments: jvm_args: -Dcatalina.home=d:\Tomcat6.0 -Dcatalina.base=d:\Tomcat6.0 -Djava.endorsed.dirs=d:\Tomcat6.0\common\endorsed -Djava.io.tmpdir=d:\Tomcat6.0\temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=d:\Tomcat6.0\conf\logging.properties -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1291 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.password.file=../conf/jmxremote.password -Dcom.sun.management.jmxremote.access.file=../conf/jmxremote.access -Xincgc -ea vfprintf java_command: <unknown> Launcher Type: generic Environment Variables: JAVA_HOME=D:\java\jdk CLASSPATH=.;c:\aspectj1.5\lib\aspectjrt.jar;D:\apache\jdepend-2.9.1\lib\ jdepend-2.9.1.jar PATH=C:\Program Files\Windows Resource Kits\Tools\;C:\Perl\bin\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\Syste m32\Wbem;D:\apache\apache-ant-1.6.5\bin;D:\java\jdk\bin;C:\Program Files\Darwin Streaming Server;C:\Program Files\QuickTime\QTSystem\;c:\aspectj1.5\bin OS=Windows_NT PROCESSOR_IDENTIFIER=x86 Family 15 Model 2 Stepping 9, GenuineIntel --------------- S Y S T E M --------------- OS: Windows XP Build 2600 Service Pack 2 CPU:total 1 (cores per cpu 1, threads per core 1) family 15 model 2 stepping 9, cmov, cx8, fxsr, mmx, sse, sse2 Memory: 4k page, physical 1015328k(160028k free), swap 2447504k(1463296k free) vm_info: Java HotSpot(TM) Client VM (1.5.0_08-b03) for windows-x86, built on Jul 26 2006 01:10:50 by "java_re" with MS VC++ 6.0
