mads1980 opened a new pull request #245: OpenSSLEngine improvements to guard against multiple shutdown() calls triggered by construction exception and finalize() later URL: https://github.com/apache/tomcat/pull/245 We started getting random SIGSEGV JVM crashes on our servers after upgrading to OpenJDK 13.0.2 The crashes are always in tcnative code, on the JVM Finalizer daemon thread, at OpenSSLEngine.shutdown(). We're on Tomcat 9.0.31, tcnative 1.2.23, OpenSSL 1.1.1d and APR 1.7.0 using NIO connectors. This combination works fine with Oracle JDK 8u202 as well as OpenJDK 8u242, but crashes began immediately after upgrading to OpenJDK 13.0.2 Since the failure is related to tcnative code and finalization, we guess the new JVM is probably invoking the finalizers sooner, so this is exposing a bug in OpenSSLEngine which was already present, but was not being triggered on previous JVMs because of different timing. The crash occurs when SSL.freeSSL() is invoked, so this probably means either the pointer is zero, or a double SSL.freeSSL() is invoked. This can happen if there are uncaught exceptions within the OpenSSLEngine constructor, as finalization can execute concurrently with object construction. We adapted the solution found in OpenSSLContext.destroy() which evidently was causing similar issues. The current volatile "destroyed" checks found in OpenSSLEngine seem to be insufficient, so we replaced it with an AtomicBoolean. Also, we check whether "networkBIO" and "ssl" are zero in shutdown() to avoid attempting to free a null pointer. We found a discussing regarding these issues in the following thread: http://mail-archives.apache.org/mod_mbox/tomcat-dev/201901.mbox/%[email protected]%3E It seems the improvements made it to OpenSSLContext but not OpenSSLEngine. This pull request should fix remaining issues. See sample crash dump extract below: # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007fc98d024f58, pid=32426, tid=32435 # # JRE version: OpenJDK Runtime Environment (13.0.2+8) (build 13.0.2+8) # Java VM: OpenJDK 64-Bit Server VM (13.0.2+8, mixed mode, tiered, compressed oops, g1 gc, linux-amd64) # Problematic frame: # C [libcrypto.so.1.1+0x16ff58] CRYPTO_get_ex_data+0x8 # # Core dump will be written. Default location: Core dumps may be processed with "/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e" (or dumping to //core.32426) # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. # --------------- S U M M A R Y ------------ Command Line: -Dcatalina.home=/home/tomcat -Dcatalina.base=/home/tomcat -Djava.library.path=/home/tomcat/lib -Djava.security.properties=/home/tomcat/conf/java.security.properties -Djava.io.tmpdir=/var/tmp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/home/tomcat/conf/logging.properties -XX:ErrorFile=/home/tomcat/logs/hs_err_pid%p.log -XX:HeapDumpPath=/home/tomcat/logs/ --illegal-access=permit -Xshare:off -XX:+PrintVMOptions -XX:+UnlockExperimentalVMOptions -XX:+UnlockDiagnosticVMOptions -XX:+UsePerfData -XX:+DoEscapeAnalysis -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -XX:-OmitStackTraceInFastThrow -XX:+PrintStringTableStatistics -XX:StringTableSize=1000003 -XX:SymbolTableSize=1000003 -XX:+PrintCompilation -XX:+PrintInlining -XX:+UseBiasedLocking -XX:BiasedLockingStartupDelay=0 -XX:MaxInlineLevel=15 -Xms5G -Xmx5G -Xss512K -XX:MetaspaceSize=128M -XX:MaxMetaspaceSize=256M -XX:+UseLargePages -XX:InitialCodeCacheSize=128M -XX:ReservedCodeCacheSize=128M -XX:+UseCodeCacheFlushing -XX:+UseCompressedOops -verbose:gc -XX:+DisableExplicitGC -XX:+ParallelRefProcEnabled -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=4 -XX:G1NewSizePercent=30 -XX:G1MaxNewSizePercent=60 -XX:InitiatingHeapOccupancyPercent=0 -XX:G1ReservePercent=10 -XX:G1HeapRegionSize=32M -XX:+UseStringDeduplication -Xlog:gc*,gc+ergo*=trace,gc+heap=debug,classhisto*=trace,safepoint:file=/home/tomcat/logs/gclog.2020-02-17:level,time,uptime,tags -Dcom.renxo.cms.util.io.network.NetworkUtils.preferredLocalHostSubnet=192.168.0.0/16 -Dcom.renxo.cms.util.io.network.NetworkUtils.developmentEnvironmentSubnet=192.168.3.0/24 -javaagent:/home/tomcat/apminsight/apminsight-javaagent.jar -Dapminsight.application.name=homer.renxo.net applications -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Djava.security.egd=file:/dev/urandom -Djava.rmi.server.hostname=homer.renxo.net -Djava.io.tmpdir=/home/tomcat/temp/ -Dcommons.daemon.process.id=32426 -Dcommons.daemon.process.parent=28765 -Dcommons.daemon.version=1.2.2 abort org.apache.catalina.startup.Bootstrap Host: Intel(R) Xeon(R) CPU 3050 @ 2.13GHz, 2 cores, 7G, Red Hat Enterprise Linux Server release 6.5 (Santiago) Time: Tue Feb 18 00:13:06 2020 GMT+3 elapsed time: 18291 seconds (0d 5h 4m 51s) --------------- T H R E A D --------------- Current thread (0x0000000000d75800): JavaThread "Finalizer" daemon [_thread_in_native, id=32435, stack(0x00007fc994890000,0x00007fc994911000)] Stack: [0x00007fc994890000,0x00007fc994911000], sp=0x00007fc99490f770, free space=509k Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code) C [libcrypto.so.1.1+0x16ff58] CRYPTO_get_ex_data+0x8 j org.apache.tomcat.util.net.openssl.OpenSSLEngine.shutdown()V+23 j org.apache.tomcat.util.net.openssl.OpenSSLEngine.finalize()V+5 J 45265 c1 java.lang.ref.Finalizer.runFinalizer(Ljdk/internal/access/JavaLangAccess;)V [email protected] (117 bytes) @ 0x00007fc9a2afd804 [0x00007fc9a2afd1a0+0x0000000000000664] j java.lang.ref.Finalizer$FinalizerThread.run()V+46 [email protected] v ~StubRoutines::call_stub V [libjvm.so+0x821d79] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x319 V [libjvm.so+0x820070] JavaCalls::call_virtual(JavaValue*, Klass*, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x160 V [libjvm.so+0x820141] JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, Thread*)+0x81 V [libjvm.so+0x8bf30c] thread_entry(JavaThread*, Thread*)+0x6c V [libjvm.so+0xd51712] JavaThread::thread_main_inner()+0xe2 V [libjvm.so+0xd5637d] Thread::call_run()+0x10d V [libjvm.so+0xba8b37] thread_native_entry(Thread*)+0xe7 Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j org.apache.tomcat.jni.SSL.freeSSL(J)V+0 j org.apache.tomcat.util.net.openssl.OpenSSLEngine.shutdown()V+23 j org.apache.tomcat.util.net.openssl.OpenSSLEngine.finalize()V+5 J 45265 c1 java.lang.ref.Finalizer.runFinalizer(Ljdk/internal/access/JavaLangAccess;)V [email protected] (117 bytes) @ 0x00007fc9a2afd804 [0x00007fc9a2afd1a0+0x0000000000000664] j java.lang.ref.Finalizer$FinalizerThread.run()V+46 [email protected] v ~StubRoutines::call_stub siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00000000000005b8
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
