The reason https://self-signed.badssl.com works is because Emacs 24 does not do SNI; badssl.com serves up a cert valid for "*.badssl.com" in the absence of SNI, which is perfectly valid for self-signed.badssl.com.
This is apparently fixed upstream already: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=20465
