On 21-02-15 14:49:46, Jan Klemkow wrote: > On Sat, Feb 13, 2021 at 03:53:48PM +0100, Theo Buehler wrote: > > On Sat, Feb 13, 2021 at 11:58:04AM +0100, Jan Klemkow wrote: > > > A coworker of mine has made tests with LibreSSL [1] and found some > > > regressions. I took his test descriptions and created the following > > > automated regression test. In the repository he described his findings > > > in detail. I kept the numbers of the files and subtests in the target > > > names for now. So, its easier to match it with his files. > > > > > > I don't know how to handle the result of "test-01-ssl". Thats why its > > > just a comment. Someone may have an idea to handle this properly. > > > > > > Any comments, wishes or OK's? > > > > > > [1]: https://github.com/noxxi/libressl-tests > > > > First of all thanks for the effort! > > > > The perl script and probably also the Makefile should have a license. > > > > Please add a check that tests whether the required perl modules are > > installed (p5-IO-Socket-SSL and p5-Net-SSLeay) and otherwise prints > > SKIPPED and their names, so I can install them if they're not present. > > I never remember their exact capitalization and hyphenation... > > > > Various comments inline, and a patch for openssl(1) at the end that may > > simplify some things. > > This is an updated version of the test including comments and wishes > from tb@ and bluhm@. > > OK?
This currently drives openssl(1) for tests, which means that it is testing openssl(1), libssl and libcrypto, when what you're really wanting to test is libcrypto's verifier. While this works, the problem is that a change or breakage in libssl or openssl(1) results in a regress failure for libcrypto. If this is to land in its current form it really should be in regress/usr.bin/openssl - alternatively, it could be reworked to explicitly test libcrypto's APIs and remain here. Some additional comments inline. > Index: regress/lib/libcrypto/validate/Makefile > =================================================================== > RCS file: regress/lib/libcrypto/validate/Makefile > diff -N regress/lib/libcrypto/validate/Makefile > --- /dev/null 1 Jan 1970 00:00:00 -0000 > +++ regress/lib/libcrypto/validate/Makefile 15 Feb 2021 13:38:22 -0000 > @@ -0,0 +1,133 @@ > +# $OpenBSD$ > + > +# Copyright (c) 2021 Jan Klemkow <j.klem...@wemelug.de> > +# > +# Permission to use, copy, modify, and distribute this software for any > +# purpose with or without fee is hereby granted, provided that the above > +# copyright notice and this permission notice appear in all copies. > +# > +# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES > +# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF > +# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR > +# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES > +# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN > +# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF > +# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. > + > +# This regression test is based on manual test descriptions from: > +# https://github.com/noxxi/libressl-tests > + > +# The following port must be installed for the regression tests: > +# p5-IO-Socket-SSL perl interface to SSL sockets > + > +PERL = perl > +OPENSSL ?= openssl > + > +PERL_REQUIRE != perl -Mstrict -Mwarnings -e ' \ > + eval { require IO::Socket::SSL } or print $@; \ > +' > +.if ! empty (PERL_REQUIRE) > +regress: > + @echo "${PERL_REQUIRE}" > + @echo install these perl packages for additional tests > + @echo SKIPPED > +.endif > + > +REGRESS_TARGETS += test-unusual-wildcard-cert-no-CA-client > +REGRESS_TARGETS += test-unusual-wildcard-cert-CA-client I'd would s/unusual-wildcard/inlabel-wildcard/g > +REGRESS_TARGETS += test-common-wildcard-cert-no-CA-client > +REGRESS_TARGETS += test-common wildcard-cert-CA-client There's a space between "test-common" and "wildcard-cert-CA-client" (presumably meant to be a hyphen) - same in two places below. > +REGRESS_TARGETS += test-verify-unusual-wildcard-cert > +REGRESS_TARGETS += test-openssl-verify-common-wildcard-cert > +REGRESS_TARGETS += test-chain-certificates-s_server > +REGRESS_TARGETS += test-alternative-chain > +REGRESS_CLEANUP = cleanup-ssl > +REGRESS_SETUP_ONCE = create-libressl-test-certs > + > +REGRESS_EXPECTED_FAILURES += test-unusual-wildcard-cert-no-CA-client > +REGRESS_EXPECTED_FAILURES += test-common-wildcard-cert-no-CA-client > +REGRESS_EXPECTED_FAILURES += test-common wildcard-cert-CA-client > +REGRESS_EXPECTED_FAILURES += test-verify-unusual-wildcard-cert > +REGRESS_EXPECTED_FAILURES += test-alternative-chain I suspect that some or all of these are expected behaviour, rather than failures. We can review and address this once it lands though. > +create-libressl-test-certs: create-libressl-test-certs.pl > + ${PERL} ${.CURDIR}/$@.pl We can see how this goes, however we may end up wanting to generate the certificates and commit them rather than regenerating on each run. The other advantage is that p5-IO-Socket-SSL would only be needed to regenerate the certificates and not actually run the tests. > +cleanup-ssl: > + rm *.pem *.key > + > +test-unusual-wildcard-cert-no-CA-client: > + # unusual wildcard cert, no CA given to client > + # start client > + ${OPENSSL} s_server -cert server-unusual-wildcard.pem \ > + -key server-unusual-wildcard.pem & \ > + timeout=$$(($$(date +%s) + 5)); \ > + while fstat -p $$! | ! grep -q 'tcp .* \*:4433$$'; \ > + do test $$(date +%s) -lt $$timeout || exit 1; done > + # start client > + echo "Q" | ${OPENSSL} s_client -verify_return_error \ > + | grep "Verify return code: 21" > + > +test-unusual-wildcard-cert-CA-client: > + # unusual wildcard cert, CA given to client > + # start server > + ${OPENSSL} s_server -cert server-unusual-wildcard.pem \ > + -key server-unusual-wildcard.pem & \ > + timeout=$$(($$(date +%s) + 5)); \ > + while fstat -p $$! | ! grep -q 'tcp .* \*:4433$$'; \ > + do test $$(date +%s) -lt $$timeout || exit 1; done > + # start client > + echo "Q" | ${OPENSSL} s_client -CAfile caR.pem \ > + | grep "Verify return code: 0" > + > +test-common-wildcard-cert-no-CA-client: > + # common wildcard cert, no CA given to client > + # start server > + ${OPENSSL} s_server -cert server-common-wildcard.pem \ > + -key server-common-wildcard.pem & \ > + timeout=$$(($$(date +%s) + 5)); \ > + while fstat -p $$! | ! grep -q 'tcp .* \*:4433$$'; \ > + do test $$(date +%s) -lt $$timeout || exit 1; done > + # start client > + echo "Q" | ${OPENSSL} s_client \ > + | grep "Verify return code: 21" > + > +test-common wildcard-cert-CA-client: Space also here. > + # common wildcard cert, CA given to client > + # start server > + ${OPENSSL} s_server -cert server-unusual-wildcard.pem \ > + -key server-unusual-wildcard.pem & \ > + timeout=$$(($$(date +%s) + 5)); \ > + while fstat -p $$! | ! grep -q 'tcp .* \*:4433$$'; \ > + do test $$(date +%s) -lt $$timeout || exit 1; done > + # start client > + echo "Q" | ${OPENSSL} s_client -CAfile caR.pem \ > + | grep "Verify return code: 21" > + > +test-verify-unusual-wildcard-cert: > + # openssl verify, unusual wildcard cert > + ${OPENSSL} verify -CAfile caR.pem server-unusual-wildcard.pem \ > + | grep "server-unusual-wildcard.pem: OK" > + > +test-openssl-verify-common-wildcard-cert: > + # openssl verify, common wildcard cert > + ${OPENSSL} verify -CAfile caR.pem server-common-wildcard.pem \ > + | grep "server-common-wildcard.pem: OK" > + > +test-chain-certificates-s_server: > + # Not all chain certificates are sent in s_server > + # start server > + # ${OPENSSL} s_server -cert server-subca.pem -CAfile subcaR.pem > + ${OPENSSL} s_server -cert server-subca-chainS.pem -CAfile subcaR.pem & \ > + timeout=$$(($$(date +%s) + 5)); \ > + while fstat -p $$! | ! grep -q 'tcp .* \*:4433$$'; \ > + do test $$(date +%s) -lt $$timeout || exit 1; done > + # start client > + ${OPENSSL} s_client -CAfile caR.pem | grep "Verify return code: 0" > + > +test-alternative-chain: > + # alternative chain not found > + ${OPENSSL} verify -verbose -trusted caR.pem -untrusted chainSX.pem \ > + server-subca.pem | grep "server-subca.pem: Ok" > + > +.include <bsd.regress.mk> > Index: regress/lib/libcrypto/validate/create-libressl-test-certs.pl > =================================================================== > RCS file: regress/lib/libcrypto/validate/create-libressl-test-certs.pl > diff -N regress/lib/libcrypto/validate/create-libressl-test-certs.pl > --- /dev/null 1 Jan 1970 00:00:00 -0000 > +++ regress/lib/libcrypto/validate/create-libressl-test-certs.pl 15 Feb > 2021 12:54:58 -0000 > @@ -0,0 +1,111 @@ > +#!/usr/bin/perl > + > +# Copyright (c) 2021 Steffen Ullrich <su...@cpan.org> > +# Public Domain > + > +use strict; > +use warnings; > +use IO::Socket::SSL::Utils; > + > +# primitive CA - ROOT > +my @ca = cert( > + CA => 1, > + subject => { CN => 'ROOT' } > +); > +out('caR.pem', pem(crt => $ca[0])); > +out('caR.key', pem(key => $ca[1])); > + > +# server certificate where SAN contains in-label wildcards which are allowed > by > +# RFC 6125 It is worth noting that per the RFC, a client MAY allow in-label wildcards (this is not a MUST or even a SHOULD). Additionally, various software does not allow or support this (for example, libtls and hence ftp(1)). > +my @leafcert = cert( > + issuer => \@ca, > + purpose => 'server', > + subject => { CN => 'server.local' }, > + subjectAltNames => [ > + [ DNS => 'bar.server.local' ], > + [ DNS => 'www*.server.local'], > + [ DNS => '*.www.server.local'], > + [ DNS => 'foo.server.local' ], > + [ DNS => 'server.local' ], > + ] > +); > +out('server-unusual-wildcard.pem', pem(@leafcert)); > + > +@leafcert = cert( > + issuer => \@ca, > + purpose => 'server', > + subject => { CN => 'server.local' }, > + subjectAltNames => [ > + [ DNS => 'bar.server.local' ], > + [ DNS => '*.www.server.local'], > + [ DNS => 'foo.server.local' ], > + [ DNS => 'server.local' ], > + ] > +); > +out('server-common-wildcard.pem', pem(@leafcert)); > + > +# alternative CA - OLD_ROOT > +my @caO = cert( > + CA => 1, > + subject => { CN => 'OLD_ROOT' } > +); > +out('caO.pem', pem(crt => $caO[0])); > +out('caO.key', pem(key => $caO[1])); > + > +# alternative ROOT CA, signed by OLD_ROOT, same key as other ROOT CA > +my @caX = cert( > + issuer => \@caO, > + CA => 1, > + subject => { CN => 'ROOT' }, > + key => $ca[1], > +); > +out('caX.pem', pem(crt => $caX[0])); > +out('caX.key', pem(key => $caX[1])); > + > +# subCA below ROOT > +my @subcaR = cert( > + issuer => \@ca, > + CA => 1, > + subject => { CN => 'SubCA.of.ROOT' } > +); > +out('subcaR.pem', pem(crt => $subcaR[0])); > +out('subcaR.key', pem(key => $subcaR[1])); > +out('chainSX.pem', pem($subcaR[0]), pem($caX[0])); > + > +@leafcert = cert( > + issuer => \@subcaR, > + purpose => 'server', > + subject => { CN => 'server.subca.local' }, > + subjectAltNames => [ > + [ DNS => 'server.subca.local' ], > + ] > +); > +out('server-subca.pem', pem(@leafcert)); > +out('server-subca-chainSX.pem', pem(@leafcert, $subcaR[0], $caX[0])); > +out('server-subca-chainS.pem', pem(@leafcert, $subcaR[0])); > + > + > +sub cert { CERT_create(not_after => 10*365*86400+time(), @_) } > +sub pem { > + my @default = qw(crt key); > + my %m = (key => \&PEM_key2string, crt => \&PEM_cert2string); > + my $result = ''; > + while (my $f = shift(@_)) { > + my $v; > + if ($f =~m{^(key|crt)$}) { > + $v = shift(@_); > + } else { > + $v = $f; > + $f = shift(@default) || 'crt'; > + } > + $f = $m{$f} || die "wrong key $f"; > + $result .= $f->($v); > + } > + return $result; > +} > + > +sub out { > + my $file = shift; > + open(my $fh,'>',"$file") or die "failed to create $file: $!"; > + print $fh @_ > +}
