This is an automated email from the git hooks/post-receive script. terceiro pushed a commit to branch master in repository devscripts.
commit f3fd7952b8d38438f193076521f9067d742f9c94 Author: Antonio Terceiro <[email protected]> Date: Sat Sep 3 13:12:27 2016 -0300 New script: debrepro --- README | 8 +++ debian/changelog | 5 ++ debian/control | 4 ++ po4a/devscripts-po4a.conf | 2 + scripts/debc.pl | 1 + scripts/debrepro.1 | 68 +++++++++++++++++++ scripts/debrepro.sh | 162 ++++++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 250 insertions(+) diff --git a/README b/README index 88da0f5..d03809e 100644 --- a/README +++ b/README @@ -112,6 +112,14 @@ And now, in mostly alphabetical order, the scripts: version to upload, and then calls dupload or dput to actually perform the upload. [dupload | dput, ssh-client] +- debrepro: A script that tests reproducibility of Debian packages. It will + build a given source directory twice, with a set of variation between the + first and second build, and compare the binary packages produced. If + diffoscope is installed, it is used to compare non-matching binaries. If + disorderfs is installed, it is used during the build to inject + non-determinism in filesystem listing operations. + [faketime, diffoscope, disorderfs] + - debrsign: This transfers a .changes/.dsc pair to a remote machine for signing, and runs debsign on the remote machine over an SSH connection. [gnupg | gnupg2, debian-keyring, ssh-client] diff --git a/debian/changelog b/debian/changelog index 010da48..4b62318 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,9 +1,14 @@ devscripts (2.16.8) UNRELEASED; urgency=medium + [ Paul Wise ] * Add some more Debian development & QA tools to Suggests: adequate autopkgtest bls-standalone check-all-the-things duck how-can-i-help piuparts ratt + [ Antonio Terceiro ] + * New script: debrepro, a simple reproducibility tester specific for Debian + packages. + -- Paul Wise <[email protected]> Sat, 27 Aug 2016 13:58:14 +0800 devscripts (2.16.7) unstable; urgency=medium diff --git a/debian/control b/debian/control index bd81de5..45ea20e 100644 --- a/debian/control +++ b/debian/control @@ -82,8 +82,10 @@ Suggests: adequate, cvs-buildpackage, devscripts-el, diffoscope, + disorderfs, dose-extra (>= 4.0), duck, + faketime, gnuplot, gpgv | gpgv2, how-can-i-help, @@ -148,6 +150,8 @@ Description: scripts to make the life of a Debian Package maintainer easier [wdiff, patchutils] - debi: install a just-built package - debpkg: dpkg wrapper to be able to manage/test packages without su + - debrepro: reproducibility tester for debian packages [faketime, + diffoscope, disorderfs] - debrelease: wrapper around dupload or dput [dupload | dput, ssh-client] - debsign, debrsign: sign a .changes/.dsc pair without needing any of the rest of the package to be present; can sign the pair remotely diff --git a/po4a/devscripts-po4a.conf b/po4a/devscripts-po4a.conf index ce956aa..2d25941 100644 --- a/po4a/devscripts-po4a.conf +++ b/po4a/devscripts-po4a.conf @@ -46,6 +46,8 @@ $lang:$lang/debi.$lang.1 add_$lang:?add_$lang/translator_man.add [type:man] ../scripts/debpkg.1 \ $lang:$lang/debpkg.$lang.1 add_$lang:?add_$lang/translator_man.add +[type:man] ../scripts/debrepro.1 \ + $lang:$lang/debrepro.$lang.1 add_$lang:?add_$lang/translator_man.add [type:man] ../scripts/debrelease.1 \ $lang:$lang/debrelease.$lang.1 add_$lang:?add_$lang/translator_man.add [type:docbook] ../scripts/deb-reversion.dbk \ diff --git a/scripts/debc.pl b/scripts/debc.pl new file mode 120000 index 0000000..1a1d45b --- /dev/null +++ b/scripts/debc.pl @@ -0,0 +1 @@ +debi.pl \ No newline at end of file diff --git a/scripts/debrepro.1 b/scripts/debrepro.1 new file mode 100644 index 0000000..216d47e --- /dev/null +++ b/scripts/debrepro.1 @@ -0,0 +1,68 @@ +.TH DEBREPRO 1 "Debian Utilities" "DEBIAN" \" -*- nroff -*- +.SH NAME +debrepro \- reproduciblity tester for Debian packages +.SH SYNOPSIS +\fBdebrepro\fR [\fISOURCEDIR\fR] +.SH DESCRIPTION +\fBdebrepro\fR will build a given source directory twice, with a set of +variation between the first and second build, and compare the binary packages +produced. If \fBdiffoscope\fR is installed, it is used to compare non-matching +binaries. If \fBdisorderfs\fR is installed, it is used during the build to +inject non-determinism in filesystem listing operations. +.PP +\fISOURCEDIR\fR must be a directory containing an unpacked Debian source +package. If \fISOURCEDIR\fR is ommited, the current directory is assumed. +.SH OUTPUT DIRECTORY +At the very end of a build, \fBdebrepro\fR will inform the location of the +output directory where the build artifacts can be found. The contents of this +directory is as follows: +.TP +.BR \fI«outdir»/first\fR +contains the results of the first build, including a copy of the source tree, +and the resulting binary packages. +.TP +.BR \fI«outdir»/first/build.sh\fR +contains the exact build script that was used in the first build. +.TP +.BR \fI«outdir»/second\fR +contains the results of the second build, including a copy of the source tree, +and the resulting binary packages. +.TP +.BR \fI«outdir»/second/build.sh\fR +contains the exact build script that was used in the second build. +.PP +diff'ing \fI«outdir»/first/build.sh\fR and \fI«outdir»/second/build.sh\fR is an +excellent way of figuring out exactly what changed between the two builds. +.SH SUPPORTED VARIATIONS +.TP +.BR \fBUSER\fR +The \fI$USER\fR environment name will contain different values between the +first and second builds. +.TP +.BR \fBPATH\fR +During the second build, a fake, unexisting directory will be appended to the +\fI$PATH\fR environment variable. +.TP +.BR \fBumask\fR +The builds will use different umask settings. +.TP +.BR \fBlocale\fR +Both \fI$LC_ALL\fR and \fI$LANG\fR will be different across the two builds. +.TP +.BR \fBtimezone\fR +\fI$TZ\fR will be different across builds. +.TP +.BR \fBfilesystem\ ordering\fR +If \fBdisorderfs\fR is installed, both builds will be done under a disorderfs +overlay directory. This will cause filesystem listing operations to be return +items in a non-deterministic order. +.TP +.BR \fBtime\fR +The second build will be executed 213 days, 7 hours and 13 minutes in the +future with regards to the current time. +.BR +.SH "SEE ALSO" +.BR diffoscope (1), +.BR disorderfs (1), +.SH AUTHOR +Antonio Terceiro <[email protected]>. diff --git a/scripts/debrepro.sh b/scripts/debrepro.sh new file mode 100755 index 0000000..70e97c7 --- /dev/null +++ b/scripts/debrepro.sh @@ -0,0 +1,162 @@ +#!/bin/sh + +# debrepro: a reproducibility tester for Debian packages +# +# © 2016 Antonio Terceiro <[email protected]> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <https://www.gnu.org/licenses/>. + +set -eu + +first_banner=y +banner() { + if [ "$first_banner" = n ]; then + echo + fi + echo "$@" | sed -e 's/./=/g' + echo "$@" + echo "$@" | sed -e 's/./=/g' + echo + first_banner=n +} + +variation() { + echo + echo "# Variation:" "$@" +} + +vary() { + local first="$1" + local second="${2:-}" + if [ -z "$second" ]; then + second="$first" + first='' + fi + if [ "$which_build" = 'first' ]; then + if [ -n "$first" ]; then + echo "$first" + fi + else + echo "$second" + fi +} + +create_build_script() { + echo 'set -eu' + + echo + echo "# this script must be run from inside an unpacked Debian source" + echo "# package" + echo + + variation PATH + vary 'export PATH="$PATH":/i/capture/the/path' + + variation USER + vary 'export USER=user1' 'export USER=user2' + + variation umask + vary 'umask 0022' 'umask 0002' + + variation locale + vary 'export LC_ALL=C.UTF-8 LANG=C.UTF-8' \ + 'export LC_ALL=pt_BR.UTF-8 LANG=pt_BR.UTF-8' + + variation timezone + vary 'export TZ=UTC' \ + 'export TZ=Asia/Tokyo' + + if which disorderfs >/dev/null; then + variation filesystem ordering + echo 'mkdir ../disorderfs' + echo 'disorderfs --shuffle-dirents=yes $(pwd) ../disorderfs' + echo 'trap "cd .. && fusermount -u disorderfs && rmdir disorderfs" INT TERM EXIT' + echo 'cd ../disorderfs' + fi + + echo + echo 'dpkg-source --before-build .' + echo 'fakeroot debian/rules clean' + + variation date + vary 'fakeroot debian/rules binary' \ + 'faketime "+213days +7hours +13minutes" fakeroot debian/rules binary' +} + + +build() { + export which_build="$1" + mkdir "$tmpdir/$which_build" + cp -r "$SOURCE" "$tmpdir/$which_build/source" + cd "$tmpdir/$which_build/source" + + create_build_script > ../build.sh + + sh ../build.sh + + cd - +} + +binmatch() { + local h1=$(sha1sum "$1" | awk '{print($1)}') + local h2=$(sha1sum "$2" | awk '{print($1)}') + test "$h1" = "$h2" +} + +compare() { + rc=0 + for first_deb in "$tmpdir"/first/*.deb; do + deb="$(basename "$first_deb")" + second_deb="$tmpdir"/second/"$deb" + if binmatch "$first_deb" "$second_deb"; then + echo "✓ $deb: binaries match" + else + echo "" + rc=1 + if which diffoscope >/dev/null; then + diffoscope "$first_deb" "$second_deb" || true + else + echo "✗ $deb: binaries don't match" + fi + fi + done + if [ "$rc" -ne 0 ]; then + echo "E: package is not reproducible." + if ! which diffoscope >/dev/null; then + echo "I: install diffoscope for a deeper comparison between binaries" + fi + fi + return "$rc" +} + +SOURCE="${1:-}" +if [ -z "$SOURCE" ]; then + SOURCE="$(pwd)" +fi +if [ ! -f "$SOURCE/debian/changelog" ]; then + echo "E: $SOURCE does not look like a Debian source package" + exit 2 +fi + +tmpdir=$(mktemp --directory --tmpdir debrepro.XXXXXXXXXX) +trap "echo; echo 'I: artifacts left in $tmpdir'" INT TERM EXIT + +banner "First build" +build first + +banner "Second build" +build second + +banner "Comparing binaries" +compare first second -- Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/collab-maint/devscripts.git _______________________________________________ devscripts-devel mailing list [email protected] http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/devscripts-devel
