Package: sbuild
Version: 0.65.1-1
Severity: wishlist
Tags: patch
User: reproducible-bui...@lists.alioth.debian.org
Usertags: toolchain
Control: block -1 by 774359
Hi,
to verify a package build for reproducible builds [1], it is necessary
to carry it out in an environment with the right package versions. This
information is stored in .buildinfo files.
[1] https://wiki.debian.org/ReproducibleBuilds
I wrote a thin wrapper for sbuild which figures out the right timestamp
from snapshot.debian.org that contains the right package versions and
then installs those in sbuild using its hook functionality.
A more detailed explanation can be found in this email:
https://lists.alioth.debian.org/pipermail/reproducible-builds/Week-of-Mon-20141229/000613.html
The functionality depends on the resolution of bug #774359, thus adding
it as a blocker for this bug.
You can find the code implementing this functionality attached or in the
pu/reproducible branch in this repository:
http://anonscm.debian.org/cgit/reproducible/sbuild.git/
Thanks!
cheers, josch
>From b926810c8d9a1111897c33aca43f0e782ca71979 Mon Sep 17 00:00:00 2001
From: josch <j.scha...@email.de>
Date: Thu, 1 Jan 2015 22:41:21 +0100
Subject: [PATCH 1/2] Add first proof-of-concept for srebuild
- limited to working on Debian sid, main
- limited to a single snapshot timestamp
---
bin/srebuild | 271 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
bin/srebuild-hook | 153 ++++++++++++++++++++++++++++++
2 files changed, 424 insertions(+)
create mode 100755 bin/srebuild
create mode 100755 bin/srebuild-hook
diff --git a/bin/srebuild b/bin/srebuild
new file mode 100755
index 0000000..fb41c1f
--- /dev/null
+++ b/bin/srebuild
@@ -0,0 +1,271 @@
+#!/usr/bin/perl
+#
+# Copyright 2014 Johannes Schauer
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+
+use strict;
+use warnings;
+
+use Dpkg::Control;
+use Dpkg::Compression::FileHandle;
+use Dpkg::Deps;
+use DateTime::Format::Strptime;
+use Compress::Zlib;
+use File::Basename;
+use Digest::SHA qw(sha256_hex);
+
+eval {
+ require LWP::Simple;
+ require LWP::UserAgent;
+ no warnings;
+ $LWP::Simple::ua = LWP::UserAgent->new(agent => 'LWP::UserAgent/srebuild');
+};
+if ($@) {
+ if ($@ =~ m/Can\'t locate LWP/) {
+ die "Unable to run: the libwww-perl package is not installed";
+ } else {
+ die "Unable to run: Couldn't load LWP::Simple: $@";
+ }
+}
+
+eval {
+ require JSON;
+};
+if ($@) {
+ if ($@ =~ m/Can\'t locate JSON/) {
+ die "Unable to run: the libjson-perl package is not installed";
+ } else {
+ die "Unable to run: Couldn't load JSON: $@";
+ }
+}
+
+# this subroutine is from debsnap(1)
+sub fetch_json_page
+{
+ my ($json_url) = @_;
+ my $content = LWP::Simple::get($json_url);
+ return unless defined $content;
+ my $json = JSON->new();
+ my $json_text = $json->allow_nonref->utf8->relaxed->decode($content);
+ return $json_text;
+}
+
+sub parse_buildinfo {
+ my $buildinfo = shift;
+
+ my $fh = Dpkg::Compression::FileHandle->new(filename => $buildinfo);
+
+ my $cdata = Dpkg::Control->new(type => CTRL_INDEX_SRC);
+ if (not $cdata->parse($fh, $buildinfo)) {
+ die "cannot parse"
+ }
+ my $arch = $cdata->{"Build-Architecture"};
+ if (not defined($arch)) {
+ die "need Build-Architecture field";
+ }
+ my $checksums = $cdata->{"Checksums-Sha256"};
+ if (not defined($checksums)) {
+ die "need Checksums-Sha256 field";
+ }
+ my $environ = $cdata->{"Build-Environment"};
+ if (not defined($environ)) {
+ die "need Build-Environment field";
+ }
+ close $fh;
+
+ # remove newline from start and end
+ $checksums =~ s{^\Q$/\E}{};
+ $checksums = [ map { [ split /\s+/ ] } ( split /\s*\n\s*/, $checksums ) ];
+
+ my @environ = ();
+ foreach my $dep (split(/\s*,\s*/m, $environ)) {
+ my $pkg = Dpkg::Deps::Simple->new($dep);
+ if (not defined($pkg->{package})) {
+ die "name undefined";
+ }
+ if (defined($pkg->{relation})) {
+ if ($pkg->{relation} ne "=") {
+ die "wrong relation";
+ }
+ if (not defined($pkg->{version})) {
+ die "version undefined"
+ }
+ } else {
+ die "no version";
+ }
+ push @environ, { name => $pkg->{package},
+ architecture => ( $pkg->{archqual} || $arch ),
+ version => $pkg->{version}
+ };
+ }
+
+ return $arch, $checksums, @environ
+}
+
+my $archive = "debian";
+my $suite = "sid";
+my $area = "main";
+
+my %reqpkgs = ();
+my @timestamps = ();
+
+my $dtparser = DateTime::Format::Strptime->new(
+ pattern => '%Y%m%dT%H%M%SZ',
+ on_error => 'croak',
+);
+
+my $buildinfo = shift @ARGV;
+if (not defined($buildinfo)) {
+ die "need buildinfo filename";
+}
+
+my ($arch, $checksums, @environ) = parse_buildinfo $buildinfo;
+
+print STDERR "check original checksums\n";
+
+my $dsc_fname;
+
+foreach my $sum (@{$checksums}) {
+ my ($chksum, $size, $fname) = @{$sum};
+ my $size2 = (stat($fname))[7];
+ if ($size != $size2) {
+ print "$size\n";
+ print "$size2\n";
+ die "size mismatch for $fname\n"
+ }
+ open my $fh, '<', $fname;
+ my $chksum2 = sha256_hex <$fh>;
+ if ($chksum ne $chksum2) {
+ print "$chksum\n";
+ print "$chksum2\n";
+ die "checksum mismatch for $fname\n";
+ }
+ close $fh;
+ if ($fname =~ /.dsc/) {
+ if (defined($dsc_fname)) {
+ die "more than one dsc\n";
+ }
+ $dsc_fname = $fname;
+ }
+}
+
+if (not defined($dsc_fname)) {
+ die "no dsc found\n";
+}
+
+print STDERR "retrieve last seen snapshot timestamps\n";
+
+foreach my $pkg (@environ) {
+ $reqpkgs{"$pkg->{name}:$pkg->{architecture}=$pkg->{version}"} = 1;
+ my $url = "http://snapshot.debian.org/mr/binary/$pkg->{name}/$pkg->{version}/binfiles?fileinfo=1";
+ my $json_text = fetch_json_page($url);
+ unless ($json_text && @{$json_text->{result}}) {
+ die "Unable to retrieve information for $pkg->{name} from $url.\n";
+ }
+ my $hash = undef;
+ if (scalar @{$json_text->{result}} == 1) {
+ if (@{$json_text->{result}}[0]->{architecture} ne "all") {
+ die "expected arch:all\n";
+ }
+ $hash = ${$json_text->{result}}[0]->{hash};
+ } else {
+ foreach my $result (@{$json_text->{result}}) {
+ if ($result->{architecture} eq $arch) {
+ $hash = $result->{hash};
+ last;
+ }
+ }
+ }
+ if (not defined($hash)) {
+ die "cannot find architecture for $pkg->{name}\n";
+ }
+ my @first_seen = grep { $_->{archive_name} eq $archive } @{$json_text->{fileinfo}->{$hash}};
+ if (scalar @first_seen != 1) {
+ die "more than one package with the same hash\n";
+ }
+ @first_seen = map { $_->{first_seen} } @first_seen;
+ push @timestamps, $dtparser->parse_datetime($first_seen[0]);
+}
+
+# @timestamps = sort { DateTime->compare($a, $b) } @timestamps;
+@timestamps = sort @timestamps;
+
+my $newest = $timestamps[$#timestamps];
+$newest = $newest->strftime("%Y%m%dT%H%M%SZ");
+
+my $snapshot_url = "http://snapshot.debian.org/archive/$archive/$newest/dists/$suite/$area/binary-$arch/Packages.gz";;
+
+print STDERR "download Packages.gz\n";
+
+my $response = LWP::Simple::get($snapshot_url);
+
+my $dest = Compress::Zlib::memGunzip($response)
+ or die "Cannot uncompress\n";
+
+print STDERR "process Packages.gz\n";
+
+open my $fh, '<', \$dest;
+
+while (1) {
+ my $cdata = Dpkg::Control->new(type => CTRL_INDEX_SRC);
+ last if not $cdata->parse($fh, "Packages.gz");
+ my $pkgname = $cdata->{"Package"};
+ next if not defined($pkgname);
+ my $pkgver = $cdata->{"Version"};
+ my $pkgarch;
+ if ($cdata->{"Architecture"} eq "all") {
+ $pkgarch = $arch;
+ } else {
+ $pkgarch = $cdata->{"Architecture"};
+ }
+ my $key = "$pkgname:$pkgarch=$pkgver";
+ if (exists $reqpkgs{$key}) {
+ delete $reqpkgs{$key};
+ }
+}
+
+if (scalar (keys %reqpkgs) != 0) {
+ die "some of the requested packages are not part of this snapshot";
+}
+
+print "architecture = $arch\n";
+print "mirror = http://snapshot.debian.org/archive/$archive/$newest/\n";;
+
+my $bn_buildinfo = basename $buildinfo;
+
+my $retval = system "sbuild", "--arch=$arch", "--dist=wheezy",
+ "--pre-build-command=cp /home/josch/sbuild/bin/srebuild-hook $buildinfo %SBUILD_CHROOT_DIR/tmp",
+ "--chroot-setup-command=/tmp/srebuild-hook chroot-setup /tmp/$bn_buildinfo $newest",
+ "--starting-build-commands=/tmp/srebuild-hook starting-build /tmp/$bn_buildinfo",
+ $dsc_fname;
+$retval >>= 8;
+if ($retval != 0) {
+ die "failed";
+}
+
+foreach my $sum (@{$checksums}) {
+ my ($chksum, $size, $fname) = @{$sum};
+ my $size2 = (stat($fname))[7];
+ if ($size != $size2) {
+ print "$size\n";
+ print "$size2\n";
+ die "size mismatch for $fname\n"
+ }
+ open my $fh, '<', $fname;
+ my $chksum2 = sha256_hex <$fh>;
+ if ($chksum ne $chksum2) {
+ print "$chksum\n";
+ print "$chksum2\n";
+ die "checksum mismatch for $fname\n";
+ }
+ close $fh;
+}
diff --git a/bin/srebuild-hook b/bin/srebuild-hook
new file mode 100755
index 0000000..056e445
--- /dev/null
+++ b/bin/srebuild-hook
@@ -0,0 +1,153 @@
+#!/usr/bin/perl
+#
+# Copyright 2014 Johannes Schauer
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+
+use strict;
+use warnings;
+
+use Dpkg::Control;
+use Dpkg::Compression::FileHandle;
+use Dpkg::Deps;
+
+sub none(&@) {
+ my $code = shift;
+ foreach (@_) {
+ return 0 if $code->();
+ }
+ return 1;
+}
+
+sub system_fatal {
+ my @args = @_;
+ print "srebuild: executing: @args\n";
+ my $retval = system @args;
+ $retval >>= 8;
+ if ($retval != 0) {
+ die "failed: @args";
+ }
+}
+
+sub parse_buildinfo {
+ my $buildinfo = shift;
+
+ my $fh = Dpkg::Compression::FileHandle->new(filename => $buildinfo);
+
+ my $cdata = Dpkg::Control->new(type => CTRL_INDEX_SRC);
+ if (not $cdata->parse($fh, $buildinfo)) {
+ die "cannot parse"
+ }
+ my $arch = $cdata->{"Build-Architecture"};
+ if (not defined($arch)) {
+ die "need Build-Architecture field"
+ }
+ my $environ = $cdata->{"Build-Environment"};
+ if (not defined($environ)) {
+ die "need Build-Environment field"
+ }
+ close $fh;
+
+ my @environ = ();
+ foreach my $dep (split(/\s*,\s*/m, $environ)) {
+ my $pkg = Dpkg::Deps::Simple->new($dep);
+ if (not defined($pkg->{package})) {
+ die "name undefined";
+ }
+ if (defined($pkg->{relation})) {
+ if ($pkg->{relation} ne "=") {
+ die "wrong relation";
+ }
+ if (not defined($pkg->{version})) {
+ die "version undefined"
+ }
+ } else {
+ die "no version";
+ }
+ push @environ, { name => $pkg->{package},
+ architecture => ( $pkg->{archqual} || $arch ),
+ version => $pkg->{version}
+ };
+ }
+
+ return $arch, @environ
+}
+
+sub chroot_setup {
+ my $buildinfo = shift;
+ my @timestamps = @_;
+
+ my ($arch, @environ) = parse_buildinfo $buildinfo;
+
+ @environ = map { "$_->{name}:$_->{architecture}=$_->{version}" } @environ;
+
+ my $fh;
+ open $fh, '>', '/etc/apt/apt.conf.d/80no-check-valid-until';
+ print $fh 'Acquire::Check-Valid-Until "false";';
+ close $fh;
+
+ open $fh, '>', '/etc/apt/apt.conf.d/99no-install-recommends';
+ print $fh 'APT::Install-Recommends "0";';
+ close $fh;
+
+ open $fh, '>', '/etc/apt/sources.list';
+ foreach my $timestamp (@timestamps) {
+ print $fh "deb http://snapshot.debian.org/archive/debian/$timestamp/ sid main\n";
+ }
+ close $fh;
+
+ system_fatal "apt-get", "update";
+
+ system_fatal "apt-get", "--yes", "--force-yes", "install", @environ;
+}
+
+sub starting_build {
+ my $buildinfo = shift;
+
+ my ($arch, @environ) = parse_buildinfo $buildinfo;
+
+ @environ = map { "$_->{name}:$_->{architecture}=$_->{version}" } @environ;
+
+ open my $fh, '-|', 'dpkg-query --show --showformat \'${Package}:${Architecture}=${Version}\n\'';
+ my @installed = ();
+ while (my $line = <$fh>) {
+ chomp $line;
+ # make arch:all packages build-arch packages
+ $line =~ s/:all=/:$arch=/;
+ push @installed, $line;
+ }
+
+ foreach my $dep (@environ) {
+ if (none {$_ eq $dep} @installed) {
+ die "require $dep to be installed but it is not";
+ }
+ }
+ print "srebuild: all packages are in the correct version\n"
+}
+
+my $mode = shift @ARGV;
+my $buildinfo = shift @ARGV;
+if (not defined($buildinfo)) {
+ die "need buildinfo filename";
+}
+
+if ($mode eq "chroot-setup") {
+ my @timestamps = @ARGV;
+ if (scalar @timestamps == 0) {
+ die "need timestamp";
+ }
+
+ chroot_setup $buildinfo, @timestamps;
+} elsif ($mode eq "starting-build") {
+ starting_build $buildinfo;
+} else {
+ die "invalid mode: $mode";
+}
--
2.0.1
>From 542ebc22f12b91d1f595db6a9f834f0e141784e7 Mon Sep 17 00:00:00 2001
From: josch <j.scha...@email.de>
Date: Fri, 2 Jan 2015 12:04:57 +0100
Subject: [PATCH 2/2] srebuild: add man page, copyright and adapt makefiles
---
bin/Makefile.am | 2 ++
bin/srebuild | 2 +-
debian/changelog | 1 +
debian/copyright | 17 ++++++++++++++++
debian/sbuild.install | 1 +
man/Makefile.am | 1 +
man/srebuild.1.in | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++
7 files changed, 78 insertions(+), 1 deletion(-)
create mode 100644 man/srebuild.1.in
diff --git a/bin/Makefile.am b/bin/Makefile.am
index 423a313..760daf2 100644
--- a/bin/Makefile.am
+++ b/bin/Makefile.am
@@ -35,6 +35,7 @@ bin_SCRIPTS = \
sbuild-shell \
sbuild-hold \
sbuild-unhold \
+ srebuild \
buildd \
buildd-mail \
buildd-uploader \
@@ -48,6 +49,7 @@ sbin_SCRIPTS = \
sbuilddata_SCRIPTS = \
create-chroot \
+ srebuild-hook \
dobuildlog
doc_DATA = \
diff --git a/bin/srebuild b/bin/srebuild
index fb41c1f..1d2172b 100755
--- a/bin/srebuild
+++ b/bin/srebuild
@@ -243,7 +243,7 @@ print "mirror = http://snapshot.debian.org/archive/$archive/$newest/\n";;
my $bn_buildinfo = basename $buildinfo;
my $retval = system "sbuild", "--arch=$arch", "--dist=wheezy",
- "--pre-build-command=cp /home/josch/sbuild/bin/srebuild-hook $buildinfo %SBUILD_CHROOT_DIR/tmp",
+ "--pre-build-command=cp /usr/share/sbuild/srebuild-hook $buildinfo %SBUILD_CHROOT_DIR/tmp",
"--chroot-setup-command=/tmp/srebuild-hook chroot-setup /tmp/$bn_buildinfo $newest",
"--starting-build-commands=/tmp/srebuild-hook starting-build /tmp/$bn_buildinfo",
$dsc_fname;
diff --git a/debian/changelog b/debian/changelog
index 6aeef32..ec7c32f 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -10,6 +10,7 @@ sbuild (0.65.1-1) UNRELEASED; urgency=medium
* Move running of pre-build-commands until after 'Chroot Dir' is defined
- this prevents the %SBUILD_CHROOT_DIR percentage escape being empty
when running the pre-build-commands hook (Closes: #774359)
+ * add srebuild, a wrapper to make reproducible builds from .buildinfo files
-- Roger Leigh <rle...@debian.org> Sun, 30 Nov 2014 22:44:47 +0000
diff --git a/debian/copyright b/debian/copyright
index 990425e..f7c4fdc 100644
--- a/debian/copyright
+++ b/debian/copyright
@@ -69,6 +69,23 @@ the GNU General Public License, version 3 or later:
On Debian systems, the complete text of the GNU General Public
License, version 3, can be found in /usr/share/common-licenses/GPL-3.
+Files:
+ bin/srebuild
+ bin/srebuild-hooks
+ man/srebuild.1.in
+Copyright: 2015 Johannes Schauer <j.scha...@email.de>
+License: Expat
+ Permission is hereby granted, free of charge, to any person
+ obtaining a copy of this software and associated documentation
+ files (the "Software"), to deal in the Software without
+ restriction, including without limitation the rights to use,
+ copy, modify, merge, publish, distribute, sublicense, and/or sell
+ copies of the Software, and to permit persons to whom the
+ Software is furnished to do so, subject to the following
+ conditions:
+ .
+ The above copyright notice and this permission notice shall be
+ included in all copies or substantial portions of the Software.
Debian packaging copyright:
© 2001-2003 Rick Younie <you...@debian.org>
diff --git a/debian/sbuild.install b/debian/sbuild.install
index 695c3ee..7b785d8 100644
--- a/debian/sbuild.install
+++ b/debian/sbuild.install
@@ -1,5 +1,6 @@
debian/install/etc/sbuild etc
debian/install/usr/bin/sbuild* usr/bin
+debian/install/usr/bin/srebuild usr/bin
debian/install/usr/sbin/sbuild* usr/sbin
debian/install/usr/share/doc/sbuild usr/share/doc
debian/install/usr/share/man/man1/sbuild* usr/share/man/man1
diff --git a/man/Makefile.am b/man/Makefile.am
index 9d1b8d0..7ee2b9d 100644
--- a/man/Makefile.am
+++ b/man/Makefile.am
@@ -28,6 +28,7 @@ man_MANS = \
buildd-uploader.1 \
buildd-vlog.1 \
buildd-watcher.1 \
+ srebuild.1 \
sbuild.1 \
sbuild.conf.5 \
sbuild-abort.1 \
diff --git a/man/srebuild.1.in b/man/srebuild.1.in
new file mode 100644
index 0000000..3d96c53
--- /dev/null
+++ b/man/srebuild.1.in
@@ -0,0 +1,55 @@
+.\" # Copyright 2014 Johannes Schauer
+.\"
+.\" Permission is hereby granted, free of charge, to any person obtaining a copy
+.\" of this software and associated documentation files (the "Software"), to deal
+.\" in the Software without restriction, including without limitation the rights
+.\" to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+.\" copies of the Software, and to permit persons to whom the Software is
+.\" furnished to do so, subject to the following conditions:
+.\"
+.\" The above copyright notice and this permission notice shall be included in
+.\" all copies or substantial portions of the Software.
+.so defs.man
+.TH SREBUILD 1 "\*[RELEASE_DATE]" "Version \*[VERSION]" "Debian sbuild"
+.SH NAME
+srebuild \- sbuild wrapper to make a reproducible build using a .buildinfo file
+.SH SYNOPSIS
+.B srebuild package.buildinfo
+.SH DESCRIPTION
+\fBsrebuild\fR is a thin wrapper around sbuild. Given a \fI.buildinfo\fP file,
+it first finds a timestamp of Debian Sid from snapshot.debian.org which
+contains the requested packages in their exact versions. It then runs sbuild
+with the right architecture as given by the .buildinfo file and the right base
+system to upgrade from, as given by the version of the base-files package
+version in the .buildinfo file. Using two hooks it will install the right
+package versions and verify that the installed packages are in the right
+version before the build starts.
+.SH OPTIONS
+There are no options yet.
+.SH BUGS
+\fBsrebuild\fR will give up if not all packages can be found in a single
+snapshot. It should be possible to retrieve the right packages from multiple
+timestamps.
+.PP
+Querying the snapshot.debian.org API for the right timestamp with the correct
+versions takes some time. It should be possible to manually supply the right
+timestamp on the command line, allowing to skip the step of figuring out the
+right timestamp. Conversely, every run should output the timestamp it found so
+that it can be used to skip the step for repeated runs.
+.PP
+The right snapshot is expected to be found in Debian sid, main. It should be
+possible to find it in stable and testing as well or even mix different suites.
+It should be possible to find the right packages in contrib and non free, too.
+Additionally, it might be worthwhiel to search debian-archive,
+debian-backports, debian-ports, debian-security or debian-volatile for the
+right version.
+.PP
+Currently, wheezy will always be used as the base. Instead, the version of the
+base-files package should be used to select the optimal release to upgrade
+from.
+.SH AUTHORS
+Johannes Schauer <j.scha...@email.de>
+.SH COPYRIGHT
+Copyright 2015 Johannes Schauer <j.scha...@email.de>
+.SH SEE ALSO
+.BR sbuild (1)
--
2.0.1
