Hi Paul & James,
Thanks for the review :)
Updated patch attached:
* Moved away from help2man.
* Changed cache filename suffix to .bz2. (It was a mistake)
* Added a missing python3-xdg to the packages "required" list in
debian/control.)
* Dropped the rather verbose "2017-08-27 09:48:02 " prefix from
log messages.
* Dropped aliasing of "architecture" to prevent an unused variable
warning from static analysis tools.
* Wrapped some long lines.
commit 381f7070f28940796ab795c600764cfdc0429924
Author: Chris Lamb <ch...@chris-lamb.co.uk>
Date: Sun Aug 20 11:18:29 2017 -0700
reproducible-check: New script to report on reproducibility status of
installed packages. (Closes: #872514)
Signed-off-by: Chris Lamb <la...@debian.org>
README | 3 +
debian/control | 8 ++
debian/copyright | 6 ++
scripts/Makefile | 2 +-
scripts/reproducible-check | 188
+++++++++++++++++++++++++++++++++++++++++++
scripts/reproducible-check.1 | 22 +++++
scripts/setup.py | 2 +-
7 files changed, 229 insertions(+), 2 deletions(-)
Regards,
--
,''`.
: :' : Chris Lamb
`. `'` la...@debian.org / chris-lamb.co.uk
`-
From 381f7070f28940796ab795c600764cfdc0429924 Mon Sep 17 00:00:00 2001
From: Chris Lamb <ch...@chris-lamb.co.uk>
Date: Sun, 20 Aug 2017 11:18:29 -0700
Subject: [PATCH] reproducible-check: New script to report on reproducibility
status of installed packages. (Closes: #872514)
Signed-off-by: Chris Lamb <la...@debian.org>
---
README | 3 +
debian/control | 8 ++
debian/copyright | 6 ++
scripts/Makefile | 2 +-
scripts/reproducible-check | 188 +++++++++++++++++++++++++++++++++++++++++++
scripts/reproducible-check.1 | 22 +++++
scripts/setup.py | 2 +-
7 files changed, 229 insertions(+), 2 deletions(-)
create mode 100755 scripts/reproducible-check
create mode 100644 scripts/reproducible-check.1
diff --git a/README b/README
index ba8b265b..e54c9eab 100644
--- a/README
+++ b/README
@@ -226,6 +226,9 @@ And now, in mostly alphabetical order, the scripts:
- rc-alert: list installed packages which have release-critical bugs
[wget | curl]
+- repoducible-check: reports on the reproducibility status of installed
+ packages. [python3-apt, python3-requests, python3-xdg]
+
- rmadison: remotely query the Debian archive database about packages
[liburi-perl, wget | curl]
diff --git a/debian/control b/debian/control
index 82165b80..6b308d19 100644
--- a/debian/control
+++ b/debian/control
@@ -29,11 +29,14 @@ Build-Depends: bash-completion,
po4a (>= 0.40),
pylint,
python3-all,
+ python3-apt,
python3-debian (>= 0.1.15),
python3-flake8,
python3-magic,
+ python3-requests,
python3-setuptools,
python3-unidiff <!nocheck>,
+ python3-xdg,
shunit2 (>= 2.1.6),
unzip,
wdiff,
@@ -72,9 +75,12 @@ Recommends: apt,
man-db,
patch,
patchutils,
+ python3-apt,
python3-debian (>= 0.1.15),
python3-magic,
+ python3-requests,
python3-unidiff,
+ python3-xdg,
sensible-utils,
strace,
unzip,
@@ -215,6 +221,8 @@ Description: scripts to make the life of a Debian Package maintainer easier
[libtimedate-perl, gnuplot]
- pts-subscribe: subscribe to the PTS for a limited period of time
[bsd-mailx | mailx, at]
+ - reproducible-check: reports on the reproducible status of installed
+ packages [python3-apt, python3-requests, python3-xdg]
- rc-alert: list installed packages which have release-critical bugs
[wget | curl]
- rmadison: remotely query the Debian archive database about packages
diff --git a/debian/copyright b/debian/copyright
index c22addad..95550dc7 100644
--- a/debian/copyright
+++ b/debian/copyright
@@ -68,6 +68,12 @@ Copyright: 2009, Jonathan Patrick Davies <j...@ubuntu.com>
2006-2008, Kees Cook <k...@ubuntu.com>
2007-2008, Siegfried-Angel Gevatter Pujals <rai...@ubuntu.com>
2013, Rafael Laboissiere <raf...@laboissiere.net>
+License: GPL-3+
+
+Files: scripts/reproducible-check
+Copyright: © 2017 Chris Lamb <la...@debian.org>
+License: GPL-3+
+
License: GPL-3+
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
diff --git a/scripts/Makefile b/scripts/Makefile
index ece5455a..47206edf 100644
--- a/scripts/Makefile
+++ b/scripts/Makefile
@@ -24,7 +24,7 @@ COMPLETION = $(patsubst %.bash_completion,$(BC_BUILD_DIR)/%,$(COMPL_FILES))
COMPL_DIR := $(shell pkg-config --variable=completionsdir bash-completion)
PKGNAMES:=wnpp-alert wnpp-check mk-build-deps rmadison mass-bug debsnap dd-list build-rdeps who-uploads transition-check getbuildlog dcontrol grep-excuses rc-alert whodepends dget pts-subscribe pts-unsubscribe debcheckout
# also update the list in setup.py
-PYTHON3_SCRIPTS:=debdiff-apply sadt suspicious-source wrap-and-sort
+PYTHON3_SCRIPTS:=debdiff-apply sadt suspicious-source wrap-and-sort reproducible-check
GEN_MAN1S += debrepro.1 devscripts.1 mk-origtargz.1 uscan.1
diff --git a/scripts/reproducible-check b/scripts/reproducible-check
new file mode 100755
index 00000000..d99d041f
--- /dev/null
+++ b/scripts/reproducible-check
@@ -0,0 +1,188 @@
+#!/usr/bin/env python3
+#
+# Copyright (C) 2017 Chris Lamb <la...@debian.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or (at
+# your option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import bz2
+import apt
+import sys
+import json
+import time
+import logging
+import requests
+import argparse
+import collections
+
+from xdg.BaseDirectory import xdg_cache_home
+
+
+class ReproducibleCheck(object):
+ HELP = """
+ Reports on the reproducible status of installed packages.
+ For more details please see <https://reproducible-builds.org>.
+ """
+
+ NAME = os.path.basename(__file__)
+ VERSION = 1
+ DATA_URL = 'https://tests.reproducible-builds.org/debian/reproducible.json.bz2'
+ CACHE_FILENAME = os.path.join(xdg_cache_home, NAME, 'reproducible.json.bz2')
+ CACHE_DURATION_SECS = 86400
+
+ @classmethod
+ def parse(cls):
+ parser = argparse.ArgumentParser(description=cls.HELP)
+
+ parser.add_argument(
+ '-d',
+ '--debug',
+ help="show debugging messages",
+ default=False,
+ action='store_true',
+ )
+
+ parser.add_argument(
+ '-r',
+ '--raw',
+ help="print unreproducible binary packages only (for dd-list -i)",
+ default=False,
+ action='store_true',
+ )
+
+ parser.add_argument(
+ '--version',
+ help="print version and exit",
+ default=False,
+ action='store_true',
+ )
+
+ return cls(parser.parse_args())
+
+ def __init__(self, args):
+ self.args = args
+
+ logging.basicConfig(
+ format='%(levelname).1s: %(message)s',
+ level=logging.DEBUG if args.debug else logging.INFO,
+ )
+
+ self.log = logging.getLogger()
+
+ def main(self):
+ if self.args.version:
+ print("{} version {}".format(self.NAME, self.VERSION))
+ return 0
+
+ if self.should_update_cache():
+ self.update_cache()
+
+ data = self.get_data()
+ installed = self.get_installed_packages()
+ unreproducible = {x: y for x, y in installed.items() if x in data}
+
+ if self.args.raw:
+ self.output_raw(unreproducible, installed)
+ else:
+ self.output_by_source(unreproducible, installed)
+
+ return 0
+
+ def should_update_cache(self):
+ self.log.debug("Checking cache file %s ...", self.CACHE_FILENAME)
+
+ if not os.path.exists(self.CACHE_FILENAME):
+ self.log.debug("Cache file does not exist")
+ return True
+
+ last_update = os.path.getmtime(self.CACHE_FILENAME)
+ if time.time() - last_update >= self.CACHE_DURATION_SECS:
+ self.log.debug("Cache file is stale")
+ return True
+
+ return False
+
+ def update_cache(self):
+ response = requests.get(self.DATA_URL, headers={
+ 'User-Agent': '{}/{} ({})'.format(
+ self.NAME,
+ self.VERSION,
+ requests.utils.default_user_agent(),
+ ),
+ })
+
+ os.makedirs(os.path.dirname(self.CACHE_FILENAME), exist_ok=True)
+
+ with open(self.CACHE_FILENAME, 'wb+') as f:
+ f.write(response.content)
+
+ def get_data(self):
+ self.log.debug("Loading data from cache %s", self.CACHE_FILENAME)
+
+ with bz2.open(self.CACHE_FILENAME) as f:
+ return {
+ (x['package'], x['architecture'], x['version'])
+ for x in json.loads(f.read().decode('utf-8'))
+ if x['status'] == 'unreproducible'
+ }
+
+ def get_installed_packages(self):
+ result = collections.defaultdict(list)
+
+ for x in apt.Cache():
+ for y in x.versions:
+ if not y.is_installed:
+ continue
+
+ key = (y.source_name, y.architecture, y.version)
+ result[key].append(x.shortname)
+
+ return result
+
+ def output_by_source(self, unreproducible, installed):
+ num_installed = sum(len(x) for x in installed.keys())
+ num_unreproducible = sum(len(x) for x in unreproducible.keys())
+
+ for key, binaries in sorted(unreproducible.items()):
+ source, _, version = key
+
+ binaries_fmt = '({}) '.format(', '.join(binaries)) \
+ if binaries != [source] else ''
+
+ print("{} ({}) is unreproducible {}".format(
+ source,
+ version,
+ binaries_fmt,
+ ), end='')
+
+ print("<https://tests.reproducible-builds.org/debian/{}>".format(
+ source,
+ ))
+
+ print("{}/{} ({:.2f}%) of installed binary packages are unreproducible.".format(
+ num_unreproducible,
+ num_installed,
+ 100. * num_unreproducible / num_installed,
+ ))
+
+ def output_raw(self, unreproducible, installed):
+ for x in sorted(x for xs in unreproducible.values() for x in xs):
+ print(x)
+
+
+if __name__ == '__main__':
+ try:
+ sys.exit(ReproducibleCheck.parse().main())
+ except (KeyboardInterrupt, BrokenPipeError):
+ sys.exit(1)
diff --git a/scripts/reproducible-check.1 b/scripts/reproducible-check.1
new file mode 100644
index 00000000..db76d680
--- /dev/null
+++ b/scripts/reproducible-check.1
@@ -0,0 +1,22 @@
+.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.47.4.
+.TH REPRODUCIBLE-CHECK "1" "August 2017" "reproducible-check" "User Commands"
+.SH NAME
+reproducible-check \- report on the reproducible status of packages
+.SH DESCRIPTION
+usage: reproducible\-check [\-h] [\-d] [\-r] [\-\-version]
+.PP
+Reports on the reproducible status of installed packages. For more details
+please see <https://reproducible\-builds.org>.
+.SS "optional arguments:"
+.TP
+\fB\-h\fR, \fB\-\-help\fR
+show this help message and exit
+.TP
+\fB\-d\fR, \fB\-\-debug\fR
+show debugging messages
+.TP
+\fB\-r\fR, \fB\-\-raw\fR
+print unreproducible binary packages only (for dd\-list \fB\-i\fR)
+.TP
+\fB\-\-version\fR
+print version and exit
diff --git a/scripts/setup.py b/scripts/setup.py
index 69884a5c..091e4d32 100755
--- a/scripts/setup.py
+++ b/scripts/setup.py
@@ -12,7 +12,7 @@ if os.path.exists(changelog):
if match:
version = match.group(1)
-scripts = "debdiff-apply sadt suspicious-source wrap-and-sort".split()
+scripts = "debdiff-apply sadt suspicious-source wrap-and-sort reproducible-check".split()
if __name__ == '__main__':
setup(
--
2.14.1
