Package: ftpmaster.debian.org
Severity: wishlist
Tags: patch
Hi,
Attached is a patch submit .buildinfo files to buildinfo.debian.net,
our experimental system for centrally storing .buildinfo files for
analysis, retrieval, etc. We almost have 2,000,000 files there.
This patch supplements the existing filesystem archiving and simply
performs a POST on the .buildinfo file itself.
As a deployment note, this will actually mean that — right now — most
.buildinfo files will be rejected by buildinfo.debian.net as it only
accepts signed .buildinfo files. However, we intend to fix that
separately via #862059 ("sbuild: please sign buildinfo files").
Also note that this patch enables this for the main archive only.
Please clarify whether I should enable this for the security archive
too; I would not want it to leak the fact we have rebuilt a package
there if itwere embargoed, etc.
commit a85df018d210c054e7ae0b5a6fe037a537b62e7a
Author: Chris Lamb <la...@debian.org>
Date: Mon May 8 01:06:03 2017 +0200
Upload buildinfo files to buildinfo.debian.net.
Signed-off-by: Chris Lamb <la...@debian.org>
config/debian/dak.conf | 5 +++++
dak/process_upload.py | 52
++++++++++++++++++++++++++++++++++++++++++++++++--
2 files changed, 55 insertions(+), 2 deletions(-)
Alternatively you can merge from the:
upload-buildinfo-files-to-buildinfo-debian-net
branch on <https://github.com/lamby/dak.git>.
Regards,
--
,''`.
: :' : Chris Lamb
`. `'` la...@debian.org / chris-lamb.co.uk
`-
>From a85df018d210c054e7ae0b5a6fe037a537b62e7a Mon Sep 17 00:00:00 2001
From: Chris Lamb <la...@debian.org>
Date: Mon, 8 May 2017 01:06:03 +0200
Subject: [PATCH] Upload buildinfo files to buildinfo.debian.net.
Signed-off-by: Chris Lamb <la...@debian.org>
---
config/debian/dak.conf | 5 +++++
dak/process_upload.py | 52 ++++++++++++++++++++++++++++++++++++++++++++++++--
2 files changed, 55 insertions(+), 2 deletions(-)
diff --git a/config/debian/dak.conf b/config/debian/dak.conf
index 4fa828e56..ad71e4ded 100644
--- a/config/debian/dak.conf
+++ b/config/debian/dak.conf
@@ -119,6 +119,11 @@ Process-New
LockDir "/srv/ftp-master.debian.org/lock/new/";
};
+BuildinfoService
+{
+ "https://buildinfo.debian.net/api/submit";;
+}
+
SuiteMappings
{
"silent-map squeeze-security oldoldstable-security";
diff --git a/dak/process_upload.py b/dak/process_upload.py
index 51104523f..98fdde4d9 100755
--- a/dak/process_upload.py
+++ b/dak/process_upload.py
@@ -168,6 +168,7 @@ import sys
import traceback
import apt_pkg
import time
+import urllib2
from sqlalchemy.orm.exc import NoResultFound
from daklib import daklog
@@ -263,7 +264,8 @@ def accept(directory, upload):
print "ACCEPT"
upload.install()
- process_buildinfos(upload)
+ archive_buildinfos(upload)
+ upload_buildinfos(upload)
accepted_to_real_suite = False
for suite in upload.final_suites:
@@ -494,7 +496,12 @@ def process_changes(changes_filenames):
for directory, c in changes:
process_it(directory, c, keyring_files)
-def process_buildinfos(upload):
+def archive_buildinfos(upload):
+ """
+ Archive .buildinfo files locally on the filesystem at
+ ``Dir::BuildinfoArchive``.
+ """
+
cnf = Config()
if not cnf.has_key('Dir::BuildinfoArchive'):
@@ -515,6 +522,47 @@ def process_buildinfos(upload):
Logger.log(["Archiving", x.filename])
upload.transaction.fs.copy(src, dst, mode=0o644)
+def upload_buildinfos(upload):
+ """
+ Upload .buildinfo files to the service(s) specified in
+ ``BuildinfoService``. The data is sent as raw HTTP POST data.
+ """
+
+ cnf = Config()
+
+ try:
+ urls = cnf.value_list('BuildinfoService')
+ except KeyError:
+ return
+
+ for x in upload.changes.files.itervalues():
+ if not re_file_buildinfo.match(x.filename):
+ continue
+
+ with open(os.path.join(upload.directory, x.filename)) as f:
+ data = f.read()
+
+ opener = urllib2.build_opener(urllib2.HTTPHandler)
+
+ for url in urls:
+ Logger.log(["Uploading", x.filename, "to", url])
+
+ request = urllib2.Request(url, data, {
+ 'Content-Type': 'application/octet-stream',
+ })
+ request.get_method = lambda: 'PUT'
+
+ try:
+ res = urllib2.urlopen(request, timeout=10)
+ except urllib2.HTTPError as e:
+ Logger.log(['HTTP {}: {}'.format(e.code, e.read().strip())])
+ continue
+ except Exception:
+ Logger.log(['Exception:', traceback.format_exc()])
+ continue
+
+ Logger.log(['HTTP {}: {}'.format(res.code, res.read().strip())])
+
###############################################################################
def main():
--
2.11.0
