-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Package: qa.debian.org Severity: important
Hello, I noticed the list of orphaned packages with wrong maintainers is outdated [1]. jwilk was kindly enough to tell me, this is because the LDAP server used to gather bug data is not accessible (anymore?). Output of the script right now is: $ /srv/qa.debian.org/data/wnpp/gen-orphaned Cannot connect to LDAP server For the records: the script tries to connect to bts2ldap.debian.net [2] which seems not to respond anymore. Therefore he suggested to switch to SOAP (or use UDD) instead. This is what I did and I hence reimplemented te gen-orphan script [3] to access the public SOAP interface. It is still loosely based on Michael's original script but it is essentially a complete rewrite, therefore the diff is not particularly useful. For the sake of completeness I attached it nonetheless. However I attached the full source as well which may be better used instead. Note I wasn't able to test the script on the destination site, but on my local system only. Let me know if it does not work. I used syntax and format of [4] as maintainer file, but I had no chance to test whether the resulting output file parses as WML input. Finally please note, the WNPP "library" the script used before is in use for several other scripts as well, e.g. wnpp-lint. Those may be still broken. [1] http://qa.debian.org/orphaned.html [2] http://anonscm.debian.org/viewvc/qa/trunk/data/wnpp/wnpp.py?view=markup [3] http://anonscm.debian.org/viewvc/qa/trunk/data/wnpp/gen-orphaned?view=markup [4] ftp://ftp.debian.org/debian/indices/Maintainers - -- with kind regards, Arno Töll IRC: daemonkeeper on Freenode/OFTC GnuPG Key-ID: 0x9D80F36D -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJOC0iOAAoJEMcrUe6dgPNtpPUP/ibhaYZs5s4x4jJwMpuI9e9S AzibOFkVZ8AZybjBs6CYtFsJSioRFlGuv2Orr/Ept89N6zVa/v3ADvPXWaxaDau1 HLygjlVTA+nBLijQJ3To4hSgXlbu9Fb1tNR37/QmJ9EtbsNoeB9CQ9P7Zk0rmaze BEHbB+l365etevXmq3FH7O+I2GFKKRiSnF7edrERBwJMDJy0XbAaFGbm+6MNEkXO Ce/1KLAVbN1hc3wtPawZpZ1cb1Jq1lodvJoFsLIdkiqq2zqGxbx8ykg72C6dTzGG lvohJ0BzGj74WuC1sxGa9Kg5Syumy1y/LJAWNvA7vVqsef9pUFb2AELrOR8ub0cg RKmQ5yMbW7ki1dS9Kddo6N5ZWEiu0T9+5gqg/hpGA2+2P7F1H8L006k9jJdT7T0S 5KQmkOYgP2cyK3hAYcwdQLool0J+P5D7GWe9RB+2hEZhTCEPltTd+mJNu2gobm4q zV8/lidob68LkrHTuUPhPQgt4BtmlNydsrVBGwPKCIiXsuCpiBIaBAnUggzsCWkM H7muryfup3zlRE9/1hPKOOCbsh5MqP2ohde1Ed3BSgRRf71wbE1hB+EZz0P9fwrb pM42kG8t5DP+xb3hW9qmuu4CP8ChyN+uc1jOjciZ6AycDE7V2STdOM4+CBiBRS63 UoHFaVNoCWAlH/cGlTRK =GD1E -----END PGP SIGNATURE-----
--- wnpp/gen-orphaned 2011-06-29 17:26:23.945892014 +0200 +++ gen-orphaned/src/gen-orphaned.py 2011-06-29 17:27:56.782352372 +0200 @@ -1,9 +1,22 @@ #!/usr/bin/env python +# -*- coding: utf-8 -*- -# Generate the orphan.txt file used by orphaned.wml -# Copyright (C) 2001, 2002, 2003, 2004 Martin Michlmayr <t...@cyrius.com> -# $Id: gen-orphaned 761 2004-04-21 13:46:12Z tbm $ - +""" + Generates the orphan.txt file used by orphaned.wml. + This version is basically a complete rewrite from the previous version + and was changed to use the SOAP interface to query the BTS instead of the LDAP + bridge. + It requires the following Debian packages: + + * Python 2.6 + * python-soappy + + Please change paths configured below if necessary +""" + +# Copyright (C) 2001, 2002, 2003, 2004 Martin Michlmayr <t...@cyrius.com> +# 2011 Arno Töll <deb...@toell.net> +# # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or @@ -11,102 +24,127 @@ # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, +# MA 02110-1301, USA. -import string +from __future__ import print_function + import sys import re -import wnpp +import string +import SOAPpy +import time + + +MAINTAINER_FILE = '/org/qa.debian.org/data/ftp/Maintainers' +OUTPUT_FILE = '/org/qa.debian.org/data/orphaned.txt' +URL = 'http://bugs.debian.org/cgi-bin/soap.cgi' +NAMESPACE = 'Debbugs/SOAP' + +### +### Nothing else should be changed below +### + +class OrphanedPackage( object ): + def __repr__( self ): + return( "%s\n%s\n%d\n%s\n%s\n%d\n\n" + % ( self._package, self._maintainer, self._bugid, self._title, self._reporter, self._age ) ) + + def __init__( self, package, maintainer, bugid, title, reporter, age ): + self._package = package + self._maintainer = maintainer + self._bugid = bugid + self._title = title + self._reporter = reporter + self._age = age try: - input = open('/org/qa.debian.org/data/ftp/Maintainers', 'r') -except (IOError, OSError): - print 'Maintainers file not found' - sys.exit(1) + input = open( MAINTAINER_FILE, 'r' ) +except ( IOError, OSError ) as e: + print( 'Couldn\'t access maintainer file `%s\': %s' % ( MAINTAINER_FILE, e ) ) + sys.exit( 1 ) + maintainers = { } +orphaned = [] + for line in input.readlines(): - # Python 1.5 ignores maxsplit in re.split() so this won't work: - ## package, maintainer = re.split('\s+', line, 2) - # Thus using the following solution: - line = string.rstrip(re.sub('\s+', ' ', line)) - package, maintainer = string.split(line, ' ', 1) - maintainers[package] = maintainer -input.close() + line = string.rstrip( re.sub( '\s+', ' ', line ) ) + package, maintainer = string.split( line, ' ', 1 ) + maintainers[package] = maintainer +input.close() -ldap_result = wnpp.query([ 'debbugsID', 'debbugsTitle', 'debbugsState', 'debbugsDate', 'debbugsSubmitter' ]) +try: + server = SOAPpy.SOAPProxy( URL, NAMESPACE, simplify_objects = 1 ) + raw_bugs = server.get_status( server.get_bugs( "package", "wnpp" ) ) +except ( Exception ) as e: + print( 'Can\'t access SOAP interface on %s (ns: %s): %s' % ( URL, NAMESPACE, e ) ) + sys.exit( 1 ) + +if ( not raw_bugs or 'item' not in raw_bugs ): + print( "Could not retrieve any bugs (but no apparent error occurred)\n" ) + sys.exit( 0 ) + +# Force argument to be a list, SOAPpy returns a dictionary instead of a dictionary list +# if a single bug was found. Not this were likely after all :> +if ( not isinstance( raw_bugs['item'], list ) ): + raw_bugs['item'] = ( raw_bugs['item'], ) + +for bugs in raw_bugs['item']: + current_bug = bugs['value'] + + # Skip packages with illegal subject line + parsed_subject = re.match( "(\w+):\s+(\S+)", current_bug['subject'] ) + if ( not parsed_subject ): + continue + + # Skip Non-O bugs + #if ( parsed_subject.group( 1 ) not in ( "O", "RFA" ) ): + if ( parsed_subject.group( 1 ) != "O" ): + continue + + package = parsed_subject.group( 2 ).lower() + + if ( not len( package ) ): + continue + + # Skip packages where the maintainer is not listed in the maintainers file + if ( package not in maintainers ): + continue + + # Skip packages already owned by the QA team + if ( string.count( maintainers[package], 'debian...@lists.debian.org' ) or + string.count( maintainers[package], 'packa...@qa.debian.org' ) ): + continue + + orphaned.append( OrphanedPackage( package = package, + maintainer = maintainers[package], + bugid = current_bug['id'], + title = current_bug['subject'], + reporter = current_bug['originator'], + age = int( time.time() - current_bug['date'] ) / 86400 + ) ) -if not ldap_result: - print 'Problem with LDAP search' - sys.exit(1) -else: - orphaned = { } - ages = { } - for _, dict in ldap_result: - if not dict: - continue - - if dict['debbugsState'][0] == 'done': - continue - - bugid = int(dict['debbugsID'][0]) - subject = dict['debbugsTitle'][0] - date = dict['debbugsDate'][0] - originater = dict['debbugsSubmitter'][0] - - result = wnpp.weak_subject.search(subject) - if result: - tag = result.group(1) - package = string.lower(result.group(2)) - else: - continue - if tag != 'O': - continue - # Package not in Maintainers file. Package name has a typo or - # the package was removed. Ignore for now and let wnpp-handle it. - if not maintainers.has_key(package): - continue - - if (string.count(maintainers[package], 'debian-qa@lists.') or - string.count(maintainers[package], 'packages@qa.')): - continue - - age = wnpp.date2days(date) - if orphaned.has_key(package): - continue - orphaned[package] = [ maintainers[package], str(bugid), subject, originater ] - - try: - ages[age].append(package) - except KeyError: - ages[age] = [ package ] - -lages = ages.keys() -lages.sort() -lages.reverse() try: - output = open('/org/qa.debian.org/data/orphaned.txt', 'w') -except (IOError, OSError): - print 'Cannot write output file' - sys.exit(1) - -output.write(str(len(orphaned)) + '\n\n') -for age in lages: - ages[age].sort() - for package in ages[age]: - output.write(package + '\n') - for i in range(4): - output.write(orphaned[package][i] + '\n') - output.write(str(age) + '\n\n') -output.close() + output = open( OUTPUT_FILE, 'w' ) +except ( IOError, OSError ) as e: + print( 'Output file not accessible `%s\': %s' % ( MAINTAINER_FILE, e ) ) + sys.exit( 1 ) + + +output.write( str( len( orphaned ) ) + "\n\n" ) +for orphaned_package in sorted( orphaned, key = lambda x: x._age ): # sort by age: + output.write( str( orphaned_package ) ) + +print( "%s: %d entries written\n" % ( OUTPUT_FILE, len( orphaned ) ) ) # vim: ts=4:expandtab:shiftwidth=4:
#!/usr/bin/env python # -*- coding: utf-8 -*- """ Generates the orphan.txt file used by orphaned.wml. This version is basically a complete rewrite from the previous version and was changed to use the SOAP interface to query the BTS instead of the LDAP bridge. It requires the following Debian packages: * Python 2.6 * python-soappy Please change paths configured below if necessary """ # Copyright (C) 2001, 2002, 2003, 2004 Martin Michlmayr <t...@cyrius.com> # 2011 Arno Töll <deb...@toell.net> # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, # MA 02110-1301, USA. from __future__ import print_function import sys import re import string import SOAPpy import time MAINTAINER_FILE = '/org/qa.debian.org/data/ftp/Maintainers' OUTPUT_FILE = '/org/qa.debian.org/data/orphaned.txt' URL = 'http://bugs.debian.org/cgi-bin/soap.cgi' NAMESPACE = 'Debbugs/SOAP' ### ### Nothing else should be changed below ### class OrphanedPackage( object ): def __repr__( self ): return( "%s\n%s\n%d\n%s\n%s\n%d\n\n" % ( self._package, self._maintainer, self._bugid, self._title, self._reporter, self._age ) ) def __init__( self, package, maintainer, bugid, title, reporter, age ): self._package = package self._maintainer = maintainer self._bugid = bugid self._title = title self._reporter = reporter self._age = age try: input = open( MAINTAINER_FILE, 'r' ) except ( IOError, OSError ) as e: print( 'Couldn\'t access maintainer file `%s\': %s' % ( MAINTAINER_FILE, e ) ) sys.exit( 1 ) maintainers = { } orphaned = [] for line in input.readlines(): line = string.rstrip( re.sub( '\s+', ' ', line ) ) package, maintainer = string.split( line, ' ', 1 ) maintainers[package] = maintainer input.close() try: server = SOAPpy.SOAPProxy( URL, NAMESPACE, simplify_objects = 1 ) raw_bugs = server.get_status( server.get_bugs( "package", "wnpp" ) ) except ( Exception ) as e: print( 'Can\'t access SOAP interface on %s (ns: %s): %s' % ( URL, NAMESPACE, e ) ) sys.exit( 1 ) if ( not raw_bugs or 'item' not in raw_bugs ): print( "Could not retrieve any bugs (but no apparent error occurred)\n" ) sys.exit( 0 ) # Force argument to be a list, SOAPpy returns a dictionary instead of a dictionary list # if a single bug was found. Not this were likely after all :> if ( not isinstance( raw_bugs['item'], list ) ): raw_bugs['item'] = ( raw_bugs['item'], ) for bugs in raw_bugs['item']: current_bug = bugs['value'] # Skip packages with illegal subject line parsed_subject = re.match( "(\w+):\s+(\S+)", current_bug['subject'] ) if ( not parsed_subject ): continue # Skip Non-O bugs #if ( parsed_subject.group( 1 ) not in ( "O", "RFA" ) ): if ( parsed_subject.group( 1 ) != "O" ): continue package = parsed_subject.group( 2 ).lower() if ( not len( package ) ): continue # Skip packages where the maintainer is not listed in the maintainers file if ( package not in maintainers ): continue # Skip packages already owned by the QA team if ( string.count( maintainers[package], 'debian...@lists.debian.org' ) or string.count( maintainers[package], 'packa...@qa.debian.org' ) ): continue orphaned.append( OrphanedPackage( package = package, maintainer = maintainers[package], bugid = current_bug['id'], title = current_bug['subject'], reporter = current_bug['originator'], age = int( time.time() - current_bug['date'] ) / 86400 ) ) try: output = open( OUTPUT_FILE, 'w' ) except ( IOError, OSError ) as e: print( 'Output file not accessible `%s\': %s' % ( MAINTAINER_FILE, e ) ) sys.exit( 1 ) output.write( str( len( orphaned ) ) + "\n\n" ) for orphaned_package in sorted( orphaned, key = lambda x: x._age ): # sort by age: output.write( str( orphaned_package ) ) print( "%s: %d entries written\n" % ( OUTPUT_FILE, len( orphaned ) ) ) # vim: ts=4:expandtab:shiftwidth=4:
