Hi, My question is, should we bump version for watch file from 3 to 4, or not if we support multiple upstream tarballs?
Support of multiple upstream source seems to require 3 key changes and other minor ones. * uscan already support multiple data line * uscan supporting component name to avoid file colliding * mk-origtargz supporting suffix without repackaging * uupdate2 supporting multiple upstream source The last one, I have posted proof of concept code on https://bugs.debian.org/797045 For uscan and mk-origtargz scripts, I have working code locally. I rebased it to the GIT head. Just to give you idea, I attach it here but do not guarantee they work as is. They are by no means ready for inclusion, yet. Before finalizing these patch, I am thinking of fixing other irregularities, first. done #796984 uscan: watch file [Version] field handles 0 as debian in progress #796986 repacksuffix does not adjust version passed to uupdate (This is patch 1 so this needs to go in first) thinking for pagemangle comment on #773390 uscan: allow to specify an alternative to "href" attribute This is very powerful feature. I think we can also search pgp signature file like tarball and download it for checking or create option to download hush value from web page. Regards, Osamu
>From d84c2891b3ad47419b42b742f9b1d56e31eb13cf Mon Sep 17 00:00:00 2001 From: Osamu Aoki <[email protected]> Date: Wed, 26 Aug 2015 09:03:17 +0000 Subject: [PATCH 2/3] mk-origtargz.pl --- scripts/mk-origtargz.pl | 48 +++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 39 insertions(+), 9 deletions(-) diff --git a/scripts/mk-origtargz.pl b/scripts/mk-origtargz.pl index ef2c75e..8602f46 100755 --- a/scripts/mk-origtargz.pl +++ b/scripts/mk-origtargz.pl @@ -41,7 +41,8 @@ mk-origtargz - rename upstream tarball, optionally changing the compression and B<mk-origtargz> renames the given file to match what is expected by B<dpkg-buildpackage>, based on the source package name and version in F<debian/changelog>. It can convert B<zip> to B<tar>, optionally change the -compression scheme and remove files according to B<Files-Excluded> in +compression scheme and remove files according to B<Files-Excluded> +and B<Files-Excluded->I<component> in F<debian/copyright>. The resulting file is placed in F<debian/../..>. If the package name is given via the B<--package> option, no information is @@ -121,10 +122,22 @@ If the file has to be modified (because it is a B<zip> file, because of B<--repa If the given file is not compressed using the desired format (see B<--compression>), recompress it. +=item B<-s>, B<--suffix> I<suffix> + +Always append I<suffix> to the upstream version. (Useful for multiple upstream tarballs under +version=4. + =item B<-S>, B<--repack-suffix> I<suffix> If the file has to be modified, because of B<Files-Excluded>, append I<suffix> to the upstream version. +=item B<-c>, B<--component> I<componentname> + +Use <componentname> as the component name for the secondary upstream tarball. +Set I<componentname> as the component namei. This is used only for the +secondary upstream tarball of the Debian source package. +Then I<packagename_version.orig-componentiname.tar.gz> is created. + =item B<--compression> [ B<gzip> | B<bzip2> | B<lzma> | B<xz> ] If B<--repack> is used, or if the given file is a B<zip> file, ensure that the resulting file is compressed using the given scheme. The default is B<gzip>. @@ -179,6 +192,9 @@ sub compress_archive($$$); my $package = undef; my $version = undef; +my $component = undef; +my $orig="orig"; +my $excludestanza="Files-Excluded"; my @exclude_globs = (); my @copyright_files = (); @@ -186,7 +202,8 @@ my $destdir = undef; my $compression = "gzip"; my $mode = undef; # can be symlink, rename or copy. Can internally be repacked if the file was repacked. my $repack = 0; -my $suffix = ''; +my $suffix = undef; +my $repacksuffix = ''; my $upstream = undef; @@ -207,6 +224,7 @@ sub setmode { GetOptions( "package=s" => \$package, "version|v=s" => \$version, + "component|c=s" => \$component, "exclude-file=s" => \@exclude_globs, "copyright-file=s" => \@copyright_files, "compression=s" => \$compression, @@ -214,13 +232,20 @@ GetOptions( "rename" => \&setmode, "copy" => \&setmode, "repack" => \$repack, - 'repack-suffix|S=s' => \$suffix, + 'suffix|s=s' => \$suffix, + 'repack-suffix|S=s' => \$repacksuffix, "directory|C=s" => \$destdir, "help|h" => sub { pod2usage({-exitval => 0, -verbose => 1}); }, ) or pod2usage({-exitval => 3, -verbose=>1}); $mode ||= "symlink"; +if (defined $suffix) { + $repacksuffix = $suffix; +} else { + $suffix = ""; +} + # sanity checks unless (compression_is_supported($compression)) { die_opts (sprintf "Unknown compression scheme %s", $compression); @@ -230,6 +255,11 @@ if (defined $package and not defined $version) { die_opts "If you use --package, you also have to specify --version." } +if (defined $component) { + $orig="orig-$component"; + $excludestanza="Files-Excluded-$component"; +} + if (@ARGV != 1) { die_opts "Please specify original tarball." } @@ -287,14 +317,14 @@ for my $copyright_file (@copyright_files) { && defined $data->{format} && $data->{format} =~ m@^$okformat/?$@) { - if ($data->{'files-excluded'}) { - push(@exclude_globs, grep { $_ } split(/\s+/, $data->{'files-excluded'})); + if ($data->{$excludestanza}) { + push(@exclude_globs, grep { $_ } split(/\s+/, $data->{$excludestanza})); } } else { open my $file, '<', $copyright_file or die "Unable to read $copyright_file: $!\n"; while (my $line = <$file>) { - if ($line =~ m/\bFiles-Excluded:/i) { - warn "WARNING: The file $copyright_file mentions Files-Excluded, but its ". + if ($line =~ m/\b${excludestanza}.*:/i) { + warn "WARNING: The file $copyright_file mentions $excludestanza, but its ". "format is not recognized. Specify Format: ". "https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ ". "in order to remove files from the tarball with mk-origtargz.\n"; @@ -343,7 +373,7 @@ if ($is_tarfile and not $repack) { # Now we know what the final filename will be -my $destfilebase = sprintf "%s_%s.orig.tar", $package, $version; +my $destfilebase = sprintf "%s_%s%s.%s.tar", $package, $version, $suffix, $orig; my $destfiletar = sprintf "%s/%s", $destdir, $destfilebase; my $destext = compression_get_property($compression, "file_ext"); my $destfile = sprintf "%s.%s", $destfiletar, $destext; @@ -455,7 +485,7 @@ if (@exclude_globs) { } if ($deletecount) { - $destfilebase = sprintf "%s_%s%s.orig.tar", $package, $version, $suffix; + $destfilebase = sprintf "%s_%s%s.%s.tar", $package, $version, $repacksuffix, $orig; $destfiletar = sprintf "%s/%s", $destdir, $destfilebase; $destfile = sprintf "%s.%s", $destfiletar, $destext; -- 2.1.4
>From 6b9928f8c70b38d8c46a0a2a30f51bf39986e56f Mon Sep 17 00:00:00 2001 From: Osamu Aoki <[email protected]> Date: Wed, 26 Aug 2015 09:02:44 +0000 Subject: [PATCH 3/3] uscan.pl --- scripts/uscan.pl | 198 ++++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 152 insertions(+), 46 deletions(-) diff --git a/scripts/uscan.pl b/scripts/uscan.pl index ef21e2c..3e3f499 100755 --- a/scripts/uscan.pl +++ b/scripts/uscan.pl @@ -50,7 +50,7 @@ BEGIN { } use Dpkg::Control::Hash; -my $CURRENT_WATCHFILE_VERSION = 3; +my $CURRENT_WATCHFILE_VERSION = 4; my $progname = basename($0); my $modified_conf_msg; @@ -67,7 +67,11 @@ my $havegpg = first { -x $_ } qw(/usr/bin/gpg2 /usr/bin/gpg); # Did we find any new upstream versions on our wanderings? our $found = 0; -sub process_watchline ($$$$$$); +# For multi tarball case, the upstream version of the main tarball is used +# for the source package as $commonversion (with epoch and suffix) +our $commonversion = undef; + +sub process_watchline ($$$$$$$); sub process_watchfile ($$$$); sub recursive_regex_dir ($$$); sub newest_dir ($$$$$); @@ -729,15 +733,41 @@ exit ($found ? 0 : 1); # opts=downloadurlmangle=s/prdownload/download/ \ # http://developer.berlios.de/project/showfiles.php?group_id=2051 \ # http://prdownload.berlios.de/softdevice/vdr-softdevice-(.+).tgz +# +# watch_version=4: +# +# This version supports the source package with multiple upstream tarballs +# by explicitly supporting multiple URL matching entry lines. The first entry +# line is for the main tarball and the following entry lines are for the +# secondary tarballs. +# +# The option is extended to support the component name of the secondary +# tarballs as: +# opts=component=<componentname> \ +# http://some.site.org/some/path/foobar-(.+)\.tar\.gz +# The downloaded file foobar-<realversion>.tar.gz is linked to +# <sourcepackagename>-<sourcepackageversion>.orig-<componentname>.tar.gz +# +# This version supports "ignore" and "match" in addition for "debian" +# in watchfile for the secondary tarballs. - -sub process_watchline ($$$$$$) +sub process_watchline ($$$$$$$) { - my ($line, $watch_version, $pkg_dir, $pkg, $pkg_version, $watchfile) = @_; + my ($line, $watch_version, $pkg_dir, $pkg, $pkg_version, $lineindex, $watchfile) = @_; + # calling arguments + # $1 = String containing a line of watchfile to be processed + # $2 = Syntactic version of watchfile + # $3 = Directory on the local system to check (CWD, undef, ...) + # $4 = Current source package name set from debian/changelog + # $5 = Current source package version from debian/changelog + # $6 = Line index: 1 for orig.tar.gz ; 2,3,... for orig-componentX.tar.gz + # $7 = Path to watchfile my $origline = $line; my ($base, $site, $dir, $filepattern, $pattern, $lastversion, $action); my $basedir; + my $orig="orig"; + my $matchmode="newer"; my (@patterns, @sites, @redirections, @basedirs); my %options = (); @@ -788,7 +818,7 @@ sub process_watchline ($$$$$$) $site = $1; $pattern = $filepattern; } else { - # version 2/3 watchfile + # version 2/3/4 watchfile if ($line =~ s/^opt(?:ion)?s=//) { my $opts; if ($line =~ s/^"(.*?)"\s+//) { @@ -809,6 +839,9 @@ sub process_watchline ($$$$$$) or $opt eq 'nopassive') { $options{'pasv'}=0; } + elsif ($watch_version>3 and $opt =~ /^suffix\s*=\s*(.+)/) { + $options{'suffix'} = $1; + } elsif ($opt =~ /^repacksuffix\s*=\s*(.+)/) { $options{'repacksuffix'} = $1; } @@ -831,10 +864,16 @@ sub process_watchline ($$$$$$) elsif ($opt =~ /^pgpsigurlmangle\s*=\s*(.+)/) { @{$options{'pgpsigurlmangle'}} = split /;/, $1; } + elsif ($lineindex > 1) { + if ($opt =~ /^component\s*=\s*(.+)/) { + $options{'component'} = $1; + } + } else { uscan_warn "$progname warning: unrecognised option $opt\n"; } } + $options{'repacksuffix'} = undef if defined $options{'suffix'}; } ($base, $filepattern, $lastversion, $action) = split ' ', $line, 4; @@ -847,9 +886,20 @@ sub process_watchline ($$$$$$) (undef, $lastversion, $action) = split ' ', $line, 3; } - if ((! defined $lastversion or $lastversion eq 'debian') and not defined $pkg_version) { - uscan_warn "$progname warning: Unable to determine current version\n in $watchfile, skipping:\n $line\n"; - return 1; + if ($lineindex>1 and defined $lastversion and $lastversion eq 'match') { + $matchmode = "match"; + $lastversion=$commonversion; + } elsif ($lineindex>1 and defined $lastversion and $lastversion eq 'ignore') { + $matchmode = "ignore"; + $lastversion="~0"; + } elsif (! defined $lastversion or $lastversion eq 'debian') { + if (defined $pkg_version) { + $lastversion=$pkg_version; + } else { + uscan_warn "$progname warning: Unable to determine current version\n in $watchfile, skipping:\n $line\n"; + return 1; + } + # $matchmode = "newer" } # Check all's OK @@ -915,14 +965,6 @@ sub process_watchline ($$$$$$) $pattern = "(?:(?:$site)?" . quotemeta($basedir) . ")?$filepattern"; } - if (! defined $lastversion or $lastversion eq 'debian') { - if (defined $pkg_version) { - $lastversion=$pkg_version; - } else { - uscan_warn "$progname warning: Unable to determine current version\n in $watchfile, skipping:\n $line\n"; - return 1; - } - } # And mangle it if requested my $mangled_lastversion; $mangled_lastversion = $lastversion; @@ -1326,14 +1368,34 @@ EOF if ($verbose or ($download == 0 and $report and ! $dehs)) { print $pkg_report_header; $pkg_report_header = ''; - print "Newest version on remote site is $newversion, local version is $lastversion\n" . - ($mangled_lastversion eq $lastversion ? "" : " (mangled local version number $mangled_lastversion)\n"); - print " => Package is up to date\n"; + if ($matchmode eq "newer") { + print "Newest version on remote site is $newversion, local version is $lastversion\n" . + ($mangled_lastversion eq $lastversion ? "" : " (mangled local version number $mangled_lastversion)\n"); + print " => Package is up to date\n"; + } elsif ($matchmode eq "ignore") { + print "Newest version on remote site is $newversion, ignore local version\n"; + } elsif ($matchmode eq "match") { + print "Newest version on remote site sub archive is $newversion, downloaded main archive version is $lastversion\n" . + ($mangled_lastversion eq $lastversion ? "" : " (mangled downloaded version number $mangled_lastversion)\n"); + print " => Package matched\n"; + } else { + uscan_warn "$progname: strange matching mode: $matchmode\n"; + return 1; + } } - $dehs_tags{'status'} = "up to date"; - if (! $force_download) { - return 0; - } else { + + if ($matchmode eq "newer") { + $dehs_tags{'status'} = "up to date"; + if (! $force_download) { + return 0; + } else { + $download = 1; + } + } elsif ($matchmode eq "ignore") { + $dehs_tags{'status'} = "version ignored"; + $download = 1; + } elsif ($matchmode eq "match") { + $dehs_tags{'status'} = "version matched"; $download = 1; } } @@ -1342,20 +1404,27 @@ EOF if ($verbose or ($download == 0 and ! $dehs)) { print $pkg_report_header; $pkg_report_header = ''; - print "Newest version on remote site is $newversion, local version is $lastversion\n" . - ($mangled_lastversion eq $lastversion ? "" : " (mangled local version number $mangled_lastversion)\n"); + if ($matchmode eq "newer") { + print "Newest version on remote site is $newversion, local version is $lastversion\n" . + ($mangled_lastversion eq $lastversion ? "" : " (mangled local version number $mangled_lastversion)\n"); + } elsif ($matchmode eq "ignore") { + print "Newest version on remote site is $newversion, ignore local version\n"; + } elsif ($matchmode eq "match") { + print "Newest version on remote site is $newversion, main archive version is $lastversion\n" . + ($mangled_lastversion eq $lastversion ? "" : " (mangled main archive version number $mangled_lastversion)\n"); + } else { + uscan_warn "$progname: strange matching mode: $matchmode\n"; + return 1; + } } - # We use dpkg's rules to determine whether our current version # is newer or older than the remote version. - if (!defined $download_version) { + if (!defined $download_version and $matchmode eq "newer") { if (system("dpkg", "--compare-versions", "1:${mangled_lastversion}-0", "gt", "1:${newversion}-0") == 0) { - if ($verbose) { - print " => remote site does not even have current version\n"; - } elsif ($dehs) { + if ($dehs) { $dehs_tags{'status'} = "Debian version newer than remote site"; } else { - print "$pkg: remote site does not even have current version\n"; + print " => $pkg: remote site does not even have current version\n"; } return 0; } else { @@ -1363,6 +1432,10 @@ EOF # be on our system or may not be $found++; } + } elsif ($matchmode eq "ignore") { + $found++; + } elsif ($matchmode eq "match") { + $found++; } else { # Flag that we found a newer upstream version, so that the exit status # is set correctly @@ -1379,9 +1452,15 @@ EOF if $verbose or ($download == 0 and ! $dehs); return 0; } + if ($lineindex>1) { + if (not defined $options{'component'}) { + $options{'component'} = "component$lineindex"; + } + $orig = "orig-$options{'component'}"; + } foreach my $suffix (qw(gz bz2 lzma xz)) { - if (-f "$destdir/${pkg}_${newversion}.orig.tar.$suffix") { - print " => ${pkg}_${newversion}.orig.tar.$suffix already in package directory '$destdir'\n" + if (-f "$destdir/${pkg}_${newversion}.${orig}.tar.$suffix") { + print " => ${pkg}_${newversion}.${orig}.tar.$suffix already in package directory '$destdir'\n" if $verbose or ($download == 0 and ! $dehs); return 0; } @@ -1493,6 +1572,9 @@ EOF } } + if ($lineindex==1) { + $commonversion=$newversion; + } # Call mk-origtargz (renames, repacks, etc.) my $mk_origtargz_out; my $path = "$destdir/$newfile_base"; @@ -1500,11 +1582,13 @@ EOF unless ($symlink eq "no") { my @cmd = ("mk-origtargz"); push @cmd, "--package", $pkg; - push @cmd, "--version", $newversion; + push @cmd, "--version", $commonversion; + push @cmd, '--suffix', $options{suffix} if defined $options{suffix}; push @cmd, '--repack-suffix', $options{repacksuffix} if defined $options{repacksuffix}; push @cmd, "--rename" if $symlink eq "rename"; push @cmd, "--copy" if $symlink eq "copy"; push @cmd, "--repack" if $repack; + push @cmd, "--component", $options{'component'} if defined $options{'component'}; push @cmd, "--compression", $repack_compression; push @cmd, "--directory", $destdir; push @cmd, "--copyright-file", "debian/copyright" @@ -1522,7 +1606,9 @@ EOF $target = basename($path); } - if (defined $options{'repacksuffix'}) { + if (defined $options{'suffix'}) { + $newversion = $newversion . $options{'suffix'} + } elsif (defined $options{'repacksuffix'}) { # assume it was repacked (if wasn't, remove opts=repacksuffix in watchfile) $newversion = $newversion . $options{'repacksuffix'} } @@ -1548,15 +1634,27 @@ EOF if ($action) { my @cmd = shellwords($action); - # Any symlink requests are already handled by uscan - if ($action =~ /^uupdate(\s|$)/) { - push @cmd, "--no-symlink"; - } - if ($watch_version > 1) { - push @cmd, "--upstream-version", $newversion, $path; + # script invocation changed in $watch_version=4 + if ($watch_version > 3) { + # invoke uupdate2 which takes version string with suffix + if ($action =~ /^uupdate(\s|$)/) { + uscan_warn "$progname warning: use uupdate2 instead of uupdate in debian/watch for version=4.\n"; + return 1; + } + push @cmd, $newversion; + # components } else { - push @cmd, $path, $newversion; + # Any symlink requests are already handled by uscan + if ($action =~ /^uupdate(\s|$)/) { + push @cmd, "--no-symlink"; + } + + if ($watch_version > 1) { + push @cmd, "--upstream-version", $newversion, $path; + } else { + push @cmd, $path, $newversion; + } } my $actioncmd = join(" ", @cmd); print "-- Executing user specified script\n $actioncmd\n" if $verbose; @@ -1736,6 +1834,7 @@ sub process_watchfile ($$$$) my ($dir, $package, $version, $watchfile) = @_; my $watch_version=0; my $status=0; + my $lineindex=0; %dehs_tags = (); unless (open WATCH, $watchfile) { @@ -1778,16 +1877,23 @@ sub process_watchfile ($$$$) # Are there any warnings from this part to give if we're using dehs? dehs_output if $dehs; + $lineindex+=1; # number of watch entries + if ($watch_version<4 and $lineindex>1) { + # This ensures $lineindex=1 for $watch_version=2 and 3 in process_watchline() + uscan_warn "$progname warning: $watchfile has multiple entries for version=$watch_version (see uscan(1) for details).\n"; + last; + } + # Handle shell \\ -> \ s/\\\\/\\/g if $watch_version==1; if ($verbose) { - print "-- In $watchfile, processing watchfile line:\n $_\n"; + print "-- In $watchfile, processing watchfile line: $lineindex\n $_\n"; } elsif ($download == 0 and ! $dehs) { - $pkg_report_header = "Processing watchfile line for package $package...\n"; + $pkg_report_header = "Processing watchfile line $lineindex for package $package...\n"; } $status += - process_watchline($_, $watch_version, $dir, $package, $version, + process_watchline($_, $watch_version, $dir, $package, $version, $lineindex, $watchfile); dehs_output if $dehs; } -- 2.1.4
_______________________________________________ devscripts-devel mailing list [email protected] http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/devscripts-devel
