Hi Joey,
Joey Hess wrote:
> * In Release, Packages, and Sources files, support all sizes of SHA
> checksums that are supported by Digest::SHA. Closes: #614383
> * Now depends on libdigest-sha-perl.
> * Check SHA512, SHA256, or SHA1 in preference to MD5.
> * Full checksum validation is now enabled by the --checksums switch.
> (The old --md5sums switch is an alias to that for backwards
> compatibility.)
Here are some patches that blindly backport the above changes to
squeeze. Completely untested, and I'd be surprised if it actually
works without tweaking. I don't use debmirror, but maybe this can
save some time for someone on squeeze who does.
Thanks,
Jonathan
From: Joey Hess <j...@kitenet.net>
Date: Mon, 21 Feb 2011 15:15:13 -0400
Subject: prepare for multi-checksum support
commit 59d7ebe779b6881d48f45f7188b09ea165cf6a61 upstream.
Mostly just s/md5sum/$more generic wording/
---
debian/changelog | 10 ++++++
debmirror | 94 ++++++++++++++++++++++++++++-------------------------
2 files changed, 60 insertions(+), 44 deletions(-)
diff --git a/debian/changelog b/debian/changelog
index 3f4150b5..9ec3b8a1 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,13 @@
+debmirror (1:2.4.5+squeeze1) UNRELEASED; urgency=low
+
+ [ Joey Hess ]
+ * Support sha checksums in Release files, and use in preference to md5.
+ Closes: #614383
+ * Full checksum validation is now enabled by the --checksums switch.
+ (The old --md5sums switch is an alias to that for backwards compatability.)
+
+ -- Jonathan Nieder <jrnie...@gmail.com> Thu, 05 Jan 2012 15:06:55 -0600
+
debmirror (1:2.4.5) unstable; urgency=low
* Drop support for the --adddir option which was obsoleted log ago.
diff --git a/debmirror b/debmirror
index 089cdac6..eeb57732 100755
--- a/debmirror
+++ b/debmirror
@@ -41,7 +41,7 @@ removed.
The Packages and Sources files are scanned, to build up a list of all the
files they refer to. A few other miscellaneous files are added to the list.
Then the program makes sure that each file in the list is present on the
-local mirror and is up-to-date, using file size (and optionally md5sum) checks.
+local mirror and is up-to-date, using file size (and optionally checksum)
checks.
Any necessary files are downloaded.
=back
@@ -246,9 +246,9 @@ files can be relatively big and can change frequently,
especially for the
testing and unstable suites. Use of the available diff files is strongly
recommended (see the --diff option).
-=item --md5sums -m
+=item --checksums
-Use md5sums to determine if files on the local mirror that are the correct
+Use checksums to determine if files on the local mirror that are the correct
size actually have the correct content. Not enabled by default, because
it is too paranoid, and too slow.
@@ -355,8 +355,8 @@ files after applying diffs. The default options are "-9 -n
--rsyncable"
which corresponds with the options used to gzip meta files for the main
Debian archive.
-These options may need to be modified if the md5sum of the file as gzipped
-by debmirror does not match the md5sum listed in the Release file (which
+These options may need to be modified if the checksum of the file as gzipped
+by debmirror does not match the checksum listed in the Release file (which
will result in the gzipped file being downloaded unnecessarily after diffs
were successfully applied).
@@ -522,8 +522,7 @@ our (@dists, @sections, @arches, @ignores, @excludes,
@includes);
our (@excludes_deb_section, @limit_priority);
our (@di_dists, @di_arches, @rsync_extra);
our $state_cache_days = 0;
-our $check_md5sums = 0;
-our $check_downloads = 0;
+our $verify_checksums = 0;
our $cleanup=0;
our $post_cleanup=1;
our $no_cleanup=0;
@@ -583,7 +582,7 @@ my %distset=();
# below! Filenames should be relative to $mirrordir.
my %files;
-# Hash to record size and md5sums of meta files and package files (from the
+# Hash to record size and checksums of meta files and package files (from the
# Release file and Source/Packages files).
my %file_lists;
@@ -607,8 +606,8 @@ GetOptions('debug' => \$debug,
'progress|p' => \$progress,
'verbose|v' => \$verbose,
'source!' => \$do_source,
- 'md5sums|m' => \$check_md5sums,
- 'nomd5sums' => \$check_downloads,
+ 'checksums!' => \$verify_checksums,
+ 'md5sums|m' => \$verify_checksums, # back compat
'passive!' => \$passive,
'host|h=s' => \$host,
'user|u=s' => \$user,
@@ -714,7 +713,7 @@ say("Including source.") if $do_source;
say("D-I arches: ".join(",", @di_arches)) if @di_arches;
say("D-I dists: ".join(",", @di_dists)) if @di_dists;
say("Pdiff mode: $diff_mode");
-say("Checking md5sums.") if $check_md5sums;
+say("Veriftying checksums.") if $verify_checksums;
say("Passive mode on.") if $passive;
say("Proxy: $proxy") if $proxy;
say("Download at most $max_batch files.") if ($max_batch > 0);
@@ -917,6 +916,7 @@ foreach my $dist (keys %distset) {
}
# Get and parse MD5SUMS files for D-I images.
+# (There are not currently other checksums for these.)
di_add_files() if @di_dists;
say("Get Packages and Sources files and other miscellany.");
@@ -1050,7 +1050,7 @@ say("Parse Packages and Sources files and add to the file
list everything therei
}
($size)=m/^Size:\s+(\d+)/im;
($md5sum)=m/^MD5sum:\s+([A-Za-z0-9]+)/im;
- if (check_file($filename, $size, $md5sum)) {
+ if (check_file($filename, $size, $md5sum, "md5")) {
$files{$filename} = 1;
} else {
$files{$filename} = 0;
@@ -1096,7 +1096,7 @@ say("Parse Packages and Sources files and add to the file
list everything therei
$files{$filename} = 0;
}
}
- if (check_file($filename, $size, $md5sum)) {
+ if (check_file($filename, $size, $md5sum, "md5")) {
$files{$filename} = 1;
} else {
$files{$filename} = 0;
@@ -1248,7 +1248,7 @@ DOWNLOAD: {
foreach my $dest (@result) {
if (-f $dest) {
if (!check_lists($dest)) {
- say("$dest failed md5sum check");
+ say("$dest failed checksum verification");
$num_errors++;
}
} elsif (!-d $dest) {
@@ -1279,7 +1279,7 @@ DOWNLOAD: {
foreach my $dest (@result) {
if (-f $dest) {
if (!check_lists($dest)) {
- say("$dest failed md5sum check");
+ say("$dest failed checksum verification");
$num_errors++;
}
} elsif (!-d $dest) {
@@ -1416,22 +1416,28 @@ sub add_bytes_gotten {
}
}
-# Pass this function a filename, a file size (bytes), and a md5sum (hex).
-# Size is always checked; checking the md5sum is optional. However, if
-# a value of -1 is passed for size, a check of the md5sum is forced.
+# Pass this function a filename, a file size (bytes), a checksum,
+# and the type of checksum ("sha1", "md5", etc).
+# Size is always checked; checking the checksum is optional. However, if
+# a value of -1 is passed for size, a check of the checksum is forced.
# It will return true if the tests show the file matches.
sub check_file {
- my ($filename, $size, $md5sum)=@_;
+ my ($filename, $size, $checksum, $checksum_type)=@_;
if (-f $filename and ($size == -s _ || $size == -1)) {
- if ($check_md5sums || $size == -1) {
+ if ($verify_checksums || $size == -1) {
open HANDLE, $filename or
die "$filename: $!";
- $md5->addfile(*HANDLE);
- my $digest = $md5->hexdigest;
- return ($md5sum eq $digest);
+ my $calculated_checksum;
+ if ($checksum_type eq 'md5') {
+ $md5->addfile(*HANDLE);
+ $calculated_checksum = $md5->hexdigest;
+ }
+ else {
+ die "unsupported checksum type: $checksum_type\n";
+ }
+ return ($checksum eq $calculated_checksum);
}
else {
- # Assume it is ok, w/o md5 check.
return 1;
}
}
@@ -1439,7 +1445,7 @@ sub check_file {
}
# Always checks both file size and sha1 as the files get updated (this is
-# similar to what is done in check_lists, which forces check_md5sums).
+# similar to what is done in check_lists, which forces verify_checksums).
sub check_i18n {
my ($filename, $size, $sha1)=@_;
my $digest = Digest::SHA1->new;
@@ -1471,17 +1477,17 @@ sub check_diff {
return $ret;
}
-# Check file against md5sum and size from the Release file.
-# It will return true if the md5sum matches.
+# Check file against checksum and size from the Release file.
+# It will return true if the checksum matches.
sub check_lists {
my $file = shift;
- my $t = $check_md5sums;
+ my $t = $verify_checksums;
my $ret = 1;
- $check_md5sums = 1;
+ $verify_checksums = 1;
if (exists $file_lists{$file}) {
- $ret = check_file($file, $file_lists{$file}{size},
$file_lists{$file}{md5});
+ $ret = check_file($file, $file_lists{$file}{size},
$file_lists{$file}{md5}, "md5");
}
- $check_md5sums = $t;
+ $verify_checksums = $t;
return $ret;
}
@@ -1500,7 +1506,7 @@ sub remote_get {
$res=hftp_get($file);
$res=$res && check_lists($file);
if (-f $file && !$res) {
- say("$file failed md5sum check, removing");
+ say("$file failed checksum verification, removing");
unlink($file) if (-f $file);
}
};
@@ -1509,7 +1515,7 @@ sub remote_get {
$res=http_get($file);
$res=$res && check_lists($file);
if (-f $file && !$res) {
- say("$file failed md5sum check, removing");
+ say("$file failed checksum verification, removing");
unlink($file) if (-f $file);
}
};
@@ -1518,7 +1524,7 @@ sub remote_get {
$res=ftp_get($file);
$res=$res && check_lists($file);
if (-f $file && !$res) {
- say("$file failed md5sum check, removing");
+ say("$file failed checksum verification, removing");
unlink($file) if (-f $file);
}
};
@@ -1527,7 +1533,7 @@ sub remote_get {
$res=rsync_get($file);
$res=$res && check_lists($file);
if (-f $file && !$res) {
- say("$file failed md5sum check");
+ say("$file failed checksum verification");
# FIXME: make sure the size doesn't match so it gets retried
}
};
@@ -1906,7 +1912,7 @@ sub get_index {
make_dir("$tempdir/$subdir/$file.diff");
say("$subdir/$file.diff/Index needs fetch");
if (!remote_get("$subdir/$file.diff/Index")) {
- push (@errlog,"$subdir/$file.diff/Index failed md5sum check,
removing\n");
+ push (@errlog,"$subdir/$file.diff/Index failed checksum verification,
removing\n");
} else {
fetch_and_apply_diffs(0, $subdir, $file);
if (check_lists ("$tempdir/$subdir/$file")) {
@@ -1933,7 +1939,7 @@ sub get_index {
system_redirect_io("gzip -d", "$tempdir/$subdir/$file.gz",
"$tempdir/$subdir/$file");
system_redirect_io("bzip2", "$tempdir/$subdir/$file",
"$tempdir/$subdir/$file.bz2");
} else {
- push (@errlog,"$subdir/$file.gz failed md5sum check\n");
+ push (@errlog,"$subdir/$file.gz failed checksum verification\n");
$num_errors++;
}
} else {
@@ -1960,7 +1966,7 @@ sub get_index {
if (remote_get("$subdir/$file")) {
system_redirect_io("bzip2", "$tempdir/$subdir/$file",
"$tempdir/$subdir/$file.bz2");
} else {
- push (@errlog,"$subdir/$file failed md5sum check\n");
+ push (@errlog,"$subdir/$file failed checksum verification\n");
$num_errors++;
}
} else {
@@ -1971,7 +1977,7 @@ sub get_index {
if (!check_lists ("$tempdir/$subdir/$file.bz2")) {
say("$subdir/$file.bz2 needs fetch");
if (!remote_get("$subdir/$file.bz2")) {
- push (@errlog,"$subdir/$file.bz2 failed md5sum check, removing\n");
+ push (@errlog,"$subdir/$file.bz2 failed checksum verification,
removing\n");
}
} else {
$bytes_gotten += $file_lists{"$tempdir/$subdir/$file.bz2"}{size};
@@ -1981,7 +1987,7 @@ sub get_index {
if (!check_lists ("$tempdir/$subdir/Release")) {
say("$subdir/Release needs fetch");
if (!remote_get("$subdir/Release")) {
- push (@errlog,"$subdir/Release failed md5sum check, removing\n");
+ push (@errlog,"$subdir/Release failed checksum verification,
removing\n");
}
} else {
$bytes_gotten += $file_lists{"$tempdir/$subdir/Release"}{size};
@@ -2018,7 +2024,7 @@ sub update_contents {
make_dir("$tempdir/$subdir/$file.diff");
say("$subdir/$file.diff/Index needs fetch");
if (!remote_get("$subdir/$file.diff/Index")) {
- push (@errlog,"$subdir/$file.diff/Index failed md5sum check,
removing\n");
+ push (@errlog,"$subdir/$file.diff/Index failed checksum verification,
removing\n");
return $file_ok;
}
#FIXME: before download
@@ -2081,7 +2087,7 @@ sub get_i18n_index {
if (!check_lists ("$tempdir/$subdir/Index")) {
say("$subdir/Release needs fetch");
if (!remote_get("$subdir/Index")) {
- push (@errlog,"$subdir/Index failed md5sum check, removing\n");
+ push (@errlog,"$subdir/Index failed checksum verification, removing\n");
}
} else {
$bytes_gotten += $file_lists{"$tempdir/$subdir/Index"}{size};
@@ -2347,7 +2353,7 @@ sub di_add_files {
#$di_files{$image_dir}{$filename}{size} =
get_http_size("$image_dir/$filename");
# Check against the version currently on the mirror
- if (check_file("$image_dir/$filename", -1, $md5sum)) {
+ if (check_file("$image_dir/$filename", -1, $md5sum, "md5")) {
$di_files{$image_dir}{$filename}{status} = 1;
} else {
$di_files{$image_dir}{$filename}{status} = 0;
@@ -2372,7 +2378,7 @@ sub di_get_files {
$file =~ m:(^.*)/:;
make_dir ("$tdir/$image_dir/$1") if $1;
if (!remote_get("$image_dir/$file", $tdir) ||
- !check_file("$tdir/$image_dir/$file", -1,
$di_files{$image_dir}{$file}{md5sum})) {
+ !check_file("$tdir/$image_dir/$file", -1,
$di_files{$image_dir}{$file}{md5sum}, "md5")) {
$lres = 0;
last if (! $do_dry_run);
}
--
1.7.8.2
From: Joey Hess <j...@kitenet.net>
Date: Mon, 21 Feb 2011 15:23:08 -0400
Subject: Now depends on libdigest-sha-perl.
commit f7e94967a56de6f1e657cfbb250c31004cfcef02 upstream.
---
debian/changelog | 1 +
debian/control | 2 +-
debmirror | 17 ++++++++---------
3 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/debian/changelog b/debian/changelog
index 9ec3b8a1..8938aec1 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -5,6 +5,7 @@ debmirror (1:2.4.5+squeeze1) UNRELEASED; urgency=low
Closes: #614383
* Full checksum validation is now enabled by the --checksums switch.
(The old --md5sums switch is an alias to that for backwards compatability.)
+ * Now depends on libdigest-sha-perl.
-- Jonathan Nieder <jrnie...@gmail.com> Thu, 05 Jan 2012 15:06:55 -0600
diff --git a/debian/control b/debian/control
index d914461f..398eaddc 100644
--- a/debian/control
+++ b/debian/control
@@ -9,7 +9,7 @@ Vcs-Browser: http://svn.debian.org/wsvn/debmirror/trunk/
Package: debmirror
Architecture: all
-Depends: ${misc:Depends}, perl (>= 5.10), libnet-perl, libdigest-md5-perl,
libdigest-sha1-perl, liblockfile-simple-perl, rsync, libcompress-zlib-perl,
bzip2, libwww-perl
+Depends: ${misc:Depends}, perl (>= 5.10), libnet-perl, libdigest-md5-perl,
libdigest-sha-perl, liblockfile-simple-perl, rsync, libcompress-zlib-perl,
bzip2, libwww-perl
Recommends: gpgv, patch, ed
Suggests: gnupg
Description: Debian partial mirror script, with ftp and package pool support
diff --git a/debmirror b/debmirror
index eeb57732..d18ad100 100755
--- a/debmirror
+++ b/debmirror
@@ -511,7 +511,7 @@ use File::Temp qw/ tempfile /;
use LockFile::Simple;
use Compress::Zlib;
use Digest::MD5;
-use Digest::SHA1;
+use Digest::SHA;
use LWP::UserAgent;
# Yeah, I use too many global variables in this program.
@@ -725,9 +725,6 @@ if ($post_cleanup) {
}
say("Dry run.") if $dry_run;
-my $md5;
-$md5=Digest::MD5->new;
-
# Set up mirror directory and resolve $mirrordir to a full path for
# locking and rsync
make_dir($mirrordir) if (! -d $mirrordir);
@@ -1418,7 +1415,7 @@ sub add_bytes_gotten {
# Pass this function a filename, a file size (bytes), a checksum,
# and the type of checksum ("sha1", "md5", etc).
-# Size is always checked; checking the checksum is optional. However, if
+# Size is always checked; verifying the checksum is optional. However, if
# a value of -1 is passed for size, a check of the checksum is forced.
# It will return true if the tests show the file matches.
sub check_file {
@@ -1429,6 +1426,8 @@ sub check_file {
die "$filename: $!";
my $calculated_checksum;
if ($checksum_type eq 'md5') {
+ my $md5;
+ $md5=Digest::MD5->new;
$md5->addfile(*HANDLE);
$calculated_checksum = $md5->hexdigest;
}
@@ -1448,7 +1447,7 @@ sub check_file {
# similar to what is done in check_lists, which forces verify_checksums).
sub check_i18n {
my ($filename, $size, $sha1)=@_;
- my $digest = Digest::SHA1->new;
+ my $digest = Digest::SHA->new(1);
my $ret = 0;
if (-f "$filename" and ($size == -s _)) {
@@ -1462,7 +1461,7 @@ sub check_i18n {
# Check uncompressed diff content against sha1sum from Index file.
sub check_diff {
my ($filename, $size, $sha1) = @_;
- my $digest = Digest::SHA1->new;
+ my $digest = Digest::SHA->new(1);
my $ret = 0;
if (-f "$filename.gz") {
@@ -2196,7 +2195,7 @@ sub fetch_and_apply_diffs {
# Apply diff files
open(FILE, "$tempdir/$subdir/$type") or return;
- $digest = Digest::SHA1->new;
+ $digest = Digest::SHA->new(1);
$digest->addfile(*FILE);
$sha1 = $digest->hexdigest;
$size = -s "$tempdir/$subdir/$type";
@@ -2208,7 +2207,7 @@ sub fetch_and_apply_diffs {
return;
}
open(FILE, "$tempdir/$subdir/$type") or return;
- $digest = Digest::SHA1->new;
+ $digest = Digest::SHA->new(1);
$digest->addfile(*FILE);
$sha1 = $digest->hexdigest;
$size = -s "$tempdir/$subdir/$type";
--
1.7.8.2
From: Joey Hess <j...@kitenet.net>
Date: Mon, 21 Feb 2011 15:27:27 -0400
Subject: add support for checking arbitrary sha checksum sizes
commit ed58c425c40e24d550103a550763340f2e40dde5 upstream.
---
debmirror | 11 +++++++++--
1 files changed, 9 insertions(+), 2 deletions(-)
diff --git a/debmirror b/debmirror
index d18ad100..14e9f801 100755
--- a/debmirror
+++ b/debmirror
@@ -1426,11 +1426,18 @@ sub check_file {
die "$filename: $!";
my $calculated_checksum;
if ($checksum_type eq 'md5') {
- my $md5;
- $md5=Digest::MD5->new;
+ my $md5=Digest::MD5->new;
$md5->addfile(*HANDLE);
$calculated_checksum = $md5->hexdigest;
}
+ elsif ($checksum_type=~/^sha(\d+)$/) {
+ my $sha=Digest::SHA($1);
+ if (! defined $sha) {
+ die "unsupported checksum type: $checksum_type (a newer
Digest::SHA may be needed)\n";
+ }
+ $sha->addfile(*HANDLE);
+ $calculated_checksum = $sha->hexdigest;
+ }
else {
die "unsupported checksum type: $checksum_type\n";
}
--
1.7.8.2
From: Joey Hess <j...@kitenet.net>
Date: Mon, 21 Feb 2011 15:56:32 -0400
Subject: parse all types of checksums in the Release file
commit bb064fdb13dd9bda9971ce27fb10d37d6eb9e487 upstream.
Changed check_file to named parameters so it can accept multiple
checksum types for a file, and choose which it prefers to check.
Changed %file_lists to use the same spelling for checksums that
the Release file uses, and the data in the hash for a given file
can now be passed directly into check_file.
---
debmirror | 84 +++++++++++++++++++++++++++++++++---------------------------
1 files changed, 46 insertions(+), 38 deletions(-)
diff --git a/debmirror b/debmirror
index 14e9f801..c3ce0d62 100755
--- a/debmirror
+++ b/debmirror
@@ -846,18 +846,18 @@ di_check_dists() if @di_dists;
foreach my $dist (keys %distset) {
next unless exists $distset{$dist}{mirror};
- # Parse the Release
+ # Parse the Release and extract the files listed for all checksum types.
if (open RELEASE, "<$tempdir/dists/$dist/Release") {
+ my $checksum_type;
while (<RELEASE>) {
- last if /^MD5Sum:/;
- }
- $_ = <RELEASE>;
- while (defined $_ && $_ =~ /^ /) {
- my ($md5sum, $size, $filename) =
- (/ ([a-z0-9]+) +(\d+) +(.*)$/);
- $file_lists{"$tempdir/dists/$dist/$filename"}{md5} = $md5sum;
- $file_lists{"$tempdir/dists/$dist/$filename"}{size} = $size;
- $_ = <RELEASE>;
+ if (/^(MD5Sum|SHA\d+):/) {
+ $checksum_type=$1;
+ }
+ elsif (/^ / && defined $checksum_type) {
+ my ($checksum, $size, $filename) = /^ +([a-z0-9]+) +(\d+)
+(.*)$/;
+ $file_lists{"$tempdir/dists/$dist/$filename"}{$checksum_type} =
$checksum;
+ $file_lists{"$tempdir/dists/$dist/$filename"}{size} = $size;
+ }
}
close RELEASE;
}
@@ -1047,11 +1047,11 @@ say("Parse Packages and Sources files and add to the
file list everything therei
}
($size)=m/^Size:\s+(\d+)/im;
($md5sum)=m/^MD5sum:\s+([A-Za-z0-9]+)/im;
- if (check_file($filename, $size, $md5sum, "md5")) {
+ if (check_file(filename => $filename, size => $size, MD5Sum => $md5sum))
{
$files{$filename} = 1;
} else {
$files{$filename} = 0;
- $file_lists{$filename}{md5} = $md5sum;
+ $file_lists{$filename}{MD5Sum} = $md5sum;
$file_lists{$filename}{size} = $size;
$bytes_to_get += $size;
}
@@ -1093,11 +1093,11 @@ say("Parse Packages and Sources files and add to the
file list everything therei
$files{$filename} = 0;
}
}
- if (check_file($filename, $size, $md5sum, "md5")) {
+ if (check_file(filename => $filename, size => $size, MD5Sum =>
$md5sum)) {
$files{$filename} = 1;
} else {
$files{$filename} = 0;
- $file_lists{$filename}{md5} = $md5sum;
+ $file_lists{$filename}{MD5Sum} = $md5sum;
$file_lists{$filename}{size} = $size;
$bytes_to_get += $size;
}
@@ -1413,35 +1413,43 @@ sub add_bytes_gotten {
}
}
-# Pass this function a filename, a file size (bytes), a checksum,
-# and the type of checksum ("sha1", "md5", etc).
+# Takes named parameters: filename, size.
+# Optionally can also be passed parameters specifying expected checksums
+# for the file, using checksum names as in the Release file
+# ("SHA1", "MD5Sum", etc).
+#
# Size is always checked; verifying the checksum is optional. However, if
# a value of -1 is passed for size, a check of the checksum is forced.
+#
# It will return true if the tests show the file matches.
sub check_file {
- my ($filename, $size, $checksum, $checksum_type)=@_;
+ my %params=@_;
+ my ($filename, $size)=delete @params{qw{filename size}};
if (-f $filename and ($size == -s _ || $size == -1)) {
if ($verify_checksums || $size == -1) {
- open HANDLE, $filename or
- die "$filename: $!";
- my $calculated_checksum;
- if ($checksum_type eq 'md5') {
- my $md5=Digest::MD5->new;
- $md5->addfile(*HANDLE);
- $calculated_checksum = $md5->hexdigest;
+ # Prefer checking stronger checksums, and failing that, fall back
+ # to whatever checksums are present and supported, trying to prefer
+ # FOObignum over FOOsmallnum.
+ my ($summer, $checksum);
+ foreach my $checksum_type ("SHA512", "SHA256", "SHA1", reverse sort keys
%params) {
+ if ($checksum_type eq 'MD5Sum') {
+ $summer=Digest::MD5->new;
+ }
+ elsif ($checksum_type=~/^SHA(\d+)$/) {
+ $summer=Digest::SHA($1);
+ # returns undef on unknown/too large SHA type
+ }
+ if (defined $summer) {
+ $checksum=$params{$checksum_type};
+ last;
+ }
}
- elsif ($checksum_type=~/^sha(\d+)$/) {
- my $sha=Digest::SHA($1);
- if (! defined $sha) {
- die "unsupported checksum type: $checksum_type (a newer
Digest::SHA may be needed)\n";
- }
- $sha->addfile(*HANDLE);
- $calculated_checksum = $sha->hexdigest;
+ if (! defined $summer) {
+ die "unsupported checksum type(s): ".(join(" ", keys
%params))."\n";
}
- else {
- die "unsupported checksum type: $checksum_type\n";
- }
- return ($checksum eq $calculated_checksum);
+ open HANDLE, $filename or die "$filename: $!";
+ $summer->addfile(*HANDLE);
+ return ($checksum eq $summer->hexdigest);
}
else {
return 1;
@@ -1491,7 +1499,7 @@ sub check_lists {
my $ret = 1;
$verify_checksums = 1;
if (exists $file_lists{$file}) {
- $ret = check_file($file, $file_lists{$file}{size},
$file_lists{$file}{md5}, "md5");
+ $ret = check_file(filename => $file, %{$file_lists{$file}});
}
$verify_checksums = $t;
return $ret;
@@ -2359,7 +2367,7 @@ sub di_add_files {
#$di_files{$image_dir}{$filename}{size} =
get_http_size("$image_dir/$filename");
# Check against the version currently on the mirror
- if (check_file("$image_dir/$filename", -1, $md5sum, "md5")) {
+ if (check_file(filename => "$image_dir/$filename", size => -1, MD5Sum
=> $md5sum)) {
$di_files{$image_dir}{$filename}{status} = 1;
} else {
$di_files{$image_dir}{$filename}{status} = 0;
@@ -2384,7 +2392,7 @@ sub di_get_files {
$file =~ m:(^.*)/:;
make_dir ("$tdir/$image_dir/$1") if $1;
if (!remote_get("$image_dir/$file", $tdir) ||
- !check_file("$tdir/$image_dir/$file", -1,
$di_files{$image_dir}{$file}{md5sum}, "md5")) {
+ !check_file(file => "$tdir/$image_dir/$file", size => -1, MD5Sum =>
$di_files{$image_dir}{$file}{md5sum})) {
$lres = 0;
last if (! $do_dry_run);
}
--
1.7.8.2
From: Joey Hess <j...@kitenet.net>
Date: Mon, 21 Feb 2011 16:11:32 -0400
Subject: extract all checksums from the Packages files
commit b14c10f0a5ad5ca792e7a04da04462b8fd5c8925 upstream.
The sources files are the only thing still only md5sums are used for now.
(Besides d-i.)
Checksum types are specified using varying case amoung the Sources
and Packages and Release file, so support all cases. (Ugh.)
---
debmirror | 20 +++++++++++++-------
1 files changed, 13 insertions(+), 7 deletions(-)
diff --git a/debmirror b/debmirror
index c3ce0d62..8dacf666 100755
--- a/debmirror
+++ b/debmirror
@@ -1000,7 +1000,7 @@ say("Parse Packages and Sources files and add to the file
list everything therei
{
local $/="\n\n"; # Set input separator to read entire package
- my ($filename, $size, $md5sum, $directory, $exclude, $include,
+ my ($filename, $size, $directory, $exclude, $include,
$architecture, $exclude_deb_section, $limit_priority, $deb_section,
$deb_priority);
my $empty_mirror = 1;
@@ -1046,12 +1046,15 @@ say("Parse Packages and Sources files and add to the
file list everything therei
}
}
($size)=m/^Size:\s+(\d+)/im;
- ($md5sum)=m/^MD5sum:\s+([A-Za-z0-9]+)/im;
- if (check_file(filename => $filename, size => $size, MD5Sum => $md5sum))
{
+ my %checksums;
+ while (m/^(MD5sum|SHA(?:\d+)):\s+([A-Za-z0-9]+)/img) {
+ $checksums{$1}=$2;
+ }
+ if (check_file(filename => $filename, size => $size, %checksums)) {
$files{$filename} = 1;
} else {
$files{$filename} = 0;
- $file_lists{$filename}{MD5Sum} = $md5sum;
+ $file_lists{$filename} = \%checksums;
$file_lists{$filename}{size} = $size;
$bytes_to_get += $size;
}
@@ -1077,6 +1080,7 @@ say("Parse Packages and Sources files and add to the file
list everything therei
next if (defined($limit_priority) && defined($deb_priority)
&& ! ($deb_priority=~/$limit_priority/o));
while (m/^ ([A-Za-z0-9]{32} .*)/mg) {
+ my $md5sum;
($md5sum, $size, $filename)=split(' ', $1, 3);
$filename="$directory/$filename";
$filename=~s:/+:/:; # remove redundant slashes in paths
@@ -1414,8 +1418,9 @@ sub add_bytes_gotten {
}
# Takes named parameters: filename, size.
+#
# Optionally can also be passed parameters specifying expected checksums
-# for the file, using checksum names as in the Release file
+# for the file, using checksum names as in the Release/Packages/Sources files
# ("SHA1", "MD5Sum", etc).
#
# Size is always checked; verifying the checksum is optional. However, if
@@ -1432,10 +1437,11 @@ sub check_file {
# FOObignum over FOOsmallnum.
my ($summer, $checksum);
foreach my $checksum_type ("SHA512", "SHA256", "SHA1", reverse sort keys
%params) {
- if ($checksum_type eq 'MD5Sum') {
+ next unless defined $params{$checksum_type};
+ if (lc $checksum_type eq 'md5sum') {
$summer=Digest::MD5->new;
}
- elsif ($checksum_type=~/^SHA(\d+)$/) {
+ elsif ($checksum_type=~/^sha(\d+)$/i) {
$summer=Digest::SHA($1);
# returns undef on unknown/too large SHA type
}
--
1.7.8.2
From: Joey Hess <j...@kitenet.net>
Date: Mon, 21 Feb 2011 16:16:07 -0400
Subject: tweak
commit 2f462e10ed5cb99b6a5f19a2b471effd30ceffa1 upstream.
---
debian/changelog | 7 ++++---
1 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/debian/changelog b/debian/changelog
index 8938aec1..89989c26 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,11 +1,12 @@
debmirror (1:2.4.5+squeeze1) UNRELEASED; urgency=low
[ Joey Hess ]
- * Support sha checksums in Release files, and use in preference to md5.
- Closes: #614383
+ * Support all sizes of SHA checksums that are supported by Digest::SHA
+ in Release and Packages files, and use SHA512, SHA256, and SHA1 in
+ preference to MD5. Closes: #614383
+ * Now depends on libdigest-sha-perl.
* Full checksum validation is now enabled by the --checksums switch.
(The old --md5sums switch is an alias to that for backwards compatability.)
- * Now depends on libdigest-sha-perl.
-- Jonathan Nieder <jrnie...@gmail.com> Thu, 05 Jan 2012 15:06:55 -0600
--
1.7.8.2
From: Joey Hess <j...@kitenet.net>
Date: Mon, 21 Feb 2011 16:39:47 -0400
Subject: bugfix
commit b9eeb3374cbc350218379b2f104cf44f198492c2 upstream.
---
debmirror | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/debmirror b/debmirror
index 8dacf666..40e71c91 100755
--- a/debmirror
+++ b/debmirror
@@ -1047,7 +1047,7 @@ say("Parse Packages and Sources files and add to the file
list everything therei
}
($size)=m/^Size:\s+(\d+)/im;
my %checksums;
- while (m/^(MD5sum|SHA(?:\d+)):\s+([A-Za-z0-9]+)/img) {
+ while (m/^(MD5sum|SHA\d+):\s+([A-Za-z0-9]+)/img) {
$checksums{$1}=$2;
}
if (check_file(filename => $filename, size => $size, %checksums)) {
@@ -1442,8 +1442,8 @@ sub check_file {
$summer=Digest::MD5->new;
}
elsif ($checksum_type=~/^sha(\d+)$/i) {
- $summer=Digest::SHA($1);
# returns undef on unknown/too large SHA type
+ $summer=Digest::SHA->new($1);
}
if (defined $summer) {
$checksum=$params{$checksum_type};
--
1.7.8.2
From: Joey Hess <j...@kitenet.net>
Date: Mon, 21 Feb 2011 17:13:24 -0400
Subject: refactor and levical scoping cleanup
commit 5aad724f5c7ffa33f6cac824b7b04a6c3190e01e upstream.
---
debian/changelog | 6 ++--
debmirror | 81 +++++++++++++++++++++++++++++++++++------------------
2 files changed, 56 insertions(+), 31 deletions(-)
diff --git a/debian/changelog b/debian/changelog
index 89989c26..8b78a151 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,9 +1,9 @@
debmirror (1:2.4.5+squeeze1) UNRELEASED; urgency=low
[ Joey Hess ]
- * Support all sizes of SHA checksums that are supported by Digest::SHA
- in Release and Packages files, and use SHA512, SHA256, and SHA1 in
- preference to MD5. Closes: #614383
+ * Support all sizes of SHA checksums (that are supported by Digest::SHA)
+ in Release, Packages, and Sources files. Check SHA512, SHA256, or SHA1
+ in preference to MD5. Closes: #614383
* Now depends on libdigest-sha-perl.
* Full checksum validation is now enabled by the --checksums switch.
(The old --md5sums switch is an alias to that for backwards compatability.)
diff --git a/debmirror b/debmirror
index 40e71c91..47d5e62b 100755
--- a/debmirror
+++ b/debmirror
@@ -1000,18 +1000,16 @@ say("Parse Packages and Sources files and add to the
file list everything therei
{
local $/="\n\n"; # Set input separator to read entire package
- my ($filename, $size, $directory, $exclude, $include,
- $architecture, $exclude_deb_section, $limit_priority, $deb_section,
- $deb_priority);
my $empty_mirror = 1;
my %arches = map { $_ => 1 } (@arches, "all");
- $include = "(".join("|", @includes).")" if @includes;
- $exclude = "(".join("|", @excludes).")" if @excludes;
- $exclude_deb_section =
+ my $include = "(".join("|", @includes).")" if @includes;
+ my $exclude = "(".join("|", @excludes).")" if @excludes;
+ my $exclude_deb_section =
"(".join("|", @excludes_deb_section).")" if @excludes_deb_section;
- $limit_priority = "(".join("|", @limit_priority).")" if @limit_priority;
+ my $limit_priority = "(".join("|", @limit_priority).")" if @limit_priority;
+
foreach my $file (@package_files) {
next if (!-f $file);
open(FILE, "<", $file) or die "$file: $!";
@@ -1022,11 +1020,11 @@ say("Parse Packages and Sources files and add to the
file list everything therei
die "$file: $!" if $!;
}
$_ = $buf;
- ($filename)=m/^Filename:\s+(.*)/im;
+ my ($filename)=m/^Filename:\s+(.*)/im;
$filename=~s:/+:/:; # remove redundant slashes in paths
- ($deb_section)=m/^Section:\s+(.*)/im;
- ($deb_priority)=m/^Priority:\s+(.*)/im;
- ($architecture)=m/^Architecture:\s+(.*)/im;
+ my ($deb_section)=m/^Section:\s+(.*)/im;
+ my ($deb_priority)=m/^Priority:\s+(.*)/im;
+ my ($architecture)=m/^Architecture:\s+(.*)/im;
next if (!$arches{$architecture});
if(!(defined($include) && ($filename=~/$include/o))) {
next if (defined($exclude) && $filename=~/$exclude/o);
@@ -1045,7 +1043,7 @@ say("Parse Packages and Sources files and add to the file
list everything therei
$files{$filename} = 0;
}
}
- ($size)=m/^Size:\s+(\d+)/im;
+ my ($size)=m/^Size:\s+(\d+)/im;
my %checksums;
while (m/^(MD5sum|SHA\d+):\s+([A-Za-z0-9]+)/img) {
$checksums{$1}=$2;
@@ -1062,7 +1060,7 @@ say("Parse Packages and Sources files and add to the file
list everything therei
}
close(FILE);
}
- foreach my $file (@source_files) {
+SOURCE: foreach my $file (@source_files) {
next if (!-f $file);
open(FILE, "<", $file) or die "$file: $!";
for (;;) {
@@ -1071,17 +1069,45 @@ say("Parse Packages and Sources files and add to the
file list everything therei
last if eof;
die "$file: $!" if $!;
}
- $_ = $buf;
- ($directory) = m/^Directory:\s+(.*)/im;
- ($deb_section)=m/^Section:\s+(.*)/im;
- ($deb_priority)=m/^Priority:\s+(.*)/im;
- next if (defined($exclude_deb_section) && defined($deb_section)
- && $deb_section=~/$exclude_deb_section/o);
- next if (defined($limit_priority) && defined($deb_priority)
- && ! ($deb_priority=~/$limit_priority/o));
- while (m/^ ([A-Za-z0-9]{32} .*)/mg) {
- my $md5sum;
- ($md5sum, $size, $filename)=split(' ', $1, 3);
+ my @lines=split(/\n/, $buf);
+
+ my $directory;
+ my %source_files;
+ my $parse_source_files=sub {
+ my $checksum_type=shift;
+ while (@lines && $lines[0] =~ m/^ ([A-Za-z0-9]+ .*)/) {
+ my ($checksum, $size, $filename)=split(' ', $1, 3);
+ print "source $checksum_type $filename = $checksum\n";
+ $source_files{$filename}{size}=$size;
+ $source_files{$filename}{$checksum_type}=$checksum;
+ shift @lines;
+ }
+ };
+
+ while (@lines) {
+ my $line=shift @lines;
+ if ($line=~/^Directory:\s+(.*)/i) {
+ $directory=$1;
+ }
+ elsif ($line=~/^Section:\s+(.*)/i) {
+ my $deb_section=$1;
+ next SOURCE if (defined($exclude_deb_section) &&
defined($deb_section)
+ && $deb_section=~/$exclude_deb_section/o);
+ }
+ elsif ($line=~/^Priority:\s+(.*)/i) {
+ my $deb_priority=$1;
+ next SOURCE if (defined($limit_priority) &&
defined($deb_priority)
+ && ! ($deb_priority=~/$limit_priority/o));
+ }
+ elsif ($line=~/^Files:/i) {
+ $parse_source_files->("MD5Sum");
+ }
+ elsif ($line=~/^Checksums-(\w+):/i) {
+ $parse_source_files->($1);
+ }
+ }
+ foreach my $filename (keys %source_files) {
+ my %file_data=%{$source_files{$filename}};
$filename="$directory/$filename";
$filename=~s:/+:/:; # remove redundant slashes in paths
if(!(defined($include) && $filename=~/$include/o)) {
@@ -1097,13 +1123,12 @@ say("Parse Packages and Sources files and add to the
file list everything therei
$files{$filename} = 0;
}
}
- if (check_file(filename => $filename, size => $size, MD5Sum =>
$md5sum)) {
+ if (check_file(filename => $filename, %file_data)) {
$files{$filename} = 1;
} else {
$files{$filename} = 0;
- $file_lists{$filename}{MD5Sum} = $md5sum;
- $file_lists{$filename}{size} = $size;
- $bytes_to_get += $size;
+ $file_lists{$filename} = \%file_data;
+ $bytes_to_get += $file_data{size};
}
}
$empty_mirror = 0;
--
1.7.8.2
From: Joey Hess <j...@kitenet.net>
Date: Mon, 21 Feb 2011 17:18:01 -0400
Subject: remove debug
commit 6711c3eb53ba8566dd86552852705617455772be upstream.
---
debmirror | 1 -
1 files changed, 0 insertions(+), 1 deletions(-)
diff --git a/debmirror b/debmirror
index 47d5e62b..00e1e7b0 100755
--- a/debmirror
+++ b/debmirror
@@ -1077,7 +1077,6 @@ SOURCE: foreach my $file (@source_files) {
my $checksum_type=shift;
while (@lines && $lines[0] =~ m/^ ([A-Za-z0-9]+ .*)/) {
my ($checksum, $size, $filename)=split(' ', $1, 3);
- print "source $checksum_type $filename = $checksum\n";
$source_files{$filename}{size}=$size;
$source_files{$filename}{$checksum_type}=$checksum;
shift @lines;
--
1.7.8.2
From: Joey Hess <j...@kitenet.net>
Date: Mon, 21 Feb 2011 19:36:24 -0400
Subject: finalize changelog
commit d0d7dd846d973eea59636fb29fc37f3c9baba18e upstream.
---
debian/changelog | 10 +++++-----
1 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/debian/changelog b/debian/changelog
index 8b78a151..6996c780 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,12 +1,12 @@
-debmirror (1:2.4.5+squeeze1) UNRELEASED; urgency=low
+debmirror (1:2.4.5+squeeze1) stable; urgency=low
[ Joey Hess ]
- * Support all sizes of SHA checksums (that are supported by Digest::SHA)
- in Release, Packages, and Sources files. Check SHA512, SHA256, or SHA1
- in preference to MD5. Closes: #614383
+ * In Release, Packages, and Sources files, support all sizes of SHA
+ checksums that are supported by Digest::SHA. Closes: #614383
* Now depends on libdigest-sha-perl.
+ * Check SHA512, SHA256, or SHA1 in preference to MD5.
* Full checksum validation is now enabled by the --checksums switch.
- (The old --md5sums switch is an alias to that for backwards compatability.)
+ (The old --md5sums switch is an alias to that for backwards compatibility.)
-- Jonathan Nieder <jrnie...@gmail.com> Thu, 05 Jan 2012 15:06:55 -0600
--
1.7.8.2
