Hello, tech@ readers. At work i use rpm-based linux distribution (i'm writing this not to fire up new holywar thread). It package manager is called `yum', which is something like pkg_* tools but all in one executable file. This yum thing has nice plugin (which is called `fastest_mirror') that allows it automatically discover package repositories across the Net. When i'm installing software I have to type something like
# yum install tmux and it will automatically find fastest mirror with tmux rpm package. Main advantage of this is that i do not have to care about manual repository selection. Since i'm ill now (damn flu) i decided to implement such functionality in Perl for pkg_add(1) tool. I was always annoyed by the fact the i have to manually find fastest mirror for my newly openbsd installation by hands. If it fails to work i have change it by hands, again :-E But why i have to do things which computer can do? We are living in 21th century. Where are domestic laser beams, light sables, anti gravity cars and automatic package mirror discovery in pkg_add tool? :) So absence of automatic mirror discovery was very disappointing me. What ordinary OpenBSD use have to do when something annoys him? Right, shut up and hack. So i implemented basic functionality in separate perl program, which i think can be easily transformed into perl module that can be integrated into pkg_add. Please do not judge my work very strictly since this is my second program in Perl. First one was 'Hello, World!' which i wrote 5 minutes before i started writing AutoMirrorDiscovery.pl. I mostly programmed in C and PHP in my life, so i think i implemented C program in Perl and it is not written in usual Perl style. I tried to copy some style guidelines from pkg_* tools source code. So what it does? Here is the description of AutoMirrorDiscovery.pl logic: 1. Try to find cache file that was created by previous execution if it exists use first available mirror, then print it to stdout and exit otherwise: 2. Load in all available mirrors listed in /etc/ftp.mirrors (this file contains just list of all FTP mirrors listed in http://www.openbsd.org/ftp.html, each mirror on separate line). 3. Start pinging each mirror. For this stuff i use Net::Ping module. Unfortunately to use icmp ping you need root privileges. I also tried to use UDP pings (which is default for Net::Ping), but they work very slowly. I mean UDP ping shows much higher response time than ICMP ping. 4. Then mirrors are sorted by response time. Fastest are getting at the beginning of hash list. 5. Then it tries to find 3 available mirrors that contain packages for our release version and application architecture. It just tries to connect to selected mirror and goes into package directory. Why this is needed (cwd to package directory)? We have to be sure that mirror which we going to use *really* contains packages. For example i found that ftp.chg.ru still does not provide packages for my 4.6 release. While we are traversing through mirror list we are writing cache to the disk. Ok not actually cache, but just a flat text file that contains 3 lines. Each line contains mirror for our location. First one is the fastest, second one is a bit slower and third is slowest from this trinity (slowest fast mirror :). Cache is needed so that second invocation will return fastest mirror in a jiffy, because for `yum' is also not ideal since it tries to find fastest mirror on every invocation, which i think is an overkill. Since PKG_PATH is allowed to contain several mirrors delimited by `:' we can just set it to contains all three mirrors. I do not know how yum determines fastest mirror, i did not looked it source code. So i just implemented simplest idea that comed to my mind. OS can be shipped with /etc/ftp.mirrors and each user will have only execute something like following or this even can be done during installation process, if perl is available during install process, which i think is not: # sudo pkg_add -v -d Automatic FTP mirror discovery started Checking ftp.openbsd.org for availability.....OK Checking ftp.chg.ru for availability....OK ... Determined three fastest mirrors for our location: ftp://mirrors.nic.funet.fi/pub/OpenBSD/4.6/packages/i386 ftp://ftp.gamma.ru/pub/OpenBSD/4.6/packages/i386 ftp://ftp.nluug.nl/pub/OpenBSD/4.6/packages/i386 Cache is written successfully. Just type pkg_add foobar and magic will happen. # pkg_add foobar ...normal installation procedure... I think that this feature will be very appreciated by mobile openbsd users when they travel a lot. You do not have to adjust your PKG_PATH anymore when you change your location. Just type pkg_add -d and the thing is done, but before it have to be fully implemented :) Maybe i'm just reinventing the wheel. Maybe all folks use just export PKG_PATH='all_ftp_mirrors_separated_by_colons' in your .profile and are happy. I think that espie@ can explain why such functionality wasn't implemented (because for me i think that it is not so hard to implement fully functional package mirror discovery), maybe there are some ideological constrains or something else. So i'm posting it here with hope that i will gain positive feedback. Currently my code has tons of limitations. Here is my TODO list: - integrate AutoMirrorDiscovery into pkg_add - handle -stable revision - handle -current revision (when calls uname -r) - handle `snapshots' - implement HTTP mirrors support - handle HTTP mirrors without LWP, since it is not available in base (better to make separate Perl module, instead of hacking this one) - implement AFS mirrors support (maybe an overkill?) - implement RSYNC mirrors support (maybe an overkill?) - make it FASTER!!! (it need to be profiled with dprofpp) - better error handling - handle situation when Internet connection goes through proxy - handle situation when icmp pings are prohibited - implement mirror cache (partly implemented) (mirrors that are sorted by response time, some kind of a file in /var/db/pkgmirror.cache) - handle link loss situation gracefully - handle signal interruption So if you want to test this piece of code just create /etc/ftp.mirrors with following content: # cat /etc/ftp.mirrors ftp://anga.funkfeuer.at/pub/OpenBSD/ ftp://carroll.cac.psu.edu/pub/OpenBSD/ ftp://filedump.se.rit.edu/pub/OpenBSD/ ftp://ftp-stud.fht-esslingen.de/pub/OpenBSD/ ftp://ftp.arcane-networks.fr/pub/OpenBSD/ ftp://ftp.aso.ee/pub/OpenBSD/ ftp://ftp.belnet.be/packages/openbsd/ ftp://ftp.bytemine.net/pub/OpenBSD/ ftp://ftp.ca.openbsd.org/pub/OpenBSD/ ftp://ftp.cc.uoc.gr/mirrors/OpenBSD/ ftp://ftp.chg.ru/pub/OpenBSD/ ftp://ftp.crans.org/pub/OpenBSD/ ftp://ftp.cs.pu.edu.tw/BSD/OpenBSD/ ftp://ftp.cse.buffalo.edu/pub/OpenBSD/ ftp://ftp.das.ufsc.br/pub/OpenBSD/ ftp://ftp.df.lth.se/pub/OpenBSD/ ftp://ftp.dkuug.dk/pub/OpenBSD/ ftp://ftp.duth.gr/pub/OpenBSD/ ftp://ftp.esat.net/pub/OpenBSD/ ftp://ftp.estpak.ee/pub/OpenBSD/ ftp://ftp.eu.openbsd.org/pub/OpenBSD/ ftp://ftp.fmed.uc.pt/pub/OpenBSD/ ftp://ftp.fr.openbsd.org/pub/OpenBSD/ ftp://ftp.freebsdchina.org/pub/OpenBSD/ ftp://ftp.freenet.de/pub/ftp.openbsd.org/pub/OpenBSD/ ftp://ftp.fsn.hu/pub/OpenBSD/ ftp://ftp.gamma.ru/pub/OpenBSD/ ftp://ftp.heanet.ie/pub/OpenBSD/ ftp://ftp.iinet.net.au/pub/OpenBSD/ ftp://ftp.inet.no/pub/OpenBSD/ ftp://ftp.irisa.fr/pub/OpenBSD/ ftp://ftp.is.co.za/mirror/ftp.openbsd.org/ ftp://ftp.jaist.ac.jp/pub/OpenBSD/ ftp://ftp.jyu.fi/pub/OpenBSD/ ftp://ftp.kaist.ac.kr/pub/OpenBSD/ ftp://ftp.kddlabs.co.jp/OpenBSD/ ftp://ftp.lambdaserver.com/pub/OpenBSD/ ftp://ftp.mirrorservice.org/pub/OpenBSD/ ftp://ftp.netbsd.se/OpenBSD/ ftp://ftp.nluug.nl/pub/OpenBSD/ ftp://ftp.obsd.si/pub/OpenBSD/ ftp://ftp.openbsd.dk/pub/OpenBSD/ ftp://ftp.openbsd.or.id/pub/OpenBSD/ ftp://ftp.openbsd.org.ar/pub/OpenBSD/ ftp://ftp.openbsd.org/pub/OpenBSD/ ftp://ftp.piotrkosoft.net/pub/OpenBSD/ ftp://ftp.plig.net/pub/OpenBSD/ ftp://ftp.rediris.es/pub/OpenBSD/ ftp://ftp.spline.de/pub/OpenBSD/ ftp://ftp.task.gda.pl/pub/OpenBSD/ ftp://ftp.tcc.edu.tw/pub/OpenBSD/ ftp://ftp.tpnet.pl/pub/OpenBSD/ ftp://ftp.tw.openbsd.org/pub/OpenBSD/ ftp://ftp.udc.es/pub/OpenBSD/ ftp://ftp.ulak.net.tr/OpenBSD/ ftp://ftp.uninett.no/pub/OpenBSD/ ftp://ftp.wu-wien.ac.at/pub/OpenBSD/ ftp://ftp2.fr.openbsd.org/pub/OpenBSD/ ftp://ftp3.usa.openbsd.org/pub/OpenBSD/ ftp://ftp5.usa.openbsd.org/pub/OpenBSD/ ftp://gulus.usherbrooke.ca/pub/distro/OpenBSD/ ftp://mirror.aarnet.edu.au/pub/OpenBSD/ ftp://mirror.cdmon.com/pub/OpenBSD/ ftp://mirror.corbina.net/pub/OpenBSD/ ftp://mirror.hostfuss.com/pub/OpenBSD/ ftp://mirror.iawnet.sandia.gov/pub/OpenBSD/ ftp://mirror.internode.on.net/pub/OpenBSD/ ftp://mirror.pacific.net.au/OpenBSD/ ftp://mirror.planetunix.net/pub/OpenBSD/ ftp://mirror.public-internet.co.uk/pub/OpenBSD/ ftp://mirror.rit.edu/pub/OpenBSD/ ftp://mirror.roothell.org/pub/OpenBSD/ ftp://mirror.switch.ch/pub/OpenBSD/ ftp://mirrors.24-7-solutions.net/pub/OpenBSD/ ftp://mirrors.localhost.net.ar/pub/OpenBSD/ ftp://mirrors.nic.funet.fi/pub/OpenBSD/ ftp://mirrors.ucr.ac.cr/OpenBSD/ ftp://obsd.cec.mtu.edu/pub/OpenBSD/ ftp://openbsd.arcticnetwork.ca/pub/OpenBSD/ ftp://openbsd.bsdforen.de/pub/OpenBSD/ ftp://openbsd.ftp.fu-berlin.de/pub/OpenBSD/ ftp://openbsd.mirror.frontiernet.net/pub/OpenBSD/ ftp://openbsd.mirrors.pair.com/ ftp://openbsd.mirrors.tds.net/pub/OpenBSD/ ftp://openbsd.noc.jgm.gov.ar/pub/OpenBSD/ So here is the code itself (i hope GMail editor wont screw it): #!/usr/bin/perl # ex:ts=8 sw=4: # $OpenBSD$ # Copyright (c) 2009 Igor Zinovik <zino...@petrsu.ru> # # Permission to use, copy, modify, and distribute this software for any # purpose with or without fee is hereby granted, provided that the above # copyright notice and this permission notice appear in all copies. # # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. use strict; use warnings; use OpenBSD::Paths; use Net::Ping; use Net::FTP; my (@mirrors) = @_; if (-e "/var/db/pkg/ftpmirror.cache") { open my $fh, '<', "/var/db/pkg/ftpmirror.cache" or die("Permission denied"); @mirrors = <$fh>; close $fh; print $mirrors[0]; exit; } if (-e "/etc/ftp.mirrors") { open my $fh, '<', "/etc/ftp.mirrors" or die("Permission denied"); @mirrors = <$fh>; close $fh; } # Root privileges are required to do ICMP pings my $p = Net::Ping->new("icmp"); $p->hires(); my (%hosts) = (); foreach my $mirror (@mirrors) { my ($host) = ($mirror =~ /ftp:\/\/([\w\.-]*)\//); chomp $host; my ($ret, $duration) = $p->ping($host, 2.0); if ($ret == 1) { $hosts{$mirror} = $duration; } } $p->close(); my $cmd = OpenBSD::Paths->uname." -r"; my ($rev) = split(/\s+/o, `$cmd`); chomp $rev; $cmd = OpenBSD::Paths->arch." -s"; my ($app_arch) = split(/\s+/o, `$cmd`); chomp $app_arch; # Sort mirrors by response time, fast are closer to the beginning my @ftp_mirrors = sort {$hosts{$a} cmp $hosts{$b}} keys %hosts; open $fh, '>', "/var/db/pkg/ftpmirror.cache" or die("Cannot create mirror cache file"); my $pkgpath = "NULL"; my $tries = 0; foreach my $p (@ftp_mirrors) { # Check only 3 available mirrors if ($tries gt 2) { last; } chomp $p; my ($ftp_host) = ($p =~ /ftp:\/\/([\w\.-]*)\//); my ($ftp_dir) = ($p =~ /ftp:\/\/$ftp_host\/([\w\.-_]*)/); # Check mirror availability my $ftp = Net::FTP->new($ftp_host, Passive=>1) or next; if ($ftp->login('anonymous', '-anonymous@')) { $ftp_dir .= $rev."/packages/".$app_arch; if ($ftp->cwd($ftp_dir)) { # Seems that this mirror contains packages we need $pkgpath = $p.$rev."/packages/".$app_arch."/"; print $fh "$pkgpath\n"; $tries++; } $ftp->quit; next; } } close $fh; if ($pkgpath eq "NULL") { die("Seems that all mirrors are unavailable"); }
