Control: reassign -1 apt 1.0.9.8 Control: affects -1 aptitude Control: severity -1 important Control: tags -1 patch Control: retitle -1 actions on specific packages don't work on single-arch systems
Hi, (the mail starts with a LOT of technical bla bla a user doesn't need to concern itself with. I have cc'ed all reporters explicitly through as I would like them to try the "workarounds" mentioned further below, so feel free to skip over the technical reasoning straight to the 'meat'. And if you gonna reply, please ONLY to the bugreport, thanks!) On Tue, Apr 21, 2015 at 08:17:36PM +0200, Axel Beckert wrote: > David already seems to on to something... After quite some hair-pulling and the occasional swearing which Axel thankfully endured I finally arrived at a rather misfortune conclusion (at least for me): I was wrong, which is quite a hurtful experience as I had put quite a bit of energy into preventing it… So, desperate me claiming the fix for #777760 would have a low regression risk as there aren't that many packages in existence which expect this behaviour, I forgot to consider a potential side-effect: The correct creation of a package foo from architecture alien1 can if we haven't seen the package foo from our native architecture yet claim the price of being the first known package foo. The first place is assumed to belong to the package of the native architecture through, so a libapt application asking for the package foo of native architecture will get unconditionally handed the first package foo found, which if we are unlucky is the one for alien1 and not native… This assumption used to be true, but with the advent of architecture specific dependencies (and the deprecated policy-forbidden packages without architecture) not so much anymore. It is to be noted that not all architecture specific dependencies actually trigger this, they just have the potential to do it, if they do depends very much on the order of things, but we will get to that in a minute. Analysing amd64 sid+experimental you can find the following effected: libstdc++6-4.9-dbg arm64 libgcc-4.9-dev arm64 libgcc1-dbg arm64 libgomp1-dbg arm64 libitm1-dbg arm64 libatomic1-dbg arm64 libasan1-dbg arm64 liblsan0-dbg arm64 libtsan0-dbg arm64 libubsan0-dbg arm64 libcilkrts5-dbg arm64 libquadmath-dbg arm64 libgfortran-4.9-dev arm64 libgfortran3-dbg arm64 u-boot armhf (the second column is the architecture the first package is created for, which should have been amd64 if the made assumption would be valid). We have seen the names of most of them in this bugreport already, Axel used libgcc1-dbg in his example. The situation is a bit different for jessie sources: Only u-boot is effected here. u-boot:amd64 doesn't exist btw. (The reason is that crossbuild-* doesn't exist there). Note that this error doesn't add up, but can decrease with additional sources. Having testing+sid+experimental enabled e.g. leaves you with only two of them still on the list (if you have it in this exact order). Architectures aren't created equally either, but they are all playing in the same ballpark, so list isn't changing much if at all between archs. Bug #782889 mentions other packages effected on jessie through, and Axel has an example with libgnutls26 on an amd64 with sid+experimental, so how does these happen? It took me longer to realize than it should have, but it is actually quite simple now that I had a good 'talk' with my pillow: In my testing I was using either only the component "main" or the components "main contrib non-free" (in that order), but if you happen to do (lets say) "contrib main" you end up with these in addition: libldap-2.4-2 i386 libdb5.3 i386 libgnutls26 i386 libsasl2-2 i386 libsasl2-modules i386 libsasl2-modules-db i386 Which minus the libgnutls26 are also exactly the packages #782889 mentions for jessie (libgnutls26 isn't existing nor even referenced in this release, so the package comes never into existence for apt here). [and they are the only packages mentioned by the bugreport, so not all jessie users are effected at all – see also "workaround" 1.] So, what is the effect you might ask now: There are more or less four ways to get to a package in libapt. 1. You can follow a dependency. There is no lookup involved here, this will always point to the correct package as this one is prebuild with 2. 2. You get a Grp-structure, which has a FindPkg(arch) method giving you the package you desire. 3. You iterate over ALL packages one after the other (that sounds ridiculous stupid, but is done a lot as you usually have to touch them all in some way anyway) 4. You are asking the cache directly with a FindPkg(name[, arch]) call for a package. Only the last one is effected as this path includes a detection for single-arch which then has the incorrect assumption of the first package being the native package. The attached patch removes the single-arch special casing in this codepath, which means that the same code as on multi-arch is used which will in turn use 2. to get the package. apt itself it turns out uses way 4 very seldom as its dealing a lot with Grps nowadays and hence does 2 (if not the other two), so, even through the output in simulation is slightly off you can interact with these packages quite normally (Axel e.g. showed already that he can install such a package). Upgrades and such will work, too. aptitude and other frontends potentially use 4 more as most of them weren't changed much since the advent of Multi-Arch (and 2 is a thing only since Multi-Arch), so I suspect interactive actions like in aptitude the marking where one of these packages is picked specifically to be effected, while non-interactive actions like upgrades where the package is picked via dependencies (1) or whole system operations (3) aren't. The worst thing which can happen is that effected packages can't be installed with (e.g.) aptitude, not forgetting new or forgetting auto (also with apt) while being slightly annoying isn't dangerous. I want to highlight especially that it isn't possible to produce a broken system with this as dependencies are created and evaluated correctly. ---- [ end of (most of) the techno-babble ] ---- Effected users have a cornucopia of options to deal with this, which brings in other small benefits as well, so users might want to try: 1. Reorder components in the sources.list if you have activated multiple, which means if you currently have e.g.: deb http://http.debian.net/debian/ jessie contrib non-free main change it to deb http://http.debian.net/debian/ jessie main contrib non-free For jessie users this will trim down the list to 'u-boot' mentioned above, which as said has no native counterpart, and is therefore handled (= ignored) correctly already, so this should make this "problem" go away entirely already for jessie users without any downside. (and is hence the reason why I am not crying RC-BUG and pull the emergency brake just yet. We can fix that in a point-release.) 2. Reorder sources.list lines: man sources.list defines that the order of these lines effect from which source a package is pulled if multiple provide the same package version. Using here a pattern of (omitting the ones you don't want of course) oldstable stable testing unstable experimental security, -updates, -backports, third-parties, … instead of the other way around allows for e.g. download progress output to be better as it will report that a package which already migrated from unstable to testing is downloaded from testing, rather than claiming it comes from unstable (which can be confusing if you happen to have pinned down unstable, but your package manager seems to install packages from it anyhow… Its also good e.g. for the security mirrors to avoid downloading from them if your usual mirror has the update already so they can serve others in need instead). This should help at least a bit in reducing the list as the cache is filled with most packages already before apt encounters the rather new architecture-specific dependencies in newer releases (see my example above - the remaining two were our friend u-boot and libquadmath-dbg, which is another non-existing package, so problem gone also here, which explains why some systems with the 'same' sources are effected while others aren't; order matters quite a bit sometimes…) 3. Ignore it for the time being Especially if you have opted for the two previous options, the still effected packages list should be small if not empty and include mostly packages most users do not care too much about in the first place (*-dev and *-dbg). So if you don't have a pressing need, an itchy trigger finger or aren't on an at least immediate Debian experience level sitting that one out until a point release fixes this is a very valid option instead of rushing further down the list. 4. Enable Multi-Arch This makes the problem go away entirely, but with all the advantages and disadvantages Multi-Arch itself might have for you. (In effect, it triggers the codepaths being used as the patch does) 5. Adventure seekers can try the attached patch. Axel already tried it out for me and confirmed it fixes things and my tests are reassuring as well, but additional feedback never hurts (still, this is for the experienced Debian user to try only!) A "thank you" goes out to everyone reporting this bug and especially to Axel for helping me debug this – and of course a "sorry" to everyone for breaking things, especially to the release team. Best regards David Kalnischkies
commit 6c9937da76b9155d166092b9dda22d06200510c1
Author: David Kalnischkies <da...@kalnischkies.de>
Date: Wed Apr 22 12:06:40 2015 +0200
remove "first package seen is native package" assumption
The fix for #777760 causes packages of foreign (and the native)
architectures, to be created correctly, but invalidates (like the
previously existing, but policy-forbidden architecture-less packages
we had to support for some upgrade scenarios) the assumption that the
first (and only) package in the cache for a single architecture system
must be the package for the native architecture (as, where should the
other architectures come from, right? Wrong.).
Depending on the order of parsing sources more or less packages can be
effected by this. The effects are strange (for apt it mostly effects
simulation/debug output, but also apt-mark on these specific packages),
which complicates debugging, but relatively harmless if understood as
most actions do not need direct named access to packages.
The problem is fixed by removing the single-arch special casing in the
paths who had them (Cache.FindPkg), so they use the same code as
multi-arch systems, which use them as a wrapper for Grp.FindPkg.
Note that single-arch system code was using Grp.FindPkg before as well
if a Grp structure was handily available, so we don't introduce new
untested code here: We remove more brittle special cases which are less
tested instead (this was planed to be done for Stretch anyhow).
Note further that the method with the assumption itself isn't fixed. As
it is a private method I opted for declaring it deprecated instead and
remove all its call positions. As it is private no-one can call this
method legally (thanks to how c++ works by default its still an exported
symbol through) and fixing it basically means reimplementing code we
already have in Grp.FindPkg.
Removing rather than fixing seems hence like a good solution.
Closes: 782777
Thanks: Axel Beckert for testing
diff --git a/apt-pkg/pkgcache.cc b/apt-pkg/pkgcache.cc
index d7c9656..a7b75da 100644
--- a/apt-pkg/pkgcache.cc
+++ b/apt-pkg/pkgcache.cc
@@ -230,12 +230,7 @@ pkgCache::PkgIterator pkgCache::SingleArchFindPkg(const string &Name)
pkgCache::PkgIterator pkgCache::FindPkg(const string &Name) {
size_t const found = Name.find(':');
if (found == string::npos)
- {
- if (MultiArchCache() == false)
- return SingleArchFindPkg(Name);
- else
- return FindPkg(Name, "native");
- }
+ return FindPkg(Name, "native");
string const Arch = Name.substr(found+1);
/* Beware: This is specialcased to handle pkg:any in dependencies as
these are linked to virtual pkg:any named packages with all archs.
@@ -249,13 +244,6 @@ pkgCache::PkgIterator pkgCache::FindPkg(const string &Name) {
// ---------------------------------------------------------------------
/* Returns 0 on error, pointer to the package otherwise */
pkgCache::PkgIterator pkgCache::FindPkg(const string &Name, string const &Arch) {
- if (MultiArchCache() == false && Arch != "none") {
- if (Arch == "native" || Arch == "all" || Arch == "any" ||
- Arch == NativeArch())
- return SingleArchFindPkg(Name);
- else
- return PkgIterator(*this,0);
- }
/* We make a detour via the GrpIterator here as
on a multi-arch environment a group is easier to
find than a package (less entries in the buckets) */
diff --git a/apt-pkg/pkgcache.h b/apt-pkg/pkgcache.h
index 5e8a963..a7e520b 100644
--- a/apt-pkg/pkgcache.h
+++ b/apt-pkg/pkgcache.h
@@ -218,7 +218,7 @@ class pkgCache /*{{{*/
private:
bool MultiArchEnabled;
- PkgIterator SingleArchFindPkg(const std::string &Name);
+ APT_DEPRECATED PkgIterator SingleArchFindPkg(const std::string &Name);
};
/*}}}*/
// Header structure /*{{{*/
diff --git a/test/integration/test-bug-782777-single-arch-weirdness b/test/integration/test-bug-782777-single-arch-weirdness
new file mode 100755
index 0000000..0049033
--- /dev/null
+++ b/test/integration/test-bug-782777-single-arch-weirdness
@@ -0,0 +1,72 @@
+#!/bin/sh
+# Ensure that the order in which packages are in the binary cache
+# does not effect if they can be found or not
+set -e
+
+TESTDIR=$(readlink -f $(dirname $0))
+. $TESTDIR/framework
+setupenvironment
+configarchitecture 'i386'
+
+insertpackage 'unstable' 'abar' 'i386' '1'
+insertpackage 'unstable' 'foobar' 'i386' '1' 'Depends: abar:amd64, zfoo:amd64'
+insertpackage 'unstable' 'zfoo' 'i386' '1'
+
+setupaptarchive
+
+testrun() {
+ rm -f rootdir/var/lib/apt/extended_states
+
+ testequal 'Reading package lists...
+Building dependency tree...
+The following NEW packages will be installed:
+ abar zfoo
+0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
+Inst abar (1 unstable [i386])
+Inst zfoo (1 unstable [i386])
+Conf abar (1 unstable [i386])
+Conf zfoo (1 unstable [i386])' aptget install abar zfoo -s
+
+ testequal 'Reading package lists...
+Building dependency tree...
+The following NEW packages will be installed:
+ abar zfoo
+0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
+Inst abar (1 unstable [i386])
+Inst zfoo (1 unstable [i386])
+Conf abar (1 unstable [i386])
+Conf zfoo (1 unstable [i386])' aptget install abar:i386 zfoo:i386 -s
+
+ testequal "Reading package lists...
+Building dependency tree...
+Package abar:amd64 is not available, but is referred to by another package.
+This may mean that the package is missing, has been obsoleted, or
+is only available from another source
+
+Package zfoo:amd64 is not available, but is referred to by another package.
+This may mean that the package is missing, has been obsoleted, or
+is only available from another source
+
+E: Package 'abar:amd64' has no installation candidate
+E: Package 'zfoo:amd64' has no installation candidate" aptget install abar:amd64 zfoo:amd64 -s
+
+ cp -f rootdir/var/lib/dpkg/status status.backup
+ insertinstalledpackage 'abar' 'i386' '1'
+ insertinstalledpackage 'zfoo' 'i386' '1'
+
+ testequal 'abar
+zfoo' aptmark showmanual abar zfoo
+ testequal 'abar set to automatically installed.
+zfoo set to automatically installed.' aptmark auto abar zfoo
+ testempty aptmark showmanual abar zfoo
+ testequal 'abar
+zfoo' aptmark showauto abar zfoo
+
+ mv -f status.backup rootdir/var/lib/dpkg/status
+}
+
+msgmsg 'Single-Arch testrun'
+testrun
+msgmsg 'Multi-Arch testrun'
+configarchitecture 'i386' 'amd64'
+testrun
signature.asc
Description: Digital signature
