Martin Herrman <mar...@herrman.nl> posted 40bb8d3b0906070049g26add8e4s78bfa9da8285b...@mail.gmail.com, excerpted below, on Sun, 07 Jun 2009 09:49:28 +0200:
>> Note that newer versions of portage have /automatic/ blocker resolution >> in many temporary cases, so depending on how old your portage is (of >> course I'm on ~amd64 and have had 3.5.10 for months now, and don't know >> if the feature is in stable portage yet), if at the --ask prompt, you >> tell it to go ahead, it may well take care of everything for you >> without you having to do anything else. =:^) > > > in which version was this feature added? I just upgraded portage to > 2.1.6.13 and could upgrade to 2.2.x, but that one is hard masked: > > http://www.gentoo-portage.com/sys-apps/portage What happened there is that while 2.1.4 was current stable, 2.2 as the next presumed stable series, with several new features, was made ~arch. However, some of those features matured faster than others. Among them, parallel merges (see below) and automatic blocker resolution (again, see below). Others such as sets (you'll see my mention of @system and @world below, tho I don't spend time on it or explain, see Zac's blog linked below for more), and FEATURES=preserved-libs (nice idea in theory but was breaking stuff due to originally poor implementation so I turned it off, I'm not sure the current status) didn't mature quite as fast. So Zac and the other portage folks decided to release an in-between series starting with (IIRC) 2.1.5, that contained the more stable new features from 2.2 (parallel merges and auto-blocker-resolution), while omitting the ones that needed more work (sets, preserved-libs). The idea was to have the new features that could go stable much faster. However, that meant that they needed testers for the 2.1.5 series, that they didn't have, since the usual testing group, ~arch users, were alreadly on 2.2. So they hard-masked 2.2 so most ~arch users would fall back to the new 2.1.5 series, thus testing it well enough so it could be stabilized. That's the status we have today. 2.2 is ~arch ready and in fact was ~arch for a short period, until they decided they needed testers for the new middle ground 2.1.5 series. But, there was a problem. With 2.2 going ~arch, KDE moved KDE 4.2 (or was it still 4.1 at the time?), to that point either out of tree or hard- masked, to ~arch as well. The problem is that the official KDE-4 migration guide referred to and continues to refer to sets, a 2.2-only feature at this point. So anyone who was using or testing ~arch KDE 4.2 and had followed the official migration guide using sets, had to either unmask portage-2.2 when it went back hard-masked, or fiddle around trying to figure out how to convert their previous set-merged kde-4.2 into a non- set config. The Gentoo/KDE-4 Migration Guide http://www.gentoo.org/proj/en/desktop/kde/kde4-guide.xml FWIW, while I'm still using KDE 3.5.10 as my "production" desktop, I've been testing 4.x since before the original release. (FWIW, KDE-4 still has problems in 4.2.x compared to later 3.5.x versions, at least for those with older video cards like the Radeon 9200 I'm still running, but it's /slowly/ getting there, and the later 4.2.x versions should be decently usable for many with newer video -- or smaller displays, I'm running 1920x2400, dual-stacked 1920x1200, and the Radeon 9200's OpenGL maxes at 2048x2048, thus the issue here, as KDE-4 really needs OpenGL to be worth the upgrade from KDE-3.) Thus, I had and have 4.2.x installed as well -- using the upgrade guide and sets -- and had to unmask portage-2.2 when it was masked to encourage ~arch users to test the new 2.1.5 series. > I will now just give it a try: start to unmerge the blocked packages and > update my system. We'll see :-) Back to the blocked packages thing, versions, etc... Note that with --ask (or --pretend), portage will still display the blocks. It'll just manage them automatically if you say "y" at the --ask prompt (or redo the emerge without the --pretend). That way, users can see any potential for temporary "dependency gaps" as I mentioned and can use their judgment and not try to do them right before that ever so important presentation or whatever. I was unsure of the precise version, but I did a quick bug search and found a comment from Zac Medico (portage maintainer and release manager) indicating that for the 2.1 series, it was added in 2.1.5. http://bugs.gentoo.org/show_bug.cgi?id=224487#c1 As he says there: >> With automatic blocker resolution like this, it should >> be relatively rare for a user to encounter a blocker >> that requires any manual intervention. That comment references the blog entry where he explains the idea, here (link wrap warning, but you can follow the link from the comment linked above): http://blogs.gentoo.org/zmedico/2008/05/09/ blocking_package_file_collisions Zac's answers to some of the comments are interesting/informative/helpful too. Note that portage's blocker resolution has gotten slightly more complex since then. In particular, in ebuilds where !<pkg-atom> originally indicated a blocker, that's still used if it's a normal blocker that it's OK for portage to try to resolve automatically, while !!<pkg-atom> is now used for complex blockers where portage must not try to resolve it automatically because something's likely to go wrong. Also, as noted in the blog, portage is more cautious about system packages and its own dependencies, where doing the wrong thing could leave the system temporarily unbootable or could kill portage itself, thus making recovery much more problematic. Also likely of interest if you've not paid too much attention recently, and have a multi-cpu/multi-core system with a decent amount (say a gig a core) of memory is the multiple package merges in parallel stuff, see the "next" link at the blog. Matter of fact, browsing around the whole blog and checking back for newer entries are both likely to be quite informative/helpful, as well. http://blogs.gentoo.org/zmedico/2008/07/23/portage_parallel_builds I've been using the parallel jobs feature for some time now, and have with experience developed some hints. First, as I said, it doesn't make a lot of sense to use it for single cpu/core systems, or for those sufficiently memory constrained (say < 1-gig/core) that resources are already stretched, and may in fact increase merge times due to increased swap usage or the like. Second, --keep-going is quite nice. =:^) That way, when something fails, portage continues other presently going merge jobs and then regenerates its dependency graph without the failed package and tries again. Thus, it installs as much as it can before failing with what it couldn't (and anything depending on what failed), leaving far less to have to merge later, after the issue with the failed package is resolved. As for --jobs and --load-average, here's what I found. First, you probably want to make these somewhat lower than the parallel MAKEOPTS settings. (This assumes that you DO use both --load-average here and -l in MAKEOPTS, if you just use --jobs and -j, adjust accordingly.) Doing so means the system will first try to run more parallel jobs on already running merges if it can, before the load drops to the point that it starts on additional package merges as well. This helps limit memory and other resource usage, particularly for those (like me) that have PORTAGE_TMPDIR pointing at a tmpfs to speed emerges and minimize disk access. Second, because the unpack and configure steps are by nature not very parallelized, what tends to happen is that in parallel mode, at first, portage will start working on one package, decide the load average is still low enough to do another... and another... and another... before any of them actually get to the more parallelized compile phase. Also, at times, there will be a bottleneck package which portage must wait on because all further packages depend on it and must therefore wait until it's installed before they can be merged. These two factors taken together tend to mean that in practice, the emerges clump up, portage will start several at a time before the load average gets high enough that it doesn't start any more, then (even with a higher MAKEOPTS setting than portage setting) when they hit the actual compile phase, it'll cap at the MAKEOPTS limit for quite some time and not start anything else until all or all but one of the packages are finished, then it'll start several more... etc. Third, and this is what took the most getting used to here, since several packages are merging in parallel, it can't show the whole output for each one as it used to. Instead, it shows a summary that shows when it starts working on a package, when it starts installing a package, how many it has completed, and how many total. It's actually pretty nice, but it did take some getting used to. Meanwhile, it's still logging the actual merges in the background, and if there's an error, it spits that info out and tells you where to look for the log. Also, if you want to track the progress of an individual package, you can use the tradition Unix tail -f command in another terminal or whatever. What I've done here is set it up so that my @system and @world update helper scriptlets (example: eaw for emerge -aNuD @world) always use the parallel merge options while ordinary single-package emerge helper scriptlets (example: ea for emerge -a1) don't do the parallel thing. I do however have a parallel variant scriptlet (eam, the m for multiplexed) that does. That way, I can see the full output when I'm merging individual packages, but for big jobs such as @world updates or if I'm upgrading KDE, I use the parallel emerges functionality. I thus do NOT have portage's --jobs, etc options set in EMERGE_DEFAULT_OPTS, because I set it on the emerge command line as fed to it by the appropriate scriptlet. One caveat! Some packages require a LOT of memory for one particular job. KDE-3's kmail (for monolithic package users I believe it's in kdepim) is one example, using over a gig of memory for a single job at one point in its compile, at least here. There are others. Also, while portage honors its --load-average option, not all packages honor the -l in makeopts. At least parts of qt-4 are this way. That can be bad when it's only a single package emerging at once, but it's made much worse if you've not allowed for it setting the portage parallel merge options and portage has several merges going at once, at the same time. I've tended to handle these as I find them by setting a lower MAKEOPTS=-j option (while eliminating the -l they don't honor anyway) for them in the appropriate /etc/portage/env/cat-egory/pkg file, thus overruling the standard/global make.conf setting. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
