Frank Peters <frank.pet...@comcast.net> posted 20090615222731.b12f5121.frank.pet...@comcast.net, excerpted below, on Mon, 15 Jun 2009 22:27:31 -0400:
> In a lot of cases, for example perl, Xorg, and gcc, the Gentoo > distribution lags far behind the latest available releases. Even > allowing the "~amd64" unstable series, this remains true. Why is this > so? Others posted their take. Here's mine. I don't know enough about perl to have a reasonable opinion there, but I besides using them, I follow the Gentoo dev list, and happen to know what's going on with the other two. Consider what a well tested and working gcc means to Gentoo, as opposed to what it means for your typical binary distribution. Really, that should be enough right there if you think about it, but let's just put it in writing. In a binary distribution, a gcc that fails to build particular packages is no big deal. In Gentoo, it's a *MAJOR* issue. In a binary distribution, a gcc that gets broken somehow is again, no big deal. In Gentoo, it's a *MAJOR* big deal, possibly necessitating a complicated recovery procedure involving unpacking a stage tarball somewhere and binpkging its gcc, to then install over the broken one on the main system. So, naturally, Gentoo's way more cautious about unmasking a particular gcc version. While micro-version changes (4.3.2 -> 4.3.3) are generally bug-fix only anyway and thus get moved thru the process pretty fast, normally for minor version changes (4.3 -> 4.4), the process is far slower and more complex. I regularly unmask and test new gcc versions, so I know from personal experience what it means and the hassle early users go thru. First, pre-release, weekly snapshots and definitely the -rcs are available in the overlays, for anyone wishing to go thru the hassle of testing them. Gentoo devs therefore get a bit of a head-start, knowing the trouble areas and often patching troublesome packages to build with the new gcc before it's even released. Then, when it's released upstream, the new gcc ebuild is usually in the tree, hard-masked for testing, the same day. Based on the pre-release overlay testing, they know the ebuild itself generally works by then, and barring unusual issues, gcc itself should compile and install from the ebuild. However, at release, there's dozens of packages that have yet to be updated to be able to actually build with the new gcc. Many of them require patches. Others require restricting certain CFLAGS or the like. A gcc upgrade tracker bug is normally open by then, with a list of all the individual package bugs. As patches are developed and applied (at this point, to ~arch packages), the bugs are resolved, and the list of unresolved bugs on the tracker shrinks. Meanwhile, keep in mind that another difference between Gentoo and binary distributions is that Gentoo effectively ships often tens, sometimes hundreds, of different versions of the same package, due to USE flags. Obviously, even testing every package with a late gcc -rc version isn't going to catch all the bugs, because each package is effectively many slightly different packages, due to USE flag combinations, and it's simply impossible to test them all, period. Thus, after release, users like me start testing, and often discover NEW ways in which a particular package breaks with the new gcc, filing even MORE bugs to add to the tracker, and eventually get fixed. The binary distributions really have it easy in this regard, as they ship one, perhaps two versions of a package, with various bits enabled or disabled, take it or leave it, or compile it yourself, in which case they do NOT provide support, while Gentoo does. So eventually an ~arch version of most packages has been patched and tested to work (under at least some configurations) with the new gcc, and it's considered on the way to stable, enough to unmask it to ~arch. Then the process not quite, but almost, starts over again. Not quite because at least they have the list of packages and known ways to fix them, now. Normal package stabilization qualification is 30-days in ~arch without an open bug, at which point a package maintainer opens a keyword stabilization bug, asking the various archs to test and keyword stable. (Note that it's not the package maintainer that marks stable, it's the arch devs and their ATs, who test it, marking it stable only if it works with an all-stable system config. Also note that while Gentoo does have package.keywords as a method to run a partially unstable system, such a system is /not/ tested, and thus is far more likely to run into issues than an all-stable or all-unstable system.) However, with a toolchain package like gcc, the process is far more complex than that. Remember that list of typically a hundred packages or more that had to be patched to build with the new gcc? NOW all those packages have to make it to stable! Only when there's a stable version of every one of those packages in the list that can build with the new gcc version, can that gcc version /itself/ move to stable! /Now/ you should be /beginning/ to appreciate just how complex the journey of a new gcc version is, from pre-release testing in the overlay, to release onward masked testing in the tree, to unmasking to ~arch, to final stabilization, along with /hundreds/ of new versions of other packages with the required patches to build with the new gcc. That's a tough order to follow, and it's really amazing that the process works as well as it does. But, let's also consider what happens from a user perspective when a new gcc is setup on the system, and what it might mean if it was done too early and broke some packages, therefore requiring keeping both it and the older version around. When a new gcc is installed and people start to use it to compile packages, a lot of things can break, until a full emerge --emptytree @system @world is performed. Again, as a new gcc tester, I'm rather well acquainted with this. But if you've done such an upgrade and removed the old gcc without doing that --emptytree rebuild, you've probably seen at least the most infamous problem, the libstdc++ errors due to all those *.la libtool files pointing at the old gcc libstdc++ location instead of the new one. For that there's the fix_libtool_files.sh script, and it does a /remarkably/ good job. However, consider those guys testing those weekly gcc snapshots in the overlay! How many times are they going to run fix_libtool_files.sh to point to the new weekly snapshot? Meanwhile, what about all those packages that break with the new gcc and don't have patches applied yet? Consider for example, kde4. kde4 depends on cmake to build. cmake is gcc version sensitive. Even when it builds against a new gcc, that means it won't work with the old one. Thus, while ordinarily if a package won't build with the new gcc, you can gcc-config back to the old one and rebuild it that way, that doesn't work for kde4, because now you must rebuild cmake, and often several other dependencies as well, just to build that one kde4 package that won't build with the new gcc. Then you gcc-config back to the new version, and have to rebuild cmake and possibly other dependencies /again/ to continue with the other packages. Sometimes the dependencies just can't be made to fit together in a working way, and a user must give up, and remerge a significant part of his system with the old gcc again, until a few more patches are available. Of course, particularly those who unmask new gcc versions that aren't even in ~arch yet but to some extent ~arch users as well, must get used to searching for bugs on packages that won't compile with the new gcc, to see if there's a patch there, that hasn't been actually applied to a package yet. Sometimes they have to go looking elsewhere as well, Debian, Fedora, etc, for patches that they may have that haven't made it to Gentoo bugs, just yet. All that's a part of running a new gcc version, and why it takes so long to get to stable. Now, all this is enough of a problem for someone who deliberately unmasked the new gcc for testing, before it was even ~arch, or even, to a lessor extent as there are fewer issues by then, for ~arch users. But consider the protest stable users would (justifiably) raise if they were subjected to this! It's bad enough that if they follow the recommendation, they have to emerge --emptytree @system @world. If they choose not to do that, they must at least run fix_libtool_files.sh, or c+ + apps quit compiling. And we have enough users (and the bugs they file) that don't even know how to do that, or complain about having to do so. NOW consider those SAME users having to deal with all the stuff the early testers and ~arch users deal with, if a gcc version is keyworded stable before its time! As you can see, life's a LOT more difficult for Gentoo devs when a new gcc version comes out than it is for your typical binary distribution. If anything, it's actually a wonder that gcc versions get stabilized as fast as they do, and that we're not still back on gcc-3.something. The problem with xorg is rather different, and much simpler to explain. Basically, it usually comes down to all those proprietaryware graphics drivers users. As with most distributions, Gentoo is hesitant to stabilize a particular xorg until the users of the various graphics cards using proprietaryware drivers have a supported graphics driver they can upgrade to as well. As the quote in my sig describes, the proprietaryware driver vendors really do in this case serve as xorg masters, holding it back otherwise unnecessarily. Then for xorg in particular, there's the whole configuration change thing going. Before a version of xorg with major config changes goes stable, an upgrade guide must be created, detailing how to deal with the config changes along with any other issues that arise. As others have mentioned, docs, formerly an area where Gentoo shined and still an area where they're reasonably good (how many other distributions cover upgrade config changes in such detail, not many!), has been dragging a bit behind, lately. Finally, because X is such a major package, there's also a bit of the same issues to deal with that I described for gcc above, but nowhere near to the same level, and at least for stable, as I said, it's very often the proprietaryware driver vendors holding things up, and when it's not, it's often the development of the upgrade docs. > I had first considered moving to Gentoo in the fall of 2008, but after > noticing that the only version of gcc available at that time was > gcc-3.x, I postponed the change. In the spring of 2009, Gentoo finally > moved up to gcc-4.3.x and then I made the transition. But the update to > the 4.3 series was a long time in coming. The above covers that in great detail. Also, again as others have noted and as I alluded to above, there are in fact snapshot and -rc ebuilds available from long before a gcc version is ever released. But for the reasons covered above, they aren't in the tree prerelease, and when they do hit the tree at upstream release, they're first hard masked, then unmasked to ~arch, thru an extremely complex process that takes quite sometime. But they're certainly available for those brave/hardy/stupid enough to try them! > The latest perl, released some time ago, is version 5.10 but Gentoo > includes only 5.8.8. As I said, no comment there, as I simply don't know enough about it to have an opinion, educated or not. > The latest Xorg has restructured certain libxcb dependencies, which has > caused a lot of problems for a lot of packages, and Gentoo is behind > these changes as well. > > (Ironically, it was this libxcb issue as well as the whole Xorg > modularity mess that first motivated me to seek out Gentoo.) Actually, libxcb was in fact one of the gcc-like complexity issues of xorg. In this case, previous versions had a particular library that many X based packages built against, that was no longer available in the newer version. For those running LDFLAGS=--as-needed (as I do), the problem wasn't all that bad. However, for those NOT running with that flag (which does break some packages, and used to break a LOT more, before flameeyes started working on the problem), the easiest way around the problem was to recompile everything that broke -- in a specific order that portage didn't always manage properly -- and there were a LOT of package in this broken list. So, stabilization (and indeed, the move from the overlay to the tree and ~arch) was delayed while they worked out and tested a solution. There's now a script that can be run that takes care of most of the problems, without a recompile. Again, it's a libtool *.la file issue. Then that solution had to be documented... Yes, some binary-only distributions beat us to it, but again, all they have to worry about is getting the compilation correct on the single version of each package they normally provide (for each arch). They don't have to worry about the recompile issue that was Gentoo's big holdup here for xorg/libxcb, or pretty much all the time for gcc. > Now I am not actually voicing a complaint. Gentoo, IMO, is still the > best distribution for Linux. I am just wondering why there is such a > great lag before a package version is deemed stable -- or even unstable. > In my experience with maintaining my own Linux system, I never had any > great issues with always installing the latest "bleeding" edge software. I'm glad you're not complaining, and actually, I understand why someone not acquainted with the particular issues a from-source distribution deals with, particularly a distribution used by and depended on by as many people as Gentoo... I can well understand how they might find that Gentoo looks slower on the uptake than it should be. But once you understand what's going on behind the scenes, and how much work Gentoo devs, ATs and ~arch users put in to ensure a particular package is really stable before it's keyworded that way (and why even with all that work sometimes things slip thru the cracks), the mystery then becomes much more one of how in the world it all gets done, and how in the world it continues to function as well as it does! Of course, if Gentoo were to even do something as simple as do away with most USE flags, cutting the number of possible package variables accordingly, it would go a HUGE way toward eliminating all that extra work and all those extra bugs from all the variations. Similarly, Gentoo could do away with the CFLAGS etc settings, or just filter them for most packages, again cutting down on the number of bugs tremendously. Gentoo's offering would then be far closer to what the binary distributions offer. Then, since we'd have just done away with pretty much the entire set of advantages of building from source, we could do away with the from-source entirely, and just ship the binaries, bringing us into pretty much direct comparison with the binary distributions. Of course, that would kill the whole reason Gentoo's so great to run for many of us. We LIKE it's customizability, even if it means thinner testing and thus dealing with a few extra bugs and longer stabilization cycles, and even if it does mean waiting a bit for packages to compile! Meanwhile, it should also be noted that "in my experience with maintaining my own Linux system", doesn't quite compare to dealing with THOUSANDS of people running one or more systems each, many with entirely DIFFERENT configurations, and the headaches that can mean, trying to get something working on ALL of them at once, without breaking the obscure and rather strange configuration that guy off in the corner with even more customization than usual (umm... that'd be me! see the make.conf I posted a few days ago for just one example of "strange" and "even more customized than usual" =:^) is running. Yes, that means even if that system one was maintaining was a Linux from Scratch system. (I've not done Linux from Scratch yet and probably never will, because Gentoo's the perfect balance of customization and automation for me and I discovered Gentoo before Linux from Scratch, but a lot of Gentoo users have.) -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
