[Most of this was written a few weeks ago when this bug was first filed, but I didn't post it then as I wanted to get better docs on the compiler builds in place at https://wiki.debian.org/ToolChain/Cross so that the substantive argument was easier to follow for those not previously involved. That could still do with more work, but it's time to actually reply, so sending. Some of this has since also been said in 766708 ]
Matthias Klose <d...@debian.org> wrote: > Recently some crippled cross toolchain packages were uploaded to > Debian unstable, I do wish you'd stop calling these toolchains 'crippled' every time you refer to them. They are the only (so far), and thus 'best', cross-toolchains uploaded to debian itself and meet the requirements of most users. In fact there are only two people who have expressed any dissatisfaction with them so far. You and Dmitry. Quite a few other people have been very pleased to see them. I do not claim that they are perfect, and expect there to be changes for Stretch, but they are serviceable and working and maintainable (if you don't actively block that maintenance). > without having any consent with the Debian GCC maintainers, nor > announcing these packages anywhere. As you know I have been working on Debian-compatible cross-toolchains since about 2004 on and off. I have discussed them many times with you personally at Connect, Debconf, given talks about them, and of course talked to many other people about them too. There was a thread with a pretty good summary of our basic disagreement here: https://lists.debian.org/debian-devel/2013/01/msg00375.html Most recently we discussed this again at the Bootstrap/Crossbuild Sprint in Paris. In all that time we have never agreed about the best way of doing it, and that disagreement has delayed any upload for the whole of wheezy. No-one uploaded packages using your preferred method in the 4 years they have been used on Ubuntu. I wanted to get some cross-toolchains in Jessie (https://wiki.debian.org/ReleaseGoals/CrossToolchains created Sept 2013) I spent quite some time trying to get the ubuntu cross-toochain-base 'full-bootstrap' packages to build on debian earlier this year but failed. I tried again at Connect in September - still not working. After a couple of weeks I decided it was easier to make the sbuild and wanna-build changes to enable multiarch-built cross-toochains than to fix that mess. So that's what I did (with much help from Dima) so now we do at least have some cross-toolchains. > Maintained by a single person, not even trying to form a maintainer group, There is a cross-toolchain maintainer group on Alioth: https://alioth.debian.org/projects/crosstoolchain/ It has existed for a long time, but was re-invigorated at Debconf13 where there were productive cross-toolchain discussions and a clear desire for the above release goal of toolchains in the archive. All the code for the recently uploaded toolchains is there: http://anonscm.debian.org/cgit/crosstoolchain/ (and other packages currently in development, like gcc-cross-support) All those packages are now team-maintained. A few people have been actively working on cross-toolchains over the last 18 months. (Me, Daniel Grosser, Dima Kogan, Helmut Grohne on the linux toolchains, and Keith Packard, Augustin Henze and Thomas Preudhomme on the bare-metal toolchains). There is a team of at least 3 people who have committed to maintaining the existing cross-toolchains (Me, Dima Kogan, Helmut Grohne). I expect that a few more will help now that it has a higher profile in the archive, and indeed YunQiang Su has been helping out recently with mips and multilib changes . You want to replace this team, but who with? > and downgrading bug reports without addressing concerns of the GCC > maintainers. You filed (#766619, and 5 others for other arches) as 'grave' the fact that the cross-toolchain (providing cpp, gcc, g++ anf gfortran) did not also provide gccgo and gobj. No-one has ever provided crosstoolchains for Debian with those frontends enabled (TTBOMK), so to claim that their abscence is a 'grave' bug seems, well, wrong. So I downgraded it to normal (not even 'wishlist' which is arguably more appropriate). No-one else has asked for these front-ends, and clearly they are not needed for the vast majority of packages. Nevertheless they are quite easy to provide, there is a branch in the cross-gcc git repo that adds those, and I have uploaded them to the cross-toolchain test repo at: http://emdebian.org/tools/debian/. I have not uploaded these to the archive yet because they would have to go through NEW. More front-ends is a fairly minor issue, and the only limitation is how many (each with corresponding meta-package from cross-gcc-defaults) do we want in the archive? Yes. There could have been better communication. However I know that any postings are likely to get an agressive, dismissive, response from you. That is a great way to discourage talking to you. Everytime I think about posting to debian-devel or cross-toolchain mailing list I know to expect some abuse about 'crippled' compilers, and make some time to refute it. It's much easier to just get on with some productive work instead, so I admit that I have discussed on mailing lists less than is ideal, to avoid conflict. (There has been plenty of IRC discussion within the team). But the idea that it is a big surprise to you make no sense: I've been following through on the same stuff for a decade. I hereby commit to continue this debate in public until we reach some kind of accomodation, or you succeed in having me declared an incompetent and unsuitable maintainer. (I'm very happy not to be the gcc cross-toolchain maintainer. It's a pile of work to keep up with, and you are not making it as much fun as it ought to be). Is anyone else vounteering? The wider issue =============== Clearly we do need to deal with this disagreement about how cross-compilers are built, and more generally about what the cross-toolchains should look like in the longer term. And maybe the technical committee is ultimately the right way to deal with that, however I think this referral may be premature. The toolchains have been available in the archive for about 6 weeks now, and from my point of view they are there so that people can test them, bugs can be found and we can see how it works in practice. Several have been found and fixed already. Two have been filed to go upstream into gcc: 760770, 770413. In our (my and doko's) discussions about this I have repeatedly had responses like 'you are doing it wrong', 'it won't work', but he has never been able to explain to my satisfaction _why_ it is 'wrong'. Perhaps in this bug report that will become clear. Doko did once say (Connect, Dublin) 'OK then, show me how it will work', so that is what I have been working on recently: making the packages in question, and using them, to see whether in fact there is some fundamental problem. Principles of cross-toolchains ============================== Mattias's framing of the basic principles as being 'can they build the current packaging of gcc and get functionaly the same as the native package build' seems somewhat arbitrary and circular to me. GCC is an unusual cross target, and this definition of 'can cross-gcc cross-build gcc' is circular, because it depends on how gcc is cross-built, and tangles up questions of multiarch and multilib when they should actually be separate issues. It _is_ useful to be able to cross-build gcc. It's a necessary part of a bootstrap, and we want to make debian bootstrappable. We both want that working, but it is not necessary to cross-build a multilibbed gcc in a bootstrap, just one targetting the architecture. And Bootstraping canbe done with or without wdotap, as Helmut has already shown. So, lets be clear that there are actually 2 separate matters of substance here: 1) Should cross-compilers be built against foreign-arch libc/kernel-headers, or not 2) Should cross-compilers be multilibbed, or not Adn there is a third question related to 1: 3) How are the libc packages the cross-toolchain is built against, built? There are quite a few other factors that come into this, but those are the 3 core questions, IMHO. And those questions are, in principle, independent. You could build a multilibbed cross-compiler against foreign-arch libc or a non-multilib cross-compiler using in-arch -cross libc. However, currently these questions get dragged together because not every combination is actually supported by current packaging. You will correct me if there is some fundamental reason why it would not be possible to make all four combinations? So, let us look at these three issues to first clarify what we are talking about, the technical details of them, and the pros/cons. To limit the length of this mail, I have tried to document the various build methods in: https://wiki.debian.org/ToolChain/Cross Please refer to that for background. MA-built vs in-arch ------------------- The cross-compiler build _does_ depend on a target-arch libc (or several for mulitlib builds). So the big questions are where should this(they) come from and how is it(are they) built? In the within-arch build the target-arch lib is installed in the 'classic compiler' location in /usr/triplet/. That works, but is the 'old' way of doing things. For all other cross-building we now recommend using multiarch locations for libraries/headers. Is there a good reason for treating gcc specially in this regard and keeping this old method around indefinitely, or should we be moving to the new paths? This was the thinking which caused me to persue building cross-toolchains aginst the multiarch paths. This has now been shown to work, and is comendably simple in packaging terms: cross- gccs are just built against their dependencies, like any other package. This is quick, simple, and the packaging is totally consistent across architectures. It seems to me to be a good thing that the default build for a cross-gcc-$arch in the archive is just a simple one-stage build as this is the normal case. Doing a whole bootstrap build every time to get a copy of a libc we already have available seems pointless. Obviously doing a bootstrap build when you do in fact need to bootstrap (because you are bootstraping, or the target arch is not in the archive) is perfectly reasonable. To make this (building against foreign-arch dependencies) work in the archive using existing packages does need tooling changes (sbuild, wanna-build, britney), which have mostly been done, but not entirely (britney still pending). The in-arch/supported build still needs the same target-arch library, but converted with dpkg-cross to the /usr/triplet/ paths. The dpkg-cross (and dpkg) maintainers think dpkg-cross should go away as multiarch should have made its path-conversion functionality unnecessary, and other useful functionality has already moved in to dpkg itself. The only useful feature it still has is the autoconf cache variables (which are planned to move into a cross-support package or similar). For gcc itself the final build is actually quite simple, and the only things that changes is whether the target-libc(s) is(are) in /usr/triplet/ or in /usr/lib/triplet/. The real interesting questions are about how those libcs are built and packaged. Are they natively-built or cross-built? Are they converted from foreign-arch packages with dpkg-cross or do we have some packages to make libc-$arch-cross packages directly in those locations? Is that a native or a cross build? The ubuntu cross-toolchain-base package is responsible for making libc-$arch-cross packages. It works by doing the whole cross-toolchain bootstrap from glibc and gcc sources. stage1 cross-gcc, stage1 cross-libc, stage2 cross-gcc built against stage1 cross-libc, stage2 cross-libc built with stage2 cross-gcc. Then dpkg-crossing the resulting cross-libc to put it into the 'classic toolchain' location. This seems like a lot of faff for arches where we already have a libc and you can replace all of that with Build-depends: libc6:$arch. Something like that _is_ needed for arches where the target-libc is not already built in debian. A halfway house would be to build-depend on libc6:$arch, but dpkg-cross that into libc-$arch-cross, rather than using it in-situ. That keeps the packaging simple. It would also be possible to get the libc packaging itself to spit out a set of -cross packages. The outcome of building a cross-gcc against a libc-$arch-cross in the 'classic toolchain' location is that as soon as you also install libc:$arch to actually build anything except a kernel against, you now have two libcs for $arch installed on both the 'old' and 'new' paths. This does not seem 'better' to me. If the -cross libc was dpkg-cross converted then it's the exact same files. I guess an interesting question is 'what does the cross-compiler actually _use_ the foreign arch libc for'? Does it need its own independent copy? What happens when the compiler libc-$arch-cross and the system libc:$arch get out of sync. Does it matter? Those packages (cross-toolchain-base) in Ubuntu are different for each supported arch and need/seem to be maintained separately. That doesn't seem very satisfactory. I am very keen that any solution we come up with is properly generalisable across architectures. multilib vs mulitarch --------------------- On the multiarch vs multilib matter, native compilers do not have the same requirements as cross-compilers. It seems to me to be sensible for Debian to use multiarch, rather than multilib, where it has a choice, and clearly cross-toolchains is one such place. Because cross-toolchains are co-installable it is simple to install gcc-arm-linux-gnueabi and gcc-arm-linux-gnueabihf if you want to target both of those ABIs, as opposed to building both of those compliers to target both, then installing either. Native compilers are not yet co-installable so have to use multilib to target more than one ABI. The main advantage of this for the cross-toolchains is that the builds are entirely consistent across architectures. As is shown by the current cross-gcc (source)/ cross-gcc-dev (binary) in expermental. One set of rules and patches works to build all the cross-gcc-4.9-$arch packages in the archive. Once you introduce multilib and the corresponding sets of 'sibling architectures' things get a lot more complicated, because there may be one, two, or three multilib siblings to deal with in deps for control, and the build rules. We may be able to get all this working in one consistent packaging, but I can't yet see how to do this. If we can't then you have to maintain each target-arch package separately, (as the cross-toolchain-base packages are in ubuntu), and that is a major minus-point. I'm not sure I want to maintain a set of cross-toolchain packages individually - I want to maintain them as a consistent, generated set. It should be the case that we can make MA-built cross-toolchains also support multilib, and indeed YunQiang Su says it works for him, and he has been submitting patches, but I have not managed to get it working myself yet (mostly due to tuits). Multilibs do make much better sense for arches that are not in the archive (x32/mipsn32) and are possibly the easier way to support those, but even there, all the difficulty is in getting the right libc:$arch or libc-$arch-cross packages. Once you have those you can build per-arch toolchains or multilib toolchains. There is some upstream expectation of multilib support, which is the best argument for enabling it. I don't think debian packages should be using it - they should use gcc-triplet thoughout (then they don't care how those triplets are packaged-up). We could use a gcc wrapper to recignise multilib options call the corresponding triplet-gcc. I beleive this was how guillems 'fake cross-toolchains' worked? Using multilib makes things irregular. See https://wiki.debian.org/ToolChain/Cross#Multiarch_vs_Multilib for details, but the point is that the options are all differrent depending which set of sibling-arches you are building. As opposed to just using <triplet>-gcc everywhere, which is nice and consistent. It makes packaging irregular too. But ultimately 'multilib or not' is an independent question from 'MA-built or not'. We should not get them unhelpfully mixed up. Current build status -------------------- Helmut has been working on getting the 'supported' cross-toolchain builds working, but the situation to-date is (ironically) that they only work if multilibs are disabled (on x86 and mips). #773266 #772665 There are also other issues which cause breakage with this build-method: (dpkg-cross abd glibc issues) #771497 #772045 #773265. So, as I say, we are just exploring now how these toolchains work in the archive, and if we have alternative builds working too then we can compare and contrast. We are getting there, but currently the MA-built, non-multilib toolchains are the only ones actually building consistently (because they are simplest). We have some working builds (varies with arch) of standalone+non-multilib and MA-built+multilib and standalone+multilib. The fact that both MAbuilt (wdotap) and non-multilib have been disabled in gcc-4.9 is not helpful, so everyone building cross-toolchains has to patch some or all of that back in to get things to actually built at the moment. The removal was premature, at best; one could be a lot ruder. Patched gcc sources ------------------- I have always been totally in favour of building the cross-toolchains directly from the gcc packaging with no extra patches. That was in fact working nicely until the maintainer uploaded gcc-4.9 4.9.1-19 which prompted the other tech-ctte cross-toolchains-related bug: 766708 Since then we have had to patch gcc in order to build cross-toolchains. Later gcc-4.9 4.9.2 (-4 particularly) uploads removed the rest of the build functionality we are using, so the cross-toolchain team has spent some time over the last month setting up a mechanism to maintian this outside gcc-4.9. No-one involved wants this extra work, and we'd all prefer it if this working functionality had not been removed from the gcc packaging. Integrating it originally was indeed a non-trivial piece of work, but leaving it there now is surely very little burden? This has moved the maintenance burden from the gcc maintainer to the cross-toolchain team, which is fair enough, but it's more work overall as this is easier to maintain in-situ than as a patchset. If the maintainer could revert that change then the current cross-toolchains would build patchless, and wouldn't have to have strong version-sync to a particular gcc upload (which is does curently due to the patches). So we are in full agreement on this point but the maintainer seems very strongly opposed to keeping it working patchless with the multiarch-build method. Packaging -------- This gets complicated and this mail is long enough already, but the existing (in Debian experimental/unstable) cross-gcc packaging is nice and simple. It is easy to maintain, and would be trivial if we didn't also have to maintain a large rules patch for the build machinery. It is not architecture-specific, which I think is important for long-term maintenance. So long as the libc build-deps are available it's a trivial packaging change to change whether it uses wdotap or not. It's a (much?) harder thing to enable multilib builds as those lack the consistency across arches that we currently rely on. And the thing about _not_ using wdotap is that you need to provide all the 'special' -cross packages rather than the standard packages (via multiarch) somehow, which is actually the fiddly bit. After thinking about it a fair amount it seems to me that this is in fact where the interesting packaging questions lie. Final remarks ------------- The gist of this bug is for the gcc maintainer to require that only complicated 'does everything' cross-toolchains should be permitted in the archive. Why is it bad to have simpler, easy-to-maintain, arch-consistent cross-toolchains in the archive, especially if the latter is what someone is volunteering to maintain, and no-one is volunteering to maintain the former (yet?). The simple ones can evolve and, in the way of things, are likely to become more capable and complicated over time. But what is the point of vetoing them, especially when we _know_ that such simpler cross-toochains (all that has been available for the last decade or so), satisfy most users already. There is more to discuss, but I've gone on long enough already :-) Wookey -- Principal hats: Linaro, Debian, Wookware, ARM http://wookware.org/ -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
