This message contains some supplemental information to go with my primary writeup, and some profound thanks for the people involved in this investigation.
I apologize for the huge volume of mail, and I know it's going to take a while to digest. I appreciate people's willingness to read all these messages in detail. This is a very complex decision-making process. I'm pretty mentally exhausted by the effort of trying to synthesize all these pieces, so my apologies in advance for the inevitable errors and omissions (some of which affected my original writeup). 1. Thanks Throughout this evaluation process, my interactions with upstart and systemd upstream developers and Debian packagers have been uniformly excellent. Bug reports filed against both systemd and upstart have received excellent and timely response, and all involved have been quite willing to explain things I've misunderstood, correct my false starts, and discuss technical and practical aspects of their designs. I was particularly impressed by the clear effort that the systemd and upstart maintainers in Debian have put into fully integrating their init systems in such a way that makes them easy to test and use with existing Debian packages. This includes but is not limited to update-rc.d support, invoke-rc.d support, status synchronization with sysvinit, past Policy discussion, and attention to upgrade paths and init-switching use cases. I also want to particularly thank the OpenRC upstream development team for their involvement in this process and their contributions to the discussion. I personally don't think that package is a god match for Debian's needs on Linux, but that's through no fault of the people involved, and I think they would be an excellent upstream if that package looked like a good fit for the needs of any of Debian's non-Linux ports. I also want to thank Petter Reinholdtsen, Roger Leigh, and everyone else who has worked on the sysvinit package over the years, the insserv conversion to dependency-based boot, and the inclusion of LSB support. If it weren't for their hard work over the years, we would be in a far worse position than we are today. It's often hard to see people discussing the inadequacies of something into which you put years of hard work. I want to call attention to their long-term contributions to the distribution and the number of Debian systems that have booted through their efforts through the years. I am carefully keeping this discussion to the Technical Committee bug report so that all of my comments stay in context with their rebuttals, but this one section I think is unrelated to the rest of the discussion and should be more widely posted, so I will also be posting the above to my blog and Planet Debian. 2. upstart vs. systemd: The Equivalencies In doing this evaluation, I looked at quite a few different things. In a great number of cases, I found both upstart and systemd to be at an equally high level of quality. There are too many of those to list here, but I wanted to mention a few points that had been the topic of wider discussion. * The documentation of both systems was uniformly excellent. Both systems had complete reference manuals plus guidance to packagers for how to integrate with them. systemd also had excellent guidance for upstream developers on how to add systemd support relevant to upstream packages. * Both packages preserve traditional logging behavior, continue to properly utilize syslog, and retain logs in the places where I expected them. This is obviously not surprising for upstart, but I wanted to mention it explicitly since those unfamiliar with the systemd journal may be afraid that it would migrate logs into a separate, unexpected store. This is not the case; systemd logs are available in the normal places and can continue to be managed through syslog configuration. The journal is supplementary and enables some additional features discussed elsewhere. * Both packages move to configuration, from code, many of the annoyances and fiddly pieces involved in starting daemons. Both support non-forking daemon models and proper PID tracking in their own ways. Both directly support such routine needs as setting user and group, OOM score adjustment, resource limits, and so forth. * Both packages provide MAC integration, which is something that I think will become more important for Debian in the future. (systemd provides some additional features around Smack, but at least given what I know right now, I don't think this is a significant differentiation.) There was another point that I am calling equivalent since I didn't evaluate it either way. It's possible that this would convert into an advantage for one side or the other after further investigation: * Both systemd and upstart provide a way to run the system as a user process to manage user jobs in a similar fashion to how they manage system jobs. I believe this is already used extensively in Ubuntu to manage user desktop sessions. I don't where the systemd equivalent has been used. Finally, my interaction with both upstreams has been excellent, as I mentioned above in the Thanks section. I believe both maintenance teams would make excellent partners and upstreams for Debian going forward. 3. upstart: The Minor Advantages The following points did not make the cut into my main analysis. I consider them minor advantages that don't carry the same weight as the things that I commented on. However, I did want to mention some that had been the topic of discussion. * The upstart and libnih code bases are beautiful pieces of work. I was quite impressed by the code quality and documentation level, and more impressed by the extensive test suite. upstart and libnih follow the gold standard, commonly advocated but rarely met, of having around half of the code in the package be test cases. These are clearly code bases that have seen a great deal of love, care, and consistent development standards. The systemd code base also struck me as solid and at the standard that we would expect for a critical package, but the testing model is not as comprehensive or as integrated with the code, and it didn't impress me the same way that the upstart code did. * Both upstart and systemd are used by other major distribution projects, but upstart is used by Ubuntu, with which Debian has a special relationship. Debian collaborates more extensively with Ubuntu than with any other project, including largely sharing packaging between the projects and benefiting greatly from each other's testing and integrations. With upstart, Debian would have the immediate use of many upstart configurations that are either already in the archive or already available in Ubuntu, and which already fit Debian packaging standards and tools. I consider this a minor advantage for upstart. * upstart provides a more straightforward escape to shell scripting for some difficult initialization situations, whereas the systemd syntax for doing so is rather awkward. This cuts both ways, so you'll find that this point makes an appearance in the section below as well. But I think that, in some situations, this is a readability and ease of integration advantage for upstart. 4. systemd: The Minor Advantages Similarly, during this evaluation, I found several things that I preferred in systemd, but which didn't make the significance cut in my final evaluation. Here are some of them for the record. * systemd does not require a CLA, whereas upstart does. The Canonical Contributor Licensing Agreement has been much-discussed in these threads, and some people find it quite intrusive and unacceptable. The upstart package maintainers have committed to carrying non-CLA-covered contributions as local patches, which does a lot to ameliorate this concern, but I still consider this a minor advantage for systemd. * systemd provides really nice command-line tools for understanding the state of the system and the relationships between the unit files. I don't believe upstart has an equivalent of systemctl list-dependencies, for example. (systemctl status got special mention in my main evaluation.) * Both upstart and systemd, due to their more declarative nature, allow for daemon configurations that are more portable between different systems running the same init system. systemd has taken this farther into comprehensive and useful advice to upstream daemon authors for how to incorporate systemd support into a package, and also provides step-by-step instructions for how to install systemd unit files during package installation. While I don't believe that full portability of unit files will be achievable, I still think Debian would benefit from this when integrating systemd, as upstream unit files will often be usable as-is or with some minor patching. * systemd puts a lot of work into providing all the configuration directives required to express a huge variety of use cases without having to write complex startup commands or escape out to shell. This is the flipside, in some ways, to upstart's easy escape to shell. I particularly noted the wide variety of Condition* settings to control whether a unit should start. This is valuable for distribution packaging cases, but I think it's even more valuable for users of systemd-controlled systems in setting up local overrides and conditions for when they want to run a particular service. -- Russ Allbery (r...@debian.org) <http://www.eyrie.org/~eagle/> -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
