Jóhann B. Guðmundsson [2016-02-18 11:35 +0000]: > Arguably not if upstream test do not detect these things then they should be > fixed to do so
The upstream tests are mostly unit tests which cover isolated logic. Of course having more coverage there would be great, but that's not the point here. systemd is not an isolated project, but it lives in an operating system, controlling services like dbus, gdm, NetworkManager, etc. So saying "fix them to do so" in an email is easy, but did you actually think about *how* this should look like? We do have some QEMU tests upstream which cover a little bit of "does this VM boot", but not much else -- there is no integration testing with other components. So for doing proper integration testing you need to take an actual OS, with actual packages, and check that they still work. If that's Fedora or Debian or whatnot is not that important in the end, and ideally we test them all. But it's certainly not practical for the upstream systemd tests to pull and build glib, X/wayland, gdm, NetworkManager, evemu-tools, etc. and construct their own synthetic OS into a VM. That would be a huge maintenance burden, take awfully long, and this synthetic mini-distro isn't actually being used by any any real person. So, is it possible that these tests will fail because of a downstream issue? Absolutely, yes, and I'm sure they will do that at some point. I'm looking at those failures, and they are a problem that needs fixing anyway (and if it's a downstream issue, I'm no less on the hook for that than now). But experience has shown that almost all test failures which aren't due to actual regressions are due to either bugs in the tests or infrastructure failure -- and both of these will apply equally to so-called "upstream tests". So integrating these into PRs instead of running them every other day on master makes it much simpler and cheaper to identify the cause of a regrssion. You can also always compare against other PRs or a test against master. Martin -- Martin Pitt | http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) _______________________________________________ systemd-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/systemd-devel
