-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi.
Another issue that was raised in the discussion with the arch teams, even though it predates the arch teams resources thread as we've talked about it on FOSDEM 2011 and even before, is getting more automatic testing done on Gentoo. I'm bcc'ing a few teams on this thread as it involves them and hopefully might interest them as well. Both Release Engineering and QA teams would like to have more automatic testing to find breakages and to help track "when" things break and more importantly *why* they break. To avoid misunderstandings, we already have testing and even automated testing being done on Gentoo. The "first line" of testing is done by developers using repoman and or the PM's QA tools. We also have individual developers and the QA team hopefully checking commits and everyone testing their packages. Furtermore, the current weekly automatic stage building has helped identify some issues with the tree. The tinderbox work done by Patrick and Diego, as well as others, has also helped finding broken packages and or identifying packages affected by major changes before they hit the tree. The use of repoman, pcheck and or paludis quality assurance tools in the past and present to generate reports about tree issues, like Michael's (mr_bones) emails have also helped identifying and addressing issues. Recently, we've got a new site to check the results of some tests http://qa-reports.gentoo.org/ with the possibility to add more scripts to provide / run even more tests. So, why "more testing"? For starters, more *automatic* testing. Then more testing as reports from testing can help greatly in identifying when things break and why they break. As someone that looks over the automatic stage building for amd64 and x86, and that has to talk to teams / developers when things break, having more, more in depth and regular automatic testing would help my (releng) job. The work for the live-dvd would also be easier if the builds were "automated" and the job wasn't "restarted" every N months. Furthermore, creating a framework for developers to be able to schedule testing for proposed changes, in particular for substantial changes, might (should?) help improve the user's experience. I hope you agree with "more testing" by now, but what testing? It's good to test something, but "what" do we want to test and "how" do we want to test? I think we should try to have at least the following categories of tests: * Portage (overlays?) QA tests tests with the existing QA tools to check the consistency of dependencies and the quality of ebuilds / eclasses. * (on demand?) package (stable / unstable) revdep rebuild (tinderbox) framework to schedule testing of proposed changes and check their impact * Weekly (?) stable / unstable stage / ISO arch builds the automatic stage building, including new specs for the testing tree as we currently only test the stable tree. * (schedule?) specific tailored stage4 builds testing of specific tailored "real world" images (web server, intranet server, generic desktop, GNOME desktop, KDE desktop, etc). * Bi-Weekly (?) stable / unstable AMD64/X86 LiveDVD builds automatic creation of the live-DVD to test a very broad set of packages * automated testing of built stage / CD / LiveDVD (KVM guest?) (CLI / GUI / log parsing ?) framework to test the built stages / install media and ensure it works as expected I don't have a framework for conducting some of these tests, including the stage/iso validation, but some of them can use the existing tools like the stage building and the tree QA tests. Do you have any suggestions about the automatic testing? Do you know of other tests or tools that we can and should use to improve QA on Gentoo? - -- Regards, Jorge Vicetto (jmbsvicetto) - jmbsvicetto at gentoo dot org Gentoo- forums / Userrel / Devrel / KDE / Elections / RelEng -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJNyZx3AAoJEC8ZTXQF1qEPqoIQAKxIUHJItLX2HCgqbjmOMOTu P7Losyu6bAi9ndtyRGYwlmEHSRBgbrkHyllx2GCMj6vR20HBYWUiUaFn3QIghLLq 2d1Z75zzL6FN9IQvAM8BgQWEj7+Fe24MdOhx8knQmXzZn4jffzxeI/Clw/YzfxWd 7uVNWh2x48+/susjLhrkpmbQfcvuSuwK/qzhMsfJcbL5G0rHweoXtOI6L2fvLd/8 VxwtNPRm0lemB2DSifN5zmDiWe7Z1Tb+qnb7XZrj4KgJB154dbnpIirqW6eilYz7 zDVzGtjRm5MdRHzNxcHZ0M1XqR0N9BcwBBsqyh2Qhr6y8W8BX7gnqC/OuT+2LPOi HzvZ4sbGq2uq6/Fqjnyv9yWtqVNDjlJI2WjuZSsmZJaPVr/zSUptPfJEO/Qdla98 6aC7zdZucQAG8ai6KccttsaVv2N9Q5YAmZygBsiMjBZqNMfb8vsxN8VtDattd16Y ICnYBIyAxkazI94dp0dAuX429c+9+jTYZVMmGSbMQ8I/jFayEkpvim9wmCtIG+nx aySk+CKUpBFxF+nAttO0NEnM5oNtoNNx8k4VtMLRVyUG/LDK7z4p1OGocGZ1uELq +0aNNrY3qmDK4Yq0ID5bhp/gppn7PGrJBvm7zrqXUk7lVqs3NJHFSz4NLNIp41le o0qGl3+8Mhbns1mljpmx =sWpj -----END PGP SIGNATURE-----
