On 17/11/16 20:16, Rich Freeman wrote:
> On Thu, Nov 17, 2016 at 2:16 AM, Michael Palimaka <kensing...@gentoo.org>
> wrote:
>>
>> In cases where all USE flags combinations are not being tested, it is
>> still recommended to test:
>> * with all USE flags enabled
>> * with all USE flags disabled
>> * the default USE flag settings
>>
>
> I imagine that in practice only the last of these really tends to get tested.
>
>> * A leaf package such as {{package|kde-apps/kcalc}} may not require any
>> runtime testing at all
>
> I'm not really a big fan of this, but if we REALLY didn't want to do
> any runtime testing on a package then we should have some way to tag
> the package as such, and then have some way to do automated
> build-test-only stabilization. If you aren't doing runtime testing
> then manual stabilization adds zero value.
How much value do you think we gain from runtime testing a package like
kcalc as part of the stabilisation process, considering that it already
sat in ~arch for at least 30 days?
Also, based on conversations with various arch team members, my
understanding is that a lot of stabilisations that happen right now are
already build-only.
>
> Overall though the writeup was good and maybe it will trigger some
> discussion. I tend to think that if we want to do things like testing
> permutations and such then automated build-only tools might be the way
> to address this. Manual effort should be focused on things like
> runtime testing where it adds the most value. This also strikes me as
> the sort of thing that could probably be assigned out to volunteers
> who do not have commit access.
There's a few tools for this out there already. I've personally been
working to update app-portage/tatt for git - see
https://asciinema.org/a/cqsy983t9jimszvypcjr3zg5m for a demo.
>
> It really seems like the sort of thing that could be managed by
> something other than bugzilla. Some tool finds out about packages
> that ought to be stabilized (probably via multiple methods), then it
> triggers the automated build tests/etc that do a lot of the low-level
> QA, and if the package looks good it gets queued for runtime testing.
> Then volunteers report in on status and when whatever criteria we
> establish is met then the tool stabilizes the package, probably in a
> dependency-aware fashion. Obviously this would require some care for
> coordinated packages like xorg/DEs/etc, and it might not be the
> preferred approach for many system packages.
>
We're working on a tool like this in #gentoo-grumpy - new contributors
welcome!
