Tracking Testing, Release Status, and Build Health

2018-07-29 Thread Scott Andreas
Hi everyone,

As September approaches, I’ve been thinking about how we might be able to share 
notes on testing and build health.

Some types of information I have in mind:

– Known issues with builds (e.g., master / nightlies) that impact the ability 
to test the project. These could include known-flaky tests, bugs impacting test 
automation, or specific features that aren’t functioning as designed. Some 
contributors may be prefer to waive specific types of testing to prevent their 
automation from failing; others may prefer to continue more exhaustive testing 
of builds known not to be impacted by an issue.
– Tracking high-priority bugs that are known blockers, either for test 
automation or suitability of a release for dev / QA environments.
– Pre-release checklists and outstanding to-do items that need to be completed 
prior to cutting a build which aren’t necessarily best represented by a JIRA 
ticket.
– Performance test status – the characteristics of tests performed, hardware 
configurations, and improvements / regressions identified over time.

Some of this information may change often as development and testing progress - 
as often as daily in some cases.

Here are a couple examples of similar work in other projects:

–
– Hadoop’s “Release Status” docs in Confluence: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8.0+Release

The Hadoop project uses these status docs to track release blockers, to-do 
items that need to be tracked ahead of release, and notes on scope as defined 
by the community.

– Hadoop’s “Road Map” docs in Confluence:
https://cwiki.apache.org/confluence/display/HADOOP/Roadmap

Similarly, the Hadoop project uses docs like this to track planned features for 
each release, and feature freeze, code freeze, and planned release dates. These 
are useful for organizing development and signposting what’s planned to the 
user community.

– “Are We Fast Yet”: A project from Mozilla tracking JS engine performance
https://arewefastyet.com

Perf testing of nightlies across a several platforms (and comparisons with 
other JavaScript engines), useful for identifying regressions quickly and 
tracking progress toward perf goals.
–

I’m curious on the dev community’s thoughts on how best to organize information 
like this. My thinking is that by having a space to share this, the community 
can be more informed on each others’ work toward testing, build health, and 
active projects.

The current Cassandra wiki (https://wiki.apache.org/cassandra) doesn’t appear 
too active and carries a warning. What do others think about filing an INFRA 
ticket to request a Confluence space at https://cwiki.apache.org for this type 
of information?

I’d be happy to help maintain information tracking build health, remaining 
to-do’s, known test automation blockers, flaky tests, etc. as well.

Cheers,

– Scott



Re: Tracking Testing, Release Status, and Build Health

2018-07-29 Thread Nate McCall
Thanks for putting this together Scott!

I like the idea of using Confluence for coordinating the bits and
pieces on the way to 4.0. There are better things out there, but It's
lightweight, straight forward and everyone with a ASF Jira account can
participate. The one thing it does very well IMO is link back and
forth with Jira so referencing issues is really easy.

Kafka also makes good use of Confluence for development process stuff.
it's not the most organized, but it's kept up to date and their
committers and contributors use it. Check our their release pages:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.2.2

On Mon, Jul 30, 2018 at 12:04 PM, Scott Andreas  wrote:
> Hi everyone,
>
> As September approaches, I’ve been thinking about how we might be able to 
> share notes on testing and build health.
>
> Some types of information I have in mind:
>
> – Known issues with builds (e.g., master / nightlies) that impact the ability 
> to test the project. These could include known-flaky tests, bugs impacting 
> test automation, or specific features that aren’t functioning as designed. 
> Some contributors may be prefer to waive specific types of testing to prevent 
> their automation from failing; others may prefer to continue more exhaustive 
> testing of builds known not to be impacted by an issue.
> – Tracking high-priority bugs that are known blockers, either for test 
> automation or suitability of a release for dev / QA environments.
> – Pre-release checklists and outstanding to-do items that need to be 
> completed prior to cutting a build which aren’t necessarily best represented 
> by a JIRA ticket.
> – Performance test status – the characteristics of tests performed, hardware 
> configurations, and improvements / regressions identified over time.
>
> Some of this information may change often as development and testing progress 
> - as often as daily in some cases.
>
> Here are a couple examples of similar work in other projects:
>
> –
> – Hadoop’s “Release Status” docs in Confluence: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8.0+Release
>
> The Hadoop project uses these status docs to track release blockers, to-do 
> items that need to be tracked ahead of release, and notes on scope as defined 
> by the community.
>
> – Hadoop’s “Road Map” docs in Confluence:
> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap
>
> Similarly, the Hadoop project uses docs like this to track planned features 
> for each release, and feature freeze, code freeze, and planned release dates. 
> These are useful for organizing development and signposting what’s planned to 
> the user community.
>
> – “Are We Fast Yet”: A project from Mozilla tracking JS engine performance
> https://arewefastyet.com
>
> Perf testing of nightlies across a several platforms (and comparisons with 
> other JavaScript engines), useful for identifying regressions quickly and 
> tracking progress toward perf goals.
> –
>
> I’m curious on the dev community’s thoughts on how best to organize 
> information like this. My thinking is that by having a space to share this, 
> the community can be more informed on each others’ work toward testing, build 
> health, and active projects.
>
> The current Cassandra wiki (https://wiki.apache.org/cassandra) doesn’t appear 
> too active and carries a warning. What do others think about filing an INFRA 
> ticket to request a Confluence space at https://cwiki.apache.org for this 
> type of information?
>
> I’d be happy to help maintain information tracking build health, remaining 
> to-do’s, known test automation blockers, flaky tests, etc. as well.
>
> Cheers,
>
> – Scott
>

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org