Thanks for starting discussion! Replying to the thread with what I would have left as comments.
–––––– > As yet, we lack empirical evidence to quantify the relative stability or > instability of our project compared to a peer cohort I think it's more important that we set a standard for the project (e.g., fundamental conformance to properties of the database) rather than attempting to measure quality relative to other DBs. That might be a useful measure, but I don't think it's the most important one. With regard to measuring against a common standard in the project, this is roughly what I had in mind when proposing "Release Quality Metrics" on the list in 2018. I still think making progress on something like this is essential toward defining a quantitative bar for release: https://www.mail-archive.com/dev@cassandra.apache.org/msg13154.html > Conversely, the ability to repeatedly and thoroughly validate the correctness > of both new and existing functionality in the system is vital to the speed > with which we can evolve it's form and function. Strongly agreed. > Utopia (and following section) Some nods to great potential refactors to consider post-4.0 here. ^ > We should productize a kubernetes-centric, infra agnostic tool that has the > following available testing paradigms: This would be an excellent set of capabilities to have. > We need to empower our user community to participate in the testing process... I really like this point. I took as a thought experiment "what would feel great to be able to say" if one were to write a product announcement for 4.0 and landed on something like "Users of Apache Cassandra can preflight their 4.0 upgrade by runing $tool to clone, upgrade, and compare their clusters, ensuring that the upgrade will complete smoothly and correctly." > The less friction and less investment we can require from ecosystem > participants, the more we can expect them to engage in desired behavior. +1 –––––– I like the document and there's a lot that has me nodding. Toward the opening statement on "empirical evidence to quantify relative stability," I'd love to revisit discussion on quantifying attributes like these here: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=93324430 – Scott ________________________________________ From: David Capwell <dcapw...@gmail.com> Sent: Tuesday, July 14, 2020 6:23 PM To: dev@cassandra.apache.org Subject: Re: [DISCUSS] A point of view on Testing Cassandra I am also not fully clear on the motives, but welcome anything which helps bring in better and more robust testing; thanks for starting this. Since I can not comment in the doc I have to copy/paste and put here... =( Reality > ... > investing in improving our smoke and integration testing as much as is > possible with our current constraints seems prudent. This section reads as very anti-adding tests to test/unit; I am 100% in favor of improving/creating our smoke, integration, regression, performance, E2E, etc. testing, but don't think I am as negative to test/unit, these tests are still valuable and more are welcome. To enumerate a punch list of traits we as engineers need from a testing > suite Would be good to speak about portability, accessibility, and version independents. If a new contributor wants to add tests to this suite they need to be able to run it, and it should run within a "reasonable" time frame; one of the big issuers with python dtests is that it takes 14+ hours to run, this makes it no longer accessible to new contributors. On Tue, Jul 14, 2020 at 11:47 AM Joshua McKenzie <jmcken...@apache.org> wrote: > The purpose is purely to signal a point of view on the state of testing in > the codebase, some shortcomings of the architecture, and what a few of us > are doing and further planning to do about it. Kind of a "prompt discussion > if anyone has a wild allergic reaction to it, or encourage collaboration if > they have a wild positive reaction" sort of thing. Maybe a spiritual > "CEP-lite". :) > > I would advocate that we be very selective about the topics on which we > strive for a consistent shared point of view as a project. There are a lot > of us and we all have different experiences and different points of view > that lead to different perspectives and value systems. Agreeing on discrete > definitions of done, 100% - that's table stakes. But agreeing on how we get > there, my personal take is we'd all be well served to spend our energy > Doing the Work and expressing these complementary positions rather than > trying to bend everyone to one consistent point of view. > > Let a thousand flowers bloom, as someone wise recently told me. :) > > That said, this work will be happening in an open source repo with a > permissive license (almost certainly ASLv2), likely using github issues, so > anyone that wants to collaborate on it would be most welcome. I can make > sure Gianluca, Charles, Berenguer, and others bring that to this ML thread > once we've started open-sourcing things. > > On Tue, Jul 14, 2020 at 4:25 AM Benedict Elliott Smith < > bened...@apache.org> > wrote: > > > It does raise the bar to critiquing the document though, but perhaps > > that's also a feature. > > > > Perhaps we can first discuss the purpose of the document? It seems to be > a > > mix of mission statement for the project, as well as your own near term > > roadmap? Should we interpret it only as an advertisement of your own > view > > of the problems the project faces, as a start to dialogue, or is the > > purpose to solicit feedback? > > > > Would it be helpful to work towards a similar document the whole > community > > endorses, with a shared mission statement, and a (perhaps loosely > defined) > > shared roadmap? > > > > I'd like to call out some specific things in the document that I am > > personally excited by: the project has long lacked a coherent, repeatable > > approach to performance testing and regressions; combined with easy > > visualisation tools this would be a huge win. The FQL sampling with data > > distribution inference is also something that has been discussed > privately > > elsewhere, and would be hugely advantageous to the former, so that we can > > discover representative workloads. > > > > Thanks for taking the time to put this together, and start this dialogue. > > > > > > On 13/07/2020, 23:41, "Joshua McKenzie" <jmcken...@apache.org> wrote: > > > > > > > > Can you please allow comments on the doc so we can leave feedback. > > > > > > > > > > Doc is view only; figured we could keep this to the ML. > > > > > That's a feature, not a bug. > > > > Happy to chat here or on slack w/anyone. This is a complex topic so > > long-form or high bandwidth communication is a better fit than gdoc > > comments. They rapidly become unwieldy. > > > > On Mon, Jul 13, 2020 at 6:17 PM sankalp kohli < > kohlisank...@gmail.com> > > wrote: > > > > > Can you please allow comments on the doc so we can leave feedback. > > > > > > On Mon, Jul 13, 2020 at 2:16 PM Joshua McKenzie < > > jmcken...@apache.org> > > > wrote: > > > > > > > Link: > > > > > > > > > > > > > > https://docs.google.com/document/d/1ktuBWpD2NLurB9PUvmbwGgrXsgnyU58koOseZAfaFBQ/edit# > > > > > > > > > > > > Myself and a few other contributors are working with this point > of > > view > > > as > > > > our frame of where we're going to work on improving testing on > the > > > project. > > > > I figured it might be useful to foster collaboration more broadly > > in the > > > > community as well as provide people with the opportunity to > > discuss work > > > > they're doing they may not yet have had a chance to bring up or > > open > > > > source. While fallout is already open-sourced, expect the schema > > > anonymizer > > > > and some of the cassandra-diff + nosqlbench framework effort to > be > > > > open-sourced / openly worked on soon. Anyone that's interested in > > > > collaborating, that would be highly welcome. > > > > > > > > Doc is view only; figured we could keep this to the ML. > > > > > > > > Thanks. > > > > > > > > ~Josh > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
