Re: Testing and jira tickets
If I remember correctly, the requirement of providing test results along with each patch was because of tick-tock, where the goal was to have stable release branches at all times. Without CI for testing each individual commit on all branches, this just won't work anymore. But would that really be that bad? Can't we just get away with a single CI run per branch and day? E.g. in the future we could commit to dev branches that are used to run all tests automatically on Apache CI on daily basis, which is then exclusively used for that. We don't have that many commits on a single day, some of them rather trivial, and I think we'd be able to figure out the one of them causing a regression on the day after. If all tests pass, we can merge dev manually or even better automatically. If anyone wants to run tests on his own CI before committing to dev, that's fine too and will help analyzing any regressions if they happen, as we then don't have to look at those patches (and all commits before on dev). On 09.03.2017 19:51, Jason Brown wrote: > Hey all, > > A nice convention we've stumbled into wrt to patches submitted via Jira is > to post the results of unit test and dtest runs to the ticket (to show the > patch doesn't break things). Many contributors have used the > DataStax-provided cassci system, but that's not the best long term > solution. To that end, I'd like to start a conversation about what is the > best way to proceed going forward, and then add it to the "How to > contribute" docs. > > As an example, should contributors/committers run dtests and unit tests on > *some* machine (publicly available or otherwise), and then post those > results to the ticket? This could be a link to a build system, like what we > have with cassci, or just upload the output of the test run(s). > > I don't have any fixed notions, and am looking forward to hearing other's > ideas. > > Thanks, > > -Jason > > p.s. a big thank you to DataStax for providing the cassci system >
Re: committing performance patches without performance numbers
The elephant in the room to me is what Ariel said: > > What about all the commits that don't intend to have a performance impact > but do? On top of that, it's all well and good to microbench the change you make in a vacuum but what happens when the resource usage implications of that change have side-effects on the rest of the system? I've always been a vocal proponent of proving your improvement w/numbers (be they microbenchmark or stress-based single node, etc), so I'm strongly +1 on us improving our hygiene on this front. On Thu, Mar 9, 2017 at 4:56 PM, Ariel Weisberg wrote: > Hi, > > I should clarify. Should in the sense of it was fine for the process we > had at the time (ugh!) but it's not what we should do in the future. > > Ariel > > On Thu, Mar 9, 2017, at 04:55 PM, Ariel Weisberg wrote: > > Hi, > > > > I agree that patch could have used it and it was amenable to > > micro-benchmarking. Just to be pedantic about process which is something > > I approve of to the chagrin of so many. > > > > On a completely related note that change also randomly boxes a boolean. > > > > Ariel > > > > On Thu, Mar 9, 2017, at 03:45 PM, Jonathan Haddad wrote: > > > I don't expect everyone to run a 500 node cluster off to the side to > test > > > their patches, but at least some indication that the contributor > started > > > Cassandra on their laptop would be a good sign. The JIRA I referenced > > > was > > > an optimization around List, Set and Map serialization. Would it > really > > > have been that crazy to run a handful of benchmarks locally and post > > > those > > > results? > > > > > > On Thu, Mar 9, 2017 at 12:26 PM Ariel Weisberg > wrote: > > > > > > > Hi, > > > > > > > > I think there are issues around the availability of hardware > sufficient > > > > to demonstrate the performance concerns under test. It's an open > source > > > > project without centralized infrastructure. A lot of performance > > > > contributions come from people running C* in production. They are > > > > already running these changes and have seen the improvement, but > > > > communicating that to the outside world in a convincing way can be > hard. > > > > > > > > Right now we don't even have performance in continuous integration. I > > > > think we are putting the cart before the horse in that respect. What > > > > about all the commits that don't intend to have a performance impact > but > > > > do? Even if we had performance metrics in CI who is going to triage > the > > > > results religiously? > > > > > > > > We also to my knowledge don't have benchmarks for key functionality > in > > > > cassandra-stress. Can cassandra-stress benchmark CAS? My > recollection is > > > > every time I looked is that it wasn't there. > > > > > > > > We can only set the bar as high as contributors are able to meet. > > > > Certainly if they can't justify why they can't benchmark the thing > they > > > > want to contribute then reviewers should make them go and benchmark > it. > > > > > > > > Regards, > > > > Ariel > > > > > > > > On Thu, Mar 9, 2017, at 03:11 PM, Jeff Jirsa wrote: > > > > > Agree. Anything that's meant to increase performance should > demonstrate > > > > > it > > > > > actually does that. We have microbench available in recent > versions - > > > > > writing a new microbenchmark isn't all that onerous. Would be > great if we > > > > > had perf tests included in the normal testall/dtest workflow for > ALL > > > > > patches so we could quickly spot regressions, but that gets pretty > > > > > expensive in terms of running long enough tests to actually see > most > > > > > common > > > > > code paths. > > > > > > > > > > > > > > > On Thu, Mar 9, 2017 at 12:00 PM, Jonathan Haddad < > j...@jonhaddad.com> > > > > > wrote: > > > > > > > > > > > I'd like to discuss what I consider to be a pretty important > matter - > > > > > > patches which are written for the sole purpose of improving > performance > > > > > > without including a single performance benchmark in the JIRA. > > > > > > > > > > > > My original email was in "Testing and Jira Tickets", i'll copy > it here > > > > > > for posterity: > > > > > > > > > > > > If you don't mind, I'd like to broaden the discussion a little > bit to > > > > also > > > > > > discuss performance related patches. For instance, > CASSANDRA-13271 > > > > was a > > > > > > performance / optimization related patch that included *zero* > > > > information > > > > > > on if there was any perf improvement or a regression as a result > of the > > > > > > change, even though I've asked twice for that information. > > > > > > > > > > > > In addition to "does this thing break anything" we should be > asking > > > > "how > > > > > > does this patch affect performance?" (and were the appropriate > docs > > > > > > included, but that's another topic altogether) > > > > > > > > > > > > There's a minor note about perf related stuff here: > > > > > > http://cassandra.apache.org/doc/latest/development/ > > > > > >
Re: Testing and jira tickets
> > I think we'd be able to figure out the one of them causing a regression > on the day after. That sounds great in theory. In practice, that doesn't happen unless one person steps up and makes themselves accountable for it. For reference, take a look at: https://cassci.datastax.com/view/trunk/, and https://issues.apache.org/jira/issues/?jql=project%20%3D%20cassandra%20and%20resolution%20%3D%20unresolved%20and%20labels%20in%20(%27test-fail%27%2C%20%27test-failure%27%2C%20%27testall%27%2C%20%27dtest%27%2C%20%27unit-test%27%2C%20%27unittest%27)%20and%20assignee%20%3D%20null%20order%20by%20created%20ASC We're thankfully still in a place where these tickets are at least being created, but unless there's a body of people that are digging in to fix those test failures they're just going to keep growing. On Fri, Mar 10, 2017 at 5:03 AM, Stefan Podkowinski wrote: > If I remember correctly, the requirement of providing test results along > with each patch was because of tick-tock, where the goal was to have > stable release branches at all times. Without CI for testing each > individual commit on all branches, this just won't work anymore. But > would that really be that bad? Can't we just get away with a single CI > run per branch and day? > > E.g. in the future we could commit to dev branches that are used to run > all tests automatically on Apache CI on daily basis, which is then > exclusively used for that. We don't have that many commits on a single > day, some of them rather trivial, and I think we'd be able to figure out > the one of them causing a regression on the day after. If all tests > pass, we can merge dev manually or even better automatically. If anyone > wants to run tests on his own CI before committing to dev, that's fine > too and will help analyzing any regressions if they happen, as we then > don't have to look at those patches (and all commits before on dev). > > > > On 09.03.2017 19:51, Jason Brown wrote: > > Hey all, > > > > A nice convention we've stumbled into wrt to patches submitted via Jira > is > > to post the results of unit test and dtest runs to the ticket (to show > the > > patch doesn't break things). Many contributors have used the > > DataStax-provided cassci system, but that's not the best long term > > solution. To that end, I'd like to start a conversation about what is the > > best way to proceed going forward, and then add it to the "How to > > contribute" docs. > > > > As an example, should contributors/committers run dtests and unit tests > on > > *some* machine (publicly available or otherwise), and then post those > > results to the ticket? This could be a link to a build system, like what > we > > have with cassci, or just upload the output of the test run(s). > > > > I don't have any fixed notions, and am looking forward to hearing other's > > ideas. > > > > Thanks, > > > > -Jason > > > > p.s. a big thank you to DataStax for providing the cassci system > > >
[RELEASE] Apache Cassandra 3.0.12 released
The Cassandra team is pleased to announce the release of Apache Cassandra version 3.0.12. Apache Cassandra is a fully distributed database. It is the right choice when you need scalability and high availability without compromising performance. http://cassandra.apache.org/ Downloads of source and binary distributions are listed in our download section: http://cassandra.apache.org/download/ This version is a bug fix release[1] on the 3.0 series. As always, please pay attention to the release notes[2] and Let us know[3] if you were to encounter any problem. Enjoy! [1]: (CHANGES.txt) https://goo.gl/sc8BvB [2]: (NEWS.txt) https://goo.gl/S5gpKA [3]: https://issues.apache.org/jira/browse/CASSANDRA -- Warm regards, Michael Shuler signature.asc Description: OpenPGP digital signature
