On Tue, May 03, 2011 at 08:23:44AM -0400, James E Keenan wrote: > > On May 3, 2011, at 8:16 AM, Patrick R. Michaud wrote: > >The size/content of the (bundled) spectests between 2011.01 and > >2011.04 doesn't seem to have changed significantly. > > Would it be possible to create a subset of the spectests that have > not changed over, say, the past 12 months and are considered > sufficiently stable that we could use them as benchmarks?
Twelve months is likely to be too ambitious -- there have been many (small) changes to the base language and/or spectest structure over the past year that would end up disqualifying a lot of useful tests. Six months or less is probably more reasonable for now. Creating such a list is relatively easy -- simply grab a tarball of the earliest and latest rakudo distributions you want to compare, then compare the spectest files from each (in the t/spec directory). I've written a short Perl script to do exactly this and placed it in the rakbench repository [1]. I've also created a file with a list of unchanged spectests from 2011.01 to 2011.04 and placed it in the repository at [2]. Currently that list contains 433 unchanged spectests. The file is formatted such that it can be easily copied to t/localtest.data in a rakudo source directory, and then "make localtest" can be used to run just the tests unchanged from 2011.01 to 2011.04. I chose 2011.01 as the starting point because there were a fair number of spectest changes in December, as well as the fact that it's served as a very useful baseline for all of the other performance testing we've done. I'll update this file after each rakudo release so we can continue to have timings in the future. I'm going to add a script to rakbench that compares the overall time needed to run the list of common spectests. This should serve as a nice apples-to-apples comparison of Rakudo+Parrot performance over time. When I have some updated results I'll post them. Pm [1] https://github.com/pmichaud/rakbench/blob/master/bin/spectest-same.pl [2] https://github.com/pmichaud/rakbench/blob/master/src/commontest.data _______________________________________________ http://lists.parrot.org/mailman/listinfo/parrot-dev
