Re: Regression testing breakthrough

2011-10-25 Thread Francois Gouget
On Tue, 25 Oct 2011, Damjan Jovanovic wrote: [...] > Now the next question is, how to get the binaries to run on any distro? Or > should I just compile on Ubuntu because most people run that (do they still, > after Unity?)? Compile on Debian Stable or even Debian OldStable, taking care to still m

Re: Regression testing breakthrough

2011-10-25 Thread Damjan Jovanovic
2011/10/18 André Hentschel > Am 18.10.2011 10:45, schrieb Damjan Jovanovic: > > This tool compiled all 35000 or so commits from Wine 1.0 to around 4th > October 2011 in only 7 days, generating a Git repository of Wine binaries > that's only 26 gigabytes in size. Regression testing with binaries i

Re: Regression testing breakthrough

2011-10-20 Thread Hin-Tak Leung
I have two suggestions - - git clone has a "--depth" option which does a shallow clone (i.e. with some history removed). - you can use "git-archive" to export arbitrary commits out as a tar ball dynamically; there is no need to have store a tar ball permanently. That said, I am doubtful about

Re: Regression testing breakthrough

2011-10-19 Thread Marcus Meissner
On Wed, Oct 19, 2011 at 04:18:50PM +0200, Frédéric Delanoy wrote: > On Wed, Oct 19, 2011 at 15:50, Marcus Meissner wrote: > > On Wed, Oct 19, 2011 at 02:42:29PM +0100, Ken Sharp wrote: > >> > >> > >> On 19/10/11 13:43, Frédéric Delanoy wrote: > >> >On Wed, Oct 19, 2011 at 14:08, Joel Holdsworth  

Re: Regression testing breakthrough

2011-10-19 Thread Frédéric Delanoy
On Wed, Oct 19, 2011 at 15:50, Marcus Meissner wrote: > On Wed, Oct 19, 2011 at 02:42:29PM +0100, Ken Sharp wrote: >> >> >> On 19/10/11 13:43, Frédéric Delanoy wrote: >> >On Wed, Oct 19, 2011 at 14:08, Joel Holdsworth   >> >wrote: >> >>Alternatively, have you considered doing a .tar.gz of every bu

Re: Regression testing breakthrough

2011-10-19 Thread Marcus Meissner
On Wed, Oct 19, 2011 at 02:42:29PM +0100, Ken Sharp wrote: > > > On 19/10/11 13:43, Frédéric Delanoy wrote: > >On Wed, Oct 19, 2011 at 14:08, Joel Holdsworth > >wrote: > >>Alternatively, have you considered doing a .tar.gz of every build snapshot, > >>and placing that on a server somewhere? > >

Re: Regression testing breakthrough

2011-10-19 Thread Ken Sharp
On 19/10/11 13:43, Frédéric Delanoy wrote: On Wed, Oct 19, 2011 at 14:08, Joel Holdsworth wrote: Alternatively, have you considered doing a .tar.gz of every build snapshot, and placing that on a server somewhere? e.g. a folder full of 36def4af0ca85a1d0e66b5207056775bcb3b09ff.tar.gz files?

Re: Regression testing breakthrough

2011-10-19 Thread Frédéric Delanoy
On Wed, Oct 19, 2011 at 14:08, Joel Holdsworth wrote: > Alternatively, have you considered doing a .tar.gz of every build snapshot, > and placing that on a server somewhere? > > e.g. a folder full of 36def4af0ca85a1d0e66b5207056775bcb3b09ff.tar.gz files? tar.xz would compress better > Then one c

Re: Regression testing breakthrough

2011-10-19 Thread Joel Holdsworth
Alternatively, have you considered doing a .tar.gz of every build snapshot, and placing that on a server somewhere?   e.g. a folder full of36def4af0ca85a1d0e66b5207056775bcb3b09ff.tar.gz files?   Then one could write a simple wine regression bisect tool that implements similar semantics to git bise

Re: Regression testing breakthrough

2011-10-18 Thread Damjan Jovanovic
2011/10/18 André Hentschel > Am 18.10.2011 10:45, schrieb Damjan Jovanovic: > > This tool compiled all 35000 or so commits from Wine 1.0 to around 4th > October 2011 in only 7 days, generating a Git repository of Wine binaries > that's only 26 gigabytes in size. Regression testing with binaries i

Re: Regression testing breakthrough

2011-10-18 Thread André Hentschel
Am 18.10.2011 10:45, schrieb Damjan Jovanovic: > This tool compiled all 35000 or so commits from Wine 1.0 to around 4th > October 2011 in only 7 days, generating a Git repository of Wine binaries > that's only 26 gigabytes in size. Regression testing with binaries is a > pleasure: it takes only

Re: Regression testing breakthrough

2011-10-18 Thread Austin English
On Tue, Oct 18, 2011 at 10:26, Dmitry Timoshkov wrote: > Austin English wrote: > > >> > Reverting a patch in latest git is not always possible, instead it's >> > a very useful test to revert the patch at the suspected regression point >> > and see if that really helps. >> >> That still doesn't re

Re: Regression testing breakthrough

2011-10-18 Thread Dmitry Timoshkov
Austin English wrote: > > Reverting a patch in latest git is not always possible, instead it's > > a very useful test to revert the patch at the suspected regression point > > and see if that really helps. > > That still doesn't require a full regression test, just: > $ git checkout -f $SHA1SUM

Re: Regression testing breakthrough

2011-10-18 Thread Austin English
On Tue, Oct 18, 2011 at 09:01, Dmitry Timoshkov wrote: > Damjan Jovanovic wrote: >> > Moreover, often users get asked 'does reverting commit ' help? Without >> > performing a proper regression test it's impossible to asnwer that >> > question. >> > >> > >> Reverting a commit in the latest git

Re: Regression testing breakthrough

2011-10-18 Thread Dmitry Timoshkov
Damjan Jovanovic wrote: > > Moreover, often users get asked 'does reverting commit ' help? Without > > performing a proper regression test it's impossible to asnwer that > > question. > > > > > Reverting a commit in the latest git is just 1 round of > patch+configure+make+run, and reverting to

Re: Regression testing breakthrough

2011-10-18 Thread Damjan Jovanovic
On Tue, Oct 18, 2011 at 2:32 PM, Dmitry Timoshkov wrote: > Henri Verbeet wrote: > > > On 18 October 2011 10:45, Damjan Jovanovic wrote: > > > (especially during "reverse regression testing"), users find it too > long and > > > technical, and only a small minority of regressions are ever bisecte

Re: Regression testing breakthrough

2011-10-18 Thread Frédéric Delanoy
On Tue, Oct 18, 2011 at 13:42, Damjan Jovanovic wrote: > If you are talking about using compiling with ccache instead of the binary > repository, "configure" alone is > 40 seconds configure -C option can speed it up a lot

Re: Regression testing breakthrough

2011-10-18 Thread Dmitry Timoshkov
Henri Verbeet wrote: > On 18 October 2011 10:45, Damjan Jovanovic wrote: > > (especially during "reverse regression testing"), users find it too long and > > technical, and only a small minority of regressions are ever bisected. And > Not true. Even for the regressions that are still open it's c

Re: Regression testing breakthrough

2011-10-18 Thread Henri Verbeet
On 18 October 2011 13:42, Damjan Jovanovic wrote: > There's currently another 182 regressions that were closed "ABANDONED". > Maybe if regression testing was easier and faster, people wouldn't abandon > them? > Maybe. That's 182 closed ABANDONED, out of 2590 total closed, so that's on the order of

Re: Regression testing breakthrough

2011-10-18 Thread Scott Ritchie
Exciting! On 10/18/2011 01:45 AM, Damjan Jovanovic wrote: > I haven't figured out how to make the binaries available to users. Few > users can clone a 26 gigabyte repository, and even fewer places can > serve that much to multiple users. Maybe Git can compress it further? > The other idea I had is

Re: Regression testing breakthrough

2011-10-18 Thread Damjan Jovanovic
On Tue, Oct 18, 2011 at 12:08 PM, Henri Verbeet wrote: > On 18 October 2011 10:45, Damjan Jovanovic wrote: > > (especially during "reverse regression testing"), users find it too long > and > > technical, and only a small minority of regressions are ever bisected. > And > Not true. Even for the

Re: Regression testing breakthrough

2011-10-18 Thread Henri Verbeet
On 18 October 2011 10:45, Damjan Jovanovic wrote: > (especially during "reverse regression testing"), users find it too long and > technical, and only a small minority of regressions are ever bisected. And Not true. Even for the regressions that are still open it's currently 276 bisected vs. 99 no

Re: Regression testing breakthrough

2011-10-18 Thread Ferenc Gergely Szilagyi
On Tue, Oct 18, 2011 at 10:45 AM, Damjan Jovanovic wrote: > Hi > > Since the beginning, I've had issues with regression testing. Despite the > fact it's very useful, it takes forever, it's easy to make a mistake > (especially during "reverse regression testing"), users find it too long and > techn

Regression testing breakthrough

2011-10-18 Thread Damjan Jovanovic
Hi Since the beginning, I've had issues with regression testing. Despite the fact it's very useful, it takes forever, it's easy to make a mistake (especially during "reverse regression testing"), users find it too long and technical, and only a small minority of regressions are ever bisected. And