Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-31 Thread Martin v. Löwis
Antoine Pitrou wrote: > Martin v. Löwis v.loewis.de> writes: >> It's not really reproducible. I think it sometimes happens when I >> restart the master; sometimes, some clients fail to reconnect >> (properly). > > Another common problem is that some buildbot fails in the middle of the test > suit

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-30 Thread Martin v. Löwis
> Since I've never used any such service ("cloud"-based VMs), I'm not sure > what the downsides would be. But it seems to be that it would be at least > worth trying. Not sure whether it's still relevant after the offers of individually donated hardware. However, if you want to look into this, f

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-30 Thread R. David Murray
On Fri, 30 Oct 2009 at 19:46, Paul Moore wrote: 2009/10/30 C. Titus Brown : Once things are up and running, I'll be prepared to do basic care and feeding of the buildslave, but as my time is limited, it would be nice if others would pitch in to help. I would be somewhat unhappy about giving mo

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-30 Thread Paul Moore
2009/10/30 C. Titus Brown : >> As a counter-offer: Given remote access to however many Windows VMs >> you want to provide, I'll get them up and running with buildslaves on >> them. If that requires software such as Visual Studio, I have copies >> via the MSDN licenses that I am happy to provide. >

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-30 Thread C. Titus Brown
On Fri, Oct 30, 2009 at 04:49:51PM +, Paul Moore wrote: > 2009/10/30 C. Titus Brown : > > On Fri, Oct 30, 2009 at 04:21:06PM +, Antoine Pitrou wrote: > >> > >> Hello, > >> > >> Sorry for the little redundancy, I would like to underline Jean-Paul's > >> suggestion here: > >> > >> Le Sun, 25

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-30 Thread Paul Moore
2009/10/30 C. Titus Brown : > On Fri, Oct 30, 2009 at 04:21:06PM +, Antoine Pitrou wrote: >> >> Hello, >> >> Sorry for the little redundancy, I would like to underline Jean-Paul's >> suggestion here: >> >> Le Sun, 25 Oct 2009 14:05:12 +, exarkun a ??crit??: >> > I think that money can help

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-30 Thread C. Titus Brown
On Fri, Oct 30, 2009 at 05:41:39PM +0100, Antoine Pitrou wrote: > Le vendredi 30 octobre 2009 ?? 09:31 -0700, C. Titus Brown a ??crit : > > [ ... ] > > > > I'm happy to provide VMs or shell access for Windows (XP, Vista, 7); Linux > > ia64; Linux x86; and Mac OS X. Others have made similar offers

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-30 Thread Antoine Pitrou
Le vendredi 30 octobre 2009 à 09:31 -0700, C. Titus Brown a écrit : > [ ... ] > > I'm happy to provide VMs or shell access for Windows (XP, Vista, 7); Linux > ia64; Linux x86; and Mac OS X. Others have made similar offers. The > architectures supported by the cloud services don't really add anyt

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-30 Thread C. Titus Brown
On Fri, Oct 30, 2009 at 04:21:06PM +, Antoine Pitrou wrote: > > Hello, > > Sorry for the little redundancy, I would like to underline Jean-Paul's > suggestion here: > > Le Sun, 25 Oct 2009 14:05:12 +, exarkun a ??crit??: > > I think that money can help in two ways in this case. > > > >

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-30 Thread Antoine Pitrou
Hello, Sorry for the little redundancy, I would like to underline Jean-Paul's suggestion here: Le Sun, 25 Oct 2009 14:05:12 +, exarkun a écrit : > I think that money can help in two ways in this case. > > First, there are now a multitude of cloud hosting providers which will > operate a sl

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-27 Thread Zooko O'Whielacronx
Right, how do developers benefit from a buildbot? >From my experience (five large buildbots with many developers plus two with only a couple of developers), a buildbot does little good unless the tests are reliable and not too noisy. "Reliable" is best achieved by having tests be deterministic an

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-27 Thread Antoine Pitrou
Martin v. Löwis v.loewis.de> writes: > > It's not really reproducible. I think it sometimes happens when I > restart the master; sometimes, some clients fail to reconnect > (properly). Another common problem is that some buildbot fails in the middle of the test suite, with the following kind of

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-26 Thread Martin v. Löwis
>>> This sounds like something that should be reported >>> upstream. Particularly if you know how to reproduce it. Has it been? >> >> No, largely because I can't reproduce it at all. It's happened maybe >> 4-5 times in the past 2 years or so. All that I see is that my end >> looks good yet the m

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-26 Thread Martin v. Löwis
David Bolen wrote: > MRAB writes: > >> Couldn't you write a script to check the status periodically? > > Sure, I suppose scraping the web status page would work. If it > happened frequently I'd probably be forced to do something like that, > but it's relatively low frequency (though I guess it

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread exarkun
On 01:28 am, db3l@gmail.com wrote: exar...@twistedmatrix.com writes: This sounds like something that should be reported upstream. Particularly if you know how to reproduce it. Has it been? No, largely because I can't reproduce it at all. It's happened maybe 4-5 times in the past 2 years

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread David Bolen
MRAB writes: > Couldn't you write a script to check the status periodically? Sure, I suppose scraping the web status page would work. If it happened frequently I'd probably be forced to do something like that, but it's relatively low frequency (though I guess it does have a big impact in terms

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread David Bolen
exar...@twistedmatrix.com writes: > This sounds like something that should be reported > upstream. Particularly if you know how to reproduce it. Has it been? No, largely because I can't reproduce it at all. It's happened maybe 4-5 times in the past 2 years or so. All that I see is that my end

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread exarkun
On 25 Oct, 09:36 pm, db3l@gmail.com wrote: I think the other issue most likely to cause a perceived "downtime" with the Windows build slave that I've had a handful of cases over the past two years where the build slave appears to be operating properly, but the master seems to just queue up j

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread MRAB
David Bolen wrote: [snip] I think the other issue most likely to cause a perceived "downtime" with the Windows build slave that I've had a handful of cases over the past two years where the build slave appears to be operating properly, but the master seems to just queue up jobs as if it were down

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread Antoine Pitrou
ssteinerX gmail.com gmail.com> writes: > On Oct 25, 2009, at 5:43 PM, Martin v. Löwis wrote: > > > Only turning on the slave occasionally makes it useless. > > For certain use cases; not mine. Let's say that for the use case we are talking here (this is python-dev), Martin's statement holds tr

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread sstein...@gmail.com
On Oct 25, 2009, at 5:43 PM, Martin v. Löwis wrote: Only turning on the slave occasionally makes it useless. For certain use cases; not mine. S ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Un

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread Martin v. Löwis
> It's not so much that I *require* the slave to shut down, more that > I'm not sure how well I'll be able to ensure that it's up all the > time, and I'm trying to understand the implications of that. My basic > impression is that it's not really going to work, unfortunately. There is a significa

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread David Bolen
"Martin v. Löwis" writes: > The remaining issue is the popups; if a process still has a popup, > you can't even terminate it properly. There are two kinds of popups: > system-generated ones, and CRT-generated ones. For the CRT ones, we > once had a way to turn them off, but I'm not sure whether t

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread Glyph Lefkowitz
On Oct 25, 2009, at 3:06 PM, Martin v. Löwis wrote: (*) it may help if Buildbot would create a Win32 job object, and then use TerminateJobObject. Contributions are welcome. Some work has already been done on this, but it needs help. At the root it's a Twisted issue: http://twistedmatr

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread Paul Moore
2009/10/25 "Martin v. Löwis" : >> I've been trying to get some feedback about firing up buildbots on Cloud >> Servers for a while now and haven't had much luck.  I'd love to find a >> way of having buildbots come to life, report to the mother ship, do the >> build, then go away 'till next time they

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread Martin v. Löwis
> The most *exciting* part of pony-build, apart from the always-riveting > spectacle of "titus rediscovering problems that buildbot solved 5 years ago", > is the loose coupling of recording server to the build slaves and build > reporters. My plan is to enable a simple and lightweight XML-RPC and/

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread sstein...@gmail.com
On Oct 25, 2009, at 3:35 PM, Martin v. Löwis wrote: I don't need to know that it works on every checkin For us, that is a fairly important requirement, though. Reports get more and more useless if they aren't instantaneous. Sometimes, people check something in just to see how the build slaves

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread Martin v. Löwis
> I don't need to know that it works on every checkin For us, that is a fairly important requirement, though. Reports get more and more useless if they aren't instantaneous. Sometimes, people check something in just to see how the build slaves react. Regards, Martin __

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread Martin v. Löwis
> This is supported in recent versions of BuildBot with a special kind of > slave: > > http://djmitche.github.com/buildbot/docs/0.7.11/#On_002dDemand- > _0028_0022Latent_0022_0029-Buildslaves Interesting. Coming back to "PSF may spend money", let me say this: If somebody would volunteer to set u

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread Martin v. Löwis
> it shouldn't be difficult to cobble together a build script that spins up a > buildslave on EC2 and runs the tests there; I wrote something similar a > few years ago for an infrequently connected home machine. Ok - so it would be the master running this script? Sounds reasonable to me. As for E

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread Martin v. Löwis
> I think that money can help in two ways in this case. > > First, there are now a multitude of cloud hosting providers which will > operate a slave machine for you. BuildBot has even begun to support > this deployment use-case by allowing you to start up and shut down vms > on demand to save on

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread Antoine Pitrou
Martin v. Löwis v.loewis.de> writes: > > If it's offline too often, I'm skeptical that it would be useful. If > you report breakage after a day, then it will be difficult to attribute > this to a specific commit. It is most useful to have continuous > integration if error reports are instantaneou

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread sstein...@gmail.com
On Oct 25, 2009, at 2:32 PM, Martin v. Löwis wrote: I've been trying to get some feedback about firing up buildbots on Cloud Servers for a while now and haven't had much luck. I'd love to find a way of having buildbots come to life, report to the mother ship, do the build, then go away '

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread Martin v. Löwis
> OK, sounds useful. If I'm offline for a while, do multiple builds get > queued, or only the "last" one? IIRC, it will only build the last one, then with a huge blame list. > If the former, I can imagine coming > back to a pretty huge load if the slave breaks while I'm on holiday > :-( If it's

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread Martin v. Löwis
> As I have no specific experience maintaining any of the CPython build > slaves, I can't speak to any maintenance issues which these slaves have > encountered. I would expect that they are as minimal as the issues I > have encountered maintaining slaves for other projects, but perhaps this > is w

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread Martin v. Löwis
Antoine Pitrou wrote: >> For a), I think we can solve this only by redundancy, i.e. create more >> build slaves, hoping that a sufficient number would be up at any point >> in time. > > We are already doing this, aren't we? > http://www.python.org/dev/buildbot/3.x/ > > It doesn't seem to work ver

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread exarkun
On 06:32 pm, mar...@v.loewis.de wrote: I've been trying to get some feedback about firing up buildbots on Cloud Servers for a while now and haven't had much luck. I'd love to find a way of having buildbots come to life, report to the mother ship, do the build, then go away 'till next time

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread C. Titus Brown
On Sun, Oct 25, 2009 at 07:32:52PM +0100, "Martin v. L?wis" wrote: > > I've been trying to get some feedback about firing up buildbots on Cloud > > Servers for a while now and haven't had much luck. I'd love to find a > > way of having buildbots come to life, report to the mother ship, do the > >

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread Martin v. Löwis
> I've been trying to get some feedback about firing up buildbots on Cloud > Servers for a while now and haven't had much luck. I'd love to find a > way of having buildbots come to life, report to the mother ship, do the > build, then go away 'till next time they're required. I'm not quite sure w

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread Paul Moore
2009/10/25 : > If you run a build slave and it's offline when a build is requested, the > build will be queued and run when the slave comes back online.  So if the > CPython developers want to work this way (I wouldn't), then we don't need > pony-build; BuildBot will do just fine. OK, sounds usef

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread exarkun
On 05:47 pm, p.f.mo...@gmail.com wrote: 2009/10/25 : Perhaps this is a significant portion of the problem.  Maintaining a build slave is remarkably simple and easy.  I maintain about half a dozen slaves and spend at most a few minutes a month operating them. Actually setting one up in the fir

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread Paul Moore
2009/10/25 : > Perhaps this is a significant portion of the problem.  Maintaining a build > slave is remarkably simple and easy.  I maintain about half a dozen slaves > and spend at most a few minutes a month operating them. Actually setting one > up in the first place might take a bit longer, sin

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread sstein...@gmail.com
On Oct 25, 2009, at 10:05 AM, exar...@twistedmatrix.com wrote: First, there are now a multitude of cloud hosting providers which will operate a slave machine for you. BuildBot has even begun to support this deployment use-case by allowing you to start up and shut down vms on demand to save

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread Antoine Pitrou
twistedmatrix.com> writes: > > To me, that raises the question of why people aren't more concerned with > the status of the build system. Shouldn't developers care if the code > they're writing works or not? The fact that we ask questions and publicly express worries should hint that we /are/

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread sstein...@gmail.com
On Oct 25, 2009, at 9:50 AM, exar...@twistedmatrix.com wrote: Actually setting one up in the first place might take a bit longer, since it involves installing the necessary software and making sure everything's set up right, but the actual slave configuration itself is one command: buildb

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread sstein...@gmail.com
On Oct 25, 2009, at 5:47 AM, Martin v. Löwis wrote: These are actually two issues: a) where do we get buildbot hardware and operators? I've been trying to get some feedback about firing up buildbots on Cloud Servers for a while now and haven't had much luck. I'd love to find a way of hav

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread Jesse Noller
On Sun, Oct 25, 2009 at 8:48 AM, C. Titus Brown wrote: > [ x-posting to testing-in-python; please redirect followups to one list or > the other! ] > > Hi Mark, > > a few bits of information... > > --- > > I have a set of VM machines running some "core" build archs -- Linux, Mac OS > X, > Win XP,

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread exarkun
On 09:47 am, mar...@v.loewis.de wrote: Mark Dickinson wrote: Would it be worth spending some time discussing the buildbot situation at the PyCon 2010 language summit? In the past, I've found the buildbots to be an incredibly valuable resource; especially when working with aspects of Python or

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread exarkun
On 12:16 pm, solip...@pitrou.net wrote: For a), I think we can solve this only by redundancy, i.e. create more build slaves, hoping that a sufficient number would be up at any point in time. We are already doing this, aren't we? http://www.python.org/dev/buildbot/3.x/ It doesn't seem to work

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread C. Titus Brown
On Sun, Oct 25, 2009 at 08:54:46AM +, Mark Dickinson wrote: > Would it be worth spending some time discussing the buildbot situation > at the PyCon 2010 language summit? In the past, I've found the > buildbots to be an incredibly valuable resource; especially when > working with aspects of Py

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread Antoine Pitrou
> For a), I think we can solve this only by redundancy, i.e. create more > build slaves, hoping that a sufficient number would be up at any point > in time. We are already doing this, aren't we? http://www.python.org/dev/buildbot/3.x/ It doesn't seem to work very well, it's a bit like a Danaides

Re: [Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread Martin v. Löwis
Mark Dickinson wrote: > Would it be worth spending some time discussing the buildbot situation > at the PyCon 2010 language summit? In the past, I've found the > buildbots to be an incredibly valuable resource; especially when > working with aspects of Python or C that tend to vary significantly

[Python-Dev] Possible language summit topic: buildbots

2009-10-25 Thread Mark Dickinson
Would it be worth spending some time discussing the buildbot situation at the PyCon 2010 language summit? In the past, I've found the buildbots to be an incredibly valuable resource; especially when working with aspects of Python or C that tend to vary significantly from platform to platform (for