Mark Rowe írta:
On 2012-03-01, at 03:37, Osztrogonac Csaba <[email protected]> wrote:
After unbanning our network, I'll copy a locally tar-ed WebKit-svn
copy to all bots and then restart them one by one not to overload
svn.webkit.org.
You can get a relatively up to date working copy from
<http://nightly.webkit.org/files/WebKit-SVN-source.tar.bz2>.
We always have relatively up-to-date working copy. It is faster then downloading
from you. ;) (I tried to download this nightly and the ETA is ~3 hours ...)
Additionally we use svn 1.7 (to make our bots faster, and less IO intensive)
which has different working copy than svn 1.6 .
And I'm thinking about how can we make buildbots more robust in the future.
My first idea is that we should setup git mirrors for all WebKit port, and
then make buildbots use these mirrors instead of svn.webkit.org. To do it,
we need to modify the master.cfg a little bit.
There are two different options that we're investigating to address this "thundering
herd" problem that tends to kill SVN after buildbot downtime:
1) Teach build slaves to fetch and unpack
<http://nightly.webkit.org/files/WebKit-SVN-source.tar.bz2> rather than using "svn
checkout".
To do it, all slave must use same svn version.
2) Add additional hardware and load-balance svn.webkit.org across several
machines so that the spike in load is distributed.
It is a good idea. ;)
To migrate the bots simple from svn to git isn't a good solution, because in
this case
we'll loose the svn revision number on the waterfall. My idea is that replace
the
"self.addStep(CheckOutSource)" to calling a shell/perl/python script which
updates
the local copy from the Qt/GTK/Apple/Chromium/... git mirror with the following
way:
- git fetch
- git reset --hard `git svn find-rev r<svn-revision-number-got-from-the-master>`
I'm going to implement this initial git updater script this week and try to
migrate our
bots on build.webkit.sed.hu to use it. If we manage to make it stable, we can
make
build.webkit.org slaves to use it too. How does it sound? Any better idea?
Pulling from git instead of SVN is certainly worth considering, but I wouldn't
be surprised if this simply shifted the performance issue from svn.webkit.org
to git.webkit.org.
No, I don't want to shift the load from svn.webkit.org to git.webkit.org. But
make all port
maintainer to use their own git mirror for the bots instead of the root
svn/git.webkit.org.
Mirroring git.webkit.org locally is the simplest thing in the world. :)
br,
Ossy
_______________________________________________
webkit-dev mailing list
[email protected]
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
