On Fri, May 13, 2011 at 7:14 PM, Ian Lance Taylor <i...@google.com> wrote: > I noticed that buglist.cgi was taking quite a bit of CPU time. I looked > at some of the long running instances, and they were coming from > searchbots. I can't think of a good reason for this, so I have > committed this patch to the gcc.gnu.org robots.txt file to not let > searchbots search through lists of bugs. I plan to make a similar > change on the sourceware.org and cygwin.com sides. Please let me know > if this seems like a mistake. > > Does anybody have any experience with > http://code.google.com/p/bugzilla-sitemap/ ? That might be a slightly > better approach.
Shouldn't we keep searchbots way from bugzilla completely? Searchbots can crawl the gcc-bugs mailinglist archives. Richard. > Ian > >
