2008/1/3, Don Armstrong <[EMAIL PROTECTED]> wrote: > On Thu, 03 Jan 2008, Jason Spiro wrote: > > http://en.wikipedia.org/wiki/Robots.txt#Crawl-delay_directive will > > help. Yahoo and MSNBot both support it. I bet other major bots > > support it too. So we can allow Yahoo and MSNBot (plus Googlebot, if > > they support it too) and block everyone else. > > Google doesn't, unfortunatly. I've already had a bit of a discussion > about this particular issue with them.
That is unfortunate. What did they tell you? (Could you forward us their last message?) Though they do provide their Google Webmaster Tools which allow you to adjust the crawl rate to "slower".[1] > [The other half of the problem > is that the Crawl delay directive is per-bot, and not indexer-wide.] Does that make it impractical for us to allow robots? Cheers, -j ^[1] http://www.google.com/webmasters/tour/tour4.html -- Jason Spiro: corporate trainer, web developer, IT consultant. I support Linux, UNIX, Windows, and more. Contact me to discuss your needs and get a free estimate. +1 (613) 668-6096 / Email: [EMAIL PROTECTED] / MSN: [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
