On Mon, Jan 26, 2015 at 7:21 AM, Florian Weimer <[email protected]> wrote: > Any idea why?
https://lists.fedoraproject.org/robots.txt User-agent: * Crawl-delay: 10 From Bing, "This means the higher your crawl delay is, the fewer pages BingBot will crawl. As crawling fewer pages may result in getting less content indexed, we usually do not recommend it, although we also understand that different web sites may have different bandwidth constraints." [1]. [1] http://blogs.bing.com/webmaster/2012/05/03/to-crawl-or-not-to-crawl-that-is-bingbots-question/ Brandon Vincent -- devel mailing list [email protected] https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
