[issue21475] Support the Sitemap and Crawl-delay extensions in robotparser

2014-05-12 Thread Berker Peksag
Berker Peksag added the comment: There is a patch for Crawl-delay in issue 16099. -- nosy: +berker.peksag ___ Python tracker ___ ___ P

[issue21475] Support the Sitemap and Crawl-delay extensions in robotparser

2014-05-11 Thread Raymond Hettinger
New submission from Raymond Hettinger: Resources: * http://en.wikipedia.org/wiki/Robots_exclusion_standard#Nonstandard_extensions * https://support.google.com/webmasters/answer/183669?hl=en * https://github.com/seomoz/reppy -- components: Library (Lib) keywords: easy messages: 218308