On Fri, Jan 23, 2009 at 9:09 AM, amit sethi <amit.pureene...@gmail.com> wrote:
> Well that is interesting but why should that happen in case I am using a
> different User Agent because I tried doing
> status=rp.can_fetch('Mozilla/5.0',
> "http://en.wikipedia.org/wiki/Sachin_Tendulkar";)
> but even that returns false
> Is there something wrong with the syntax , Is there a catch that i don't
> understand.
The problem is that you are using the standard Python user agent when
getting the robots.txt. Because the user agent is refused, it cannot
get the robots.txt file itself to look at.
--
André Engels, andreeng...@gmail.com
_______________________________________________
Tutor maillist - Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor
