so is there a way around that problem ?? On Fri, Jan 23, 2009 at 2:25 PM, Andre Engels <andreeng...@gmail.com> wrote:
> On Fri, Jan 23, 2009 at 9:09 AM, amit sethi <amit.pureene...@gmail.com> > wrote: > > Well that is interesting but why should that happen in case I am using a > > different User Agent because I tried doing > > status=rp.can_fetch('Mozilla/5.0', > > "http://en.wikipedia.org/wiki/Sachin_Tendulkar") > > but even that returns false > > Is there something wrong with the syntax , Is there a catch that i don't > > understand. > > The problem is that you are using the standard Python user agent when > getting the robots.txt. Because the user agent is refused, it cannot > get the robots.txt file itself to look at. > > -- > André Engels, andreeng...@gmail.com > -- A-M-I-T S|S
_______________________________________________ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor