Hi,
Baruch Even ([EMAIL PROTECTED]) wrote on 2007-03-10:
> I'm not even sure that it should go for robots.txt, the idea of websec
> is that it behaves more like a browser, it doesn't crawl the website, it
> doesn't try to go anywhere it's not directly instructed to and if a user
> insists on misbeh
* Thomas Themel <[EMAIL PROTECTED]> [070310 16:56]:
> Hi,
>
> as Dani??l states, it would be nice if websec behaved like a proper web
> robot. Baruch, are you aware that the change is basically a two-liner?
I'm not even sure that it should go for robots.txt, the idea of websec
is that it behaves
Hi,
as Daniël states, it would be nice if websec behaved like a proper web
robot. Baruch, are you aware that the change is basically a two-liner?
LWP, the web library you're using, provides a subclass of UserAgent that
respects the robots.txt protocol, see here:
http://search.cpan.org/dist/libww
3 matches
Mail list logo
