Bug#459498: closed by Micah Cowan <[EMAIL PROTECTED]> (Re: Bug#459498: option to ignore robots.txt)

Hi Micah, * Micah Cowan <[EMAIL PROTECTED]> [2008-01-07 19:34]: > Nico Golde wrote: > > Then the manual of wget really sucks, searching for robots > > returns three matches and none these is related to this > > option. > > Only if you're searching the man page (which, as I already mentioned, is

Bug#459498: closed by Micah Cowan <[EMAIL PROTECTED]> (Re: Bug#459498: option to ignore robots.txt)

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Nico Golde wrote: > Then the manual of wget really sucks, searching for robots > returns three matches and none these is related to this > option. Only if you're searching the man page (which, as I already mentioned, is not the full manual). Use the

Bug#459498: closed by Micah Cowan <[EMAIL PROTECTED]> (Re: Bug#459498: option to ignore robots.txt)

Hi, > It has never been necessary to modify the source code to ignore > robots.txt files; rather, you can specify "robots=off" in your .wgetrc; > or "-e robots=off" on the commandline. More information about this is in > the "Robot Exclusion" section of the Texinfo manual; use "info wget" to > acc

Bug#459498: option to ignore robots.txt

Package: wget Version: 1.10.2-3 Severity: wishlist Hi, I know it's not nice to do it, but it would be great if there would be an option to ignore the robots.txt file without modifying RES_SPECS_LOCATION in the source. Kind regards Nico -- Nico Golde - http://www.ngolde.de - [EMAIL PROTECTED] -