On Sat, 5 Feb 2000, Geoff Hutchison wrote:
GH> At 10:29 PM +0200 2/5/00, Valdas Andrulis wrote:
GH> >User-agent: *
GH> >Disallow: /cat/
GH> >User-agent: htdig
GH> >Disallow: /cat/
GH> >
GH> >but htdig doesn't pay attention on that, it still indexes pages like
GH> >/cat/add_new_entry.php3?cat_id=121
GH>
GH> This is a bit strange. The robots.txt you mentioned is redundant--it
GH> says "no one should index /cat/" and then "htdig should not index
GH> /cat/"
GH>
GH> Nevertheless, it shouldn't do this. What version is this? 3.2.0b1?
GH> Does it seem to work under 3.1.4?
I doesnt work with 3.2.0b1, and do work with 3.1.4, with that redundant
robots.txt configurations, or just
User-agent: *
Disallow: /cat/
or
User-agent: htdig
Disallow: /cat/
seprately.
As i said i'll send logs later.
VAldas
GH>
GH> Thanks,
GH>
GH> -Geoff Hutchison
GH> Williams Students Online
GH> http://wso.williams.edu/
GH>
GH> ------------------------------------
GH> To unsubscribe from the htdig3-dev mailing list, send a message to
GH> [EMAIL PROTECTED]
GH> You will receive a message to confirm this.
GH>
GH>
GH>
------------------------------------
To unsubscribe from the htdig3-dev mailing list, send a message to
[EMAIL PROTECTED]
You will receive a message to confirm this.
