On Sun, 6 Feb 2000, Geoff Hutchison wrote:
GH> At 7:40 PM +0200 2/6/00, Valdas Andrulis wrote:
GH> >I doesnt work with 3.2.0b1, and do work with 3.1.4, with that redundant
GH> >robots.txt configurations, or just
GH> >User-agent: *
GH> >Disallow: /cat/
GH> >
GH> >or
GH> >
GH> >User-agent: htdig
GH> >Disallow: /cat/
GH>
GH> Hmph. This one's harder than your other bug report. The pattern is
GH> set correctly and the code looks correct. The new version switched to
GH> using regex internally so I bet this is a bug somewhere in the logic.
GH>
GH> First off, have you set case_sensitive to anything in your config file?
No.
GH>
GH> Secondly, could you try setting another (bogus) pattern in your robots.txt?
GH>
GH> Add:
GH> User-agent: htdig
GH> Disallow: /foobar/
GH>
GH> Then let us know what pattern it sets in the debug output--I don't
GH> really want the whole thing but I want to see if it's setting the
GH> pattern OK.
Trying to retrieve robots.txt file
Parsing robots.txt file using myname = htdig
Found 'user-agent' line: htdig
Found 'disallow' line: /cat/
Found 'user-agent' line: htdig
Found 'disallow' line: /foobar/
Pattern: /foobar/
VAldas
GH>
GH> I'll try something like this on my machine to see if I can reproduce it.
GH>
GH> -Geoff
GH>
GH>
GH> ------------------------------------
GH> To unsubscribe from the htdig3-dev mailing list, send a message to
GH> [EMAIL PROTECTED]
GH> You will receive a message to confirm this.
GH>
GH>
GH>
------------------------------------
To unsubscribe from the htdig3-dev mailing list, send a message to
[EMAIL PROTECTED]
You will receive a message to confirm this.
