>
> Found it!
Great. Thanks a lot for this good lesson about robots.txt. I'm
downloading ~2000 robots.txt and store them in
http://www.senga.org/htdig/robots/. The list of servers that have robots.txt
files is http://www.senga.org/htdig/robots/robots-list and was extracted
from our search engine.
This will, at least, give us an idea of what people actually use in their
robots.txt. My intuitive understanding was that each section was self
contained. Despite of that I did not suggest a solution to the reported
problem. I'm in favour of the most restrictive interpretation because I
think most people will think this way.
I know for sure that some site use the Allow tag and would be in favour of
supporting it.
Cheers,
--
Loic Dachary
24 av Secretan
75019 Paris
Tel: 33 1 42 45 09 16
e-mail: [EMAIL PROTECTED]
URL: http://www.senga.org/
------------------------------------
To unsubscribe from the htdig3-dev mailing list, send a message to
[EMAIL PROTECTED]
You will receive a message to confirm this.
