On Wednesday, November 20, 2002, at 10:38 AM, Renick, Garrel wrote:
It seems to work OK for me. Is there a chance that some of the pages that are not being indexed are only reachable through a page with nofollow? If you explicitly add the pages not being indexed to your start_url, are the pages then indexed?Hello. I'm using ht://Dig 3.1.6. I seem to be having the same problem reported in 1998 where ht:dig was not indexing pages with the robots meta tag set to content="index,nofollow":http://www.htdig.org/mail/1998/09/0235.html http://www.htdig.org/mail/1998/09/0303.html Are any of you experiencing this same problem with version 3.1.6? Could there be some other configuration problem I'm overlooking?
Jim
-------------------------------------------------------
This sf.net email is sponsored by: Battle your brains against the best in the Thawte Crypto Challenge. Be the first to crack the code - register now: http://www.gothawte.com/rd521.html
_______________________________________________
htdig-general mailing list <[EMAIL PROTECTED]>
To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe
FAQ: http://htdig.sourceforge.net/FAQ.html
