Hi Don,
How do you work this out? ie how do you know when you are spidered? I'm
interested in what logs and what patterns lead you to know this?
Thanks,
Tom
>
> On 07-Jul-01 Mark Lo wrote:
> > Hi,
> >
> > I don't know how to write a robot.txt file. Can anyone send me a
> > link or a sample file.
> >
>
> that's 'robots.txt'
>
> User-Agent: *
> Disallow: /images/
> Disallow: /icons/
> Disallow: /cgi-bin/
> Disallow: /php/
> Disallow: /tmp/
> Disallow: /corp/login.html
> Disallow: /cust/login.html
>
> ---
> But don't expect it to do anything for you, it's just a 'suggestion' to
> well-behaved spiders.
>
> My observation:
> 50% of them check & obey the file, 45% don't even check, & some 5% check
> then try the 'prohibited' URLs
>
> Regards,
> --
> Don Read [EMAIL PROTECTED]
> -- It's always darkest before the dawn. So if you are going to
> steal the neighbor's newspaper, that's the time to do it.
>
> --
> PHP General Mailing List (http://www.php.net/)
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
> To contact the list administrators, e-mail: [EMAIL PROTECTED]
>
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]
