Jasper Bryant-Greene wrote:
Jochem Maas wrote:
Jasper Bryant-Greene wrote:
robots.txt will not do what you want it to.
Just sniff for those robots' User-Agents (Google, MSN and Yahoo all
publish their UA strings on their websites, AFAIK) and send different
content if it's one of those.
Evert | Collab wrote:
Lets just put it this way:
if you don't want your site indexed, use robots.txt
if you want to hide your site from search engines [ which won't even
touch your files if you use robots.txt ] check the UA string.
I can't imagine a situation where you want to hide your conte
* Brian Dunning <[EMAIL PROTECTED]> :
> On Aug 11, 2005, at 4:06 PM, Evert | Collab wrote:
>
> > First hit on google:
> > http://www.searchengineworld.com/robots/robots_tutorial.htm
> > Search engines check for a robots.txt on your site, in the
> > robots.txt file you can specify that certain or
Philip Hallstrom wrote:
robots.txt will not do what you want it to.
Just sniff for those robots' User-Agents (Google, MSN and Yahoo all
publish their UA strings on their websites, AFAIK) and send different
content if it's one of those.
they will hammer you for it eventually - AFAICT all maj
robots.txt will not do what you want it to.
Just sniff for those robots' User-Agents (Google, MSN and Yahoo all publish
their UA strings on their websites, AFAIK) and send different content if
it's one of those.
they will hammer you for it eventually - AFAICT all major SEs send out their
spid
Jochem Maas wrote:
Jasper Bryant-Greene wrote:
robots.txt will not do what you want it to.
Just sniff for those robots' User-Agents (Google, MSN and Yahoo all
publish their UA strings on their websites, AFAIK) and send different
content if it's one of those.
they will hammer you for it eve
Jasper Bryant-Greene wrote:
Brian Dunning wrote:
On Aug 11, 2005, at 3:44 PM, Evert | Collab wrote:
Use robots.txt
'evil' searchengines will spoof the user-agent string anyway
Can you be more specific about what you mean by "use robots.txt"?
I just want to cloak for Google, MSN, and Yaho
Brian Dunning wrote:
On Aug 11, 2005, at 3:44 PM, Evert | Collab wrote:
Use robots.txt
'evil' searchengines will spoof the user-agent string anyway
Can you be more specific about what you mean by "use robots.txt"?
I just want to cloak for Google, MSN, and Yahoo. I couldn't care less
about
On Aug 11, 2005, at 4:06 PM, Evert | Collab wrote:
First hit on google:
http://www.searchengineworld.com/robots/robots_tutorial.htm
Search engines check for a robots.txt on your site, in the
robots.txt file you can specify that certain or all search engines
shouldn't index your site
I know
On Aug 11, 2005, at 3:44 PM, Evert | Collab wrote:
Use robots.txt
'evil' searchengines will spoof the user-agent string anyway
Can you be more specific about what you mean by "use robots.txt"?
I just want to cloak for Google, MSN, and Yahoo. I couldn't care less
about what any other search
10 matches
Mail list logo
