Loans - 0% credit cards - Web Advertising - Loans - Credit Cards

PDA

View Full Version : Block Google Spider/Bot


movidalatina
Feb 5th 2008, 3:40 pm
hello! anyone know how i can stop
the Google spider bot from indexing
my site? the whole site. I know there must
be a file named robots.txt , but what should
go in there?

Alffy
Feb 5th 2008, 3:59 pm
put this in your robots.txt:

UserAgent: Googlebot
Disallow: /

Can also add:

<meta name="robots" content="noindex, nofollow" />
<meta name="robots" content="noarchive" />

to the header sections of your web pages.

WebGeek182
Feb 5th 2008, 4:09 pm
<meta name="robots" content="noindex, nofollow" />
<meta name="robots" content="noarchive" />

Those two lines should be combined into one line:
<meta name="robots" content="noindex,nofollow,noarchive" />

You don't want to have multiple meta robots tags (technically there only be one) because not all spiders interpret it the same way and some will only read one of the lines.

Webray
Feb 5th 2008, 4:33 pm
... or tell Google to go fly-a-kite with something like this:

SetEnvIf User-Agent ^Googlebot stealthed #block googleBot - later dude

<Limit GET POST >
order allow,deny
allow from all
deny from env=stealthed
</Limit>


.htaccess ONLY

movidalatina
Feb 5th 2008, 6:13 pm
Excellent. I'll try these.
Thanks fellas.