Stop search engines from crawling my site

how can i stop search engines from crawling my site. I don't wish for my site to be listed on any pages untill its finished which is why ive only given it out to about 10 people "WHO" i think i can trust but just incase i want to make sure that it dont get listed in case someone post the link somewhere.

 

Here is a good read for you.
Robots.txt

 

Yup defaintely exclude the bot from indexing you by using the robots file. The above is a good read and explanation.

cheers

 

You can stops SEs from crawling site by using robots text file .

 

As well as using robots.txt which is certainly a good idea, it may also be advisable to use the robot meta tag in the head section of the html code of your index/home page.

The values ALL and NONE set all directives on or off: ALL=INDEX,FOLLOW and NONE=NOINDEX,NOFOLLOW.

Some examples:

<meta name="robots" content="index,follow">
<meta name="robots" content="noindex,follow">
<meta name="robots" content="index,nofollow">
<meta name="robots" content="noindex,nofollow">

Code (markup):

Note the "robots" name of the tag and the content are case insensitive.

You obviously should not specify conflicting or repeating directives such as:

<meta name="robots" content="INDEX,NOINDEX,NOFOLLOW,FOLLOW,FOLLOW">

Code (markup):

A formal syntax for the Robots META tag content is:

content = all | none | directives
all = "ALL"
none = "NONE"
directives = directive ["," directives]
directive = index | follow
index = "INDEX" | "NOINDEX"
follow = "FOLLOW" | "NOFOLLOW"

So as you don't want the engines to list you, you would use:

<meta name="robots" content="noindex,nofollow">

Code (markup):

Hope this helps

Regards Roger