Robots.txt file questions

I know some basic HTML and can get my sites to generally do what I want, but one thing I'm curious on is how to set up a robots.txt file. I am setting up some course sites behind a squeeze page and I don't want them to get crawled. I've looked a little bit and just can't seem to find a decent tutorial on this (maybe I'm just looking in the wrong places). Does anyone know how to do this or where a decent tutorial on this may be?

 

Why dont you look at their site http://www.robotstxt.org/

 

put this in a text document:

User-agent: *
Disallow: *

save as robots.txt

upload to root

 

Uhhh, no, do not do that, that will not allow the robots to crawl anything.

http://en.wikipedia.org/wiki/Robots.txt

 

this could also help...

http://www.askapache.com/seo/seo-with-robotstxt.html

 

Thanks for all the help and the great links!

 

If you read his post thats what he was looking for.

 

Anybody ever using this:

http://forums.digitalpoint.com/forumdisplay.php?f=50

I think this can be a proper place to ask that kind of question. But I think that this will do:

User-Agent: *
Disallow: /hidden-dir/

 

Also, if your directory will be password protected, then the search engine spiders (bots) will not be able to access the content anyway, since all they'll see is the login prompt.

Just another thing to consider (and yes, the robots.txt board here on the forums is the appropriate place to ask about such matters - but here works as well).