Okay, so I have my page up, and not everthing is complete. I do not want spiders to crawl me yet. What do I do? Keep in my mind I have google adsense on my site to.
No Indexand No Follow robots tag i believe does the trick <meta name="robots" content="noindex,nofollow"> noindex tells the bots to refrain from indexing the page and nofollow tells the bots to not follow any links
why not just place robots.txt file into the root and specify what would you need all the spiders to skip... More info @ http://en.wikipedia.org/wiki/Robots.txt
Not entirely true, Ive had sites crawled within days. Though i've got one in developemnt now that i have been working on for months. And I have not bothered with a no index tag. Google finds new domains quite quickly but wont bother to crawl it till its found some links pointing to it. If you dont tell anyone about the site, and place no links to it, your likely not going to get crawled.
I heard that the noindex tag doesn't always work. Is this true? That a bot may still index your site even if you tell it not to?
Apparently the major search engines have all agreed to respect this tag. Those bots that may ignore it will have no bearing on your search engine indexing. Private bots sent out to mine data or email address likely dont bother to respect the tag.