Please give me a robots.txt file to exclude as many search engines as possible!!

Hi there...I was wondering if anyone has a sample of a good robots.txt file to put on my server to make sure no pages get spidered. (By as many search engines as possible)

I had a developement site set up and somehow it got indexed (the dev site wasn't at index.html) I had some feedback from a forum post I think so it got spidered.

Also...Once I put this up will Google delist it next time it crawls?

Thanks.

 

SOmeone correct me if I am wrong, but I thnk this is all you need:

User-agent: *
Disallow: *

 

Disallow: /

You got the useragent right. The / means root.

 

DOH! I knew that. I dont know what I was thinking when I typed that in. The wild card isnt used in the disallow part but the url, or directory is.

Thanks for correcting me on that silly mistake.

So all you need in the robots.txt is

User-agent: *
Disallow: /

 

just remember that not all robots obey the robots.txt file

 

yes u need only :
user-agent: *
Disallow: /

i agree with above

 

Is this firm, or do some spiders crawl what ever the hell they want?

 

not all robots obey the robot.txt file. i have a dev site also that's listen on noname search engines. it gets traffic too lol

 

That is true, Not all robots will obey such 'commands' as dissallow or even rel="nofollow". There really is no way to absolutely guarantee 100% that your pages will not be crawled

 

That's what I kind of figured on my own, after checking the file a million times.
Thanks for confirming.

 

do you think the search engines may delist the site? or just not update it?

 

also does google obey robots.txt?

 

Google will listen to the robots.txt. However, it may take quite awhile until the site gets dropped from the index.