How do they work? What do I put in them? Basically I don't want a specific page on the site or file in my directory to be indexed.
User-agent: * Disallow: /directorynamehere/ Disallow: /pagenamehere.html Code (markup): This will tell all spiders not to index the directorynamehere directory and also not to index the pagenamehere document.