I need to create a robots.txt for an HTML template site I am running. I'm not too great with the robot files and was wondering what suggesstions some of you may have. Are there any good example site's out there? Should images be disallowed?
Its is not hard to create a robots.txt file. Just create a .txt file using notepad name it robots. Whatever you want to disallow a folder or a file, put a specific name of particular. You can use it as: User-agent: * Disallow: /folders/ Disallow: /filename1.ext Disallow: /filename1.ext Disallow: /filename1.ext Put these lines in, you created a file name robots.txt and replace the name of folders and files in the place of /folders/ and /filename1.ext. Try it without any fear.
A rogbots text file stops robots from indexing things you dont want listed... (admin folders, includes folders, etc). Also, you will get some robots that will use alot of bandwidth and give nothing to your site. Sitebuildit (SBIDER) comes to mind. They crawled and crawled, but help my site in no way. However, some bandwidth hog robots disregard your robots txt anyways. With robots.txt and ip ban, you can usually control alot of these bandwidth hogs. Also, you can specify a sitemap in your robots text in this format: SITEMAP: http://www.mysite.com/sitemap.xml which may or may not help.