Take a read of Google's help for creating robots.txt file. Whenever you create one use notepad. Its a very easy process to create one depending on what you want or do not want accessed by the search engine bots.
Are you trying to create a robot.txt for a stand alone website? What website? With robot.txt you can prevent search engines bots to crawl certain parts of your website. and of course to tell them where to crawl (but they are pretty good at finding the pages on their own)....
see stoping them from crawling my site is a problem there is so many pages how do I pick which ones its for My Forum certain parts of the site have no follow tags so can these parts of the site be left out of it?
Is this right? I used a free generator do I still need to find all the pages with no follow tags and enter them or will they still follow the no follow rule? User-agent: Googlebot Disallow: User-agent: Googlebot-Image Disallow: User-agent: MSNBot Disallow: User-agent: Slurp Disallow: User-agent: Teoma Disallow: User-agent: Gigabot Disallow: User-agent: Scrubby Disallow: User-agent: Robozilla Disallow: User-agent: Nutch Disallow: User-agent: ia_archiver Disallow: User-agent: baiduspider Disallow: User-agent: yahoo-mmcrawler Disallow: User-agent: psbot Disallow: User-agent: asterias Disallow: User-agent: yahoo-blogs/v3.9 Disallow: User-agent: * Disallow: Disallow: /cgi-bin/ Sitemap: http://www.websiteforumz.com/sitemap.xml Sitemap: http://www.websiteforumz.com/generator/data/sitemap.html Sitemap: http://www.websiteforumz.com/generator/data/urllist.txt Sitemap: http://www.websiteforumz.com/ror.xml Code (markup):