How do I write a robot text file? Where should I put it? Should this include keywords? What keywords..this is my latest blog: http://carpartsreview.blogspot.com/
A robots.txt doesn't have any keywords. It's just a file telling the search bots which directories to look at and which to ignore. It also indicates the location of your site's sitemap. An example robots.txt may contain User-agent: * Allow: / Sitemap: http://buycheap-oem-software.com/sitemap.xml Code (markup): and would be stored as "robots.txt" in the root directory of your domain (i.e. the top-most location).
Ive heard that some people think that adding a sitemap pointed such as this may cause the robots.txt file to become un validated? What is your opinion of this? Kind regards
All crap.. this is the ideal format one should use in his website's robots.txt. The reason sitemap should be put in the robots file is that since robots visit robots.txt file first when they arrive on a website, so they would also visit our sitemap.xml file as well and could crawl our website better.
Adding a Site map to your Robots.txt is a normal way of adding a sitemap to your website, all major Search engines use it ( Bing ASK Google) and will not make your Robots.txt un-verified Heres an example of mine User-agent: * Allow: / Sitemap: http://www.fta-n-more.net/sitemap.xml Code (markup): When I ran all the test on it , it says it's valid , and Google's webmaster tools Now caries my sitemap, so you can use the same formate... If you need to create a sitemap, theres tons of FREE websites to do it for you... Best of luck