Hi Is the robots.txt file important? Do I need it? I currently dont use it and wondered if I should, does it benefit you in any way? Thanks
Having one saves on server error logs and if you have some areas of the site you would rather weren't indexed by the major players then yes, having one is useful. If you're not bothered about keeping the spiders out of areas of your site, simply save a blank page in Notepad, call it robots.txt and upload it to your site to stop the server error logs.
A robots.txt file is useful only if you wish that spiders do not visit some pages of your web site. This could be the case if your site has many very similar pages with different addresses (URL's). Allowing the visits of the spiders would consume bandwidth without any value to you. Many very good web sites do not have a robots.txt file. Jean-Luc
Users of search engines do not wish to get several times the same page in search results. Search engines do not wish to use their resources for pages that are identical or almost identical. What is "almost identical" or "very similar" ? That's hard to say. There are various tools that measure the similarity of web pages, for example this Similar Page Checker. Jean-Luc
Ok, thanks If I just upload a blank page like said above do I need to add any code to the pages as well?
Nope. A blank page will do fine if you aren't trying to exclude the spiders from parts of your site. More info about robots.txt http://en.wikipedia.org/wiki/Robots.txt
So I dont need to put in '<meta name="robots" content="noindex,nofollow" />'? What does '<meta name="robots" content="INDEX,FOLLOW">' mean? Thanks
No. Those are META tags. They go onto individual web pages in the <head></head> section, before the <body></body> They tell the spiders whether to follow the hyperlinks on a particular page or whether to ignore them. INDEX, FOLLOW means the spiders should follow the hyperlinks. NOINDEX, NOFOLLOW means the spiders should ignore the hyperlinks on that page.