hey all im in the middle of sorting out keyword relevance and have came across quiet a few websites that had this code in it??? <meta name="robots" content="follow,index" /> can anyone explain what its for or how it would help me???
It's a simple robots metatag that tell's search engines' bot to index the page and follow the links. The opposite would be <meta name="robots" content="nofollow,noindex" /> which would tell the bots not to index the page and not to follow the links.
You are just telling the search engines to index the page and follow all links on the page. Usually the same information is place in a robots.txt file in a slightly different format.
Greater assurances that spiders will index the page and follow the links. The likes of Google it will have almost no impact as I believe it assumes everything is follow and index as default unless it says otherwise but there may be some old spiders out there which default to no and dont read the robots.txt file
If you left it off a page and didn't have it in a robots.txt file either will it make a difference if its not in place. Basically is it a must have?
For Google NO. Maybe for some other search engine. BTW, a robots.txt file prevents a lot of 404's from showing up in your stats.
a robots.txt isn't necessary, but in a robots.txt file you can specify WHICH searchbots can enter your site, and what folders they are allowed to cache or not cache, based on what you want. Got a wonderful charity site, with an anti-semetic hatred forum in the /forum folder? use the robots.txt file to tell them not to look at it and your charity site won't get misrepresented in the search results.