Get rid of the meta name="robots" content="index, follow" tag?

Seen on other SEO forums that they recommend removing it as it can hinder the bot.

Does the tag cause any harm?

 

The robots tag is a limiting tag, it has no positive effect. By that I mean that it is really an exclusion of the robots instruction, so, if you do not intedn restricting the robot, there is really no point in it being there IMO.

 

I agree completely. I use either the meta or robots.txt to block bots when desired, but that's about it. I do not have any tags asking to be indexed and have never really noticed any kind of hinderance - thus I really don't think they are necessary at all.

 

"index, follow" is the default, so it makes no sense to repeat it by adding a meta tag.

Craig

 

But it's not doing any real harm then, just using up a few kb's??

Surely some robots aren't as clever as Google and need a bit of guidance?

 

I know of no spider/bot that requires that tag in order to crawl a site.

 

No real harm, but you might have another instance of a keyword at the 98KB mark of the page and if you have unessasary code at the top it might get pushed to the 101KB mark. Then when Google or whoever pays attention to only the first 100KB then you lose one more keyword mention.

Of course this is a little extreme as an example, but why let it have a chance of happening?

It's like having one operating system on your PC and telling the computer to ask you which OS to boot up everytime you boot. Just an extra button to push, or in the spider's case just another line of text to read.

 

I also was informed that "bad robots/spider" or "spam robots" ignore the "no index.." tag or when you limit the "index tag" to certain "bots"

Just found a interesting posting by "Matt Cutts..the Google quy" http://www.mattcutts.com/blog/handling-noindex-meta-tags/

 

it works for certain site and not for others

 

I think the point of index, follow is when you exclude a group of robots with robots.txt and then want to let one or two etc in

 

The last two posts are completely wrong.

 

Restricting the bots will practically isolate your website from the search engines. Now why would you want to do that (unless you have something to hide).

 

Exactly its pointless to add that tag.

 

i agrea too, tags are a strange thing and some work and others dont

 

it will not hurt you seo.

that tag tell search tell search to index the page and follows the links on that page

 

I agree. They do not hurt your site. They basically let the robots know where to go. I added it and the site got indexed by google the next day. I use google webmaster tools.

If it doesn't hurt why not keep it?

Good Luck,

 

Whoever told you that they will hurt your rankings or prevent your site from being spidered (and indexed) needs to get out of SEO and take up something far less dangerous, like getting a McJob. Seriously.

Think about it for a second. What is a search engine spider's default action? To crawl Web pages and index them into the search engine's database. So why would you bother telling the spider to do what it's going to do anyway?

Now, if you want to prevent a search engine spider from crawling and indexing a page, use a robots.txt file and save yourself the unnecessary weight in the (X)HTML code.

 

take a look at what Matt Cutts says about ways to manage what the bots crawl and don't crawl, you should find it imformative.

http://googlewebmastercentral.blogspot.com/2008/01/remove-your-content-from-google.html

 

Do these tags need to be there on a site to get indexed??

<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW"><META NAME="GOOGLEBOT" CONTENT="INDEX, FOLLOW">

 

No not at all.