1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Get rid of the meta name="robots" content="index, follow" tag?

Discussion in 'Search Engine Optimization' started by misohoni, May 20, 2004.

  1. #1
    Seen on other SEO forums that they recommend removing it as it can hinder the bot.

    Does the tag cause any harm?
    SEMrush
     
    misohoni, May 20, 2004 IP
    SEMrush
  2. Old Welsh Guy

    Old Welsh Guy Notable Member

    Messages:
    2,699
    Likes Received:
    291
    Best Answers:
    0
    Trophy Points:
    205
    #2
    The robots tag is a limiting tag, it has no positive effect. By that I mean that it is really an exclusion of the robots instruction, so, if you do not intedn restricting the robot, there is really no point in it being there IMO.
     
    Old Welsh Guy, May 20, 2004 IP
  3. ephricon

    ephricon Peon

    Messages:
    250
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #3
    I agree completely. I use either the meta or robots.txt to block bots when desired, but that's about it. I do not have any tags asking to be indexed and have never really noticed any kind of hinderance - thus I really don't think they are necessary at all.
     
    ephricon, May 20, 2004 IP
  4. TechEvangelist

    TechEvangelist Guest

    Messages:
    919
    Likes Received:
    140
    Best Answers:
    0
    Trophy Points:
    133
    #4
    "index, follow" is the default, so it makes no sense to repeat it by adding a meta tag.

    Craig
     
    TechEvangelist, May 20, 2004 IP
  5. misohoni

    misohoni Notable Member

    Messages:
    1,717
    Likes Received:
    32
    Best Answers:
    0
    Trophy Points:
    200
    #5
    But it's not doing any real harm then, just using up a few kb's??

    Surely some robots aren't as clever as Google and need a bit of guidance?
     
    misohoni, May 20, 2004 IP
  6. Bernard

    Bernard Well-Known Member

    Messages:
    1,604
    Likes Received:
    107
    Best Answers:
    0
    Trophy Points:
    185
    #6
    I know of no spider/bot that requires that tag in order to crawl a site.
     
    Bernard, May 20, 2004 IP
  7. Such Great Heights

    Such Great Heights Peon

    Messages:
    715
    Likes Received:
    19
    Best Answers:
    0
    Trophy Points:
    0
    #7
    No real harm, but you might have another instance of a keyword at the 98KB mark of the page and if you have unessasary code at the top it might get pushed to the 101KB mark. Then when Google or whoever pays attention to only the first 100KB then you lose one more keyword mention.

    Of course this is a little extreme as an example, but why let it have a chance of happening? ;)

    It's like having one operating system on your PC and telling the computer to ask you which OS to boot up everytime you boot. Just an extra button to push, or in the spider's case just another line of text to read.
     
    Such Great Heights, May 20, 2004 IP
  8. briguy

    briguy Well-Known Member

    Messages:
    691
    Likes Received:
    46
    Best Answers:
    0
    Trophy Points:
    170
    #8
    briguy, Jan 23, 2007 IP
  9. gerer

    gerer Peon

    Messages:
    136
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    it works for certain site and not for others
     
    gerer, Jul 21, 2007 IP
  10. bdude

    bdude Peon

    Messages:
    124
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #10
    I think the point of index, follow is when you exclude a group of robots with robots.txt and then want to let one or two etc in
     
    bdude, Jul 21, 2007 IP
  11. Bernard

    Bernard Well-Known Member

    Messages:
    1,604
    Likes Received:
    107
    Best Answers:
    0
    Trophy Points:
    185
    #11
    The last two posts are completely wrong.
     
    Bernard, Jul 22, 2007 IP
  12. elena29

    elena29 Peon

    Messages:
    25
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #12
    Restricting the bots will practically isolate your website from the search engines. Now why would you want to do that (unless you have something to hide).
     
    elena29, Jul 22, 2007 IP
  13. Toopac

    Toopac Peon

    Messages:
    4,451
    Likes Received:
    166
    Best Answers:
    0
    Trophy Points:
    0
    #13
    Exactly its pointless to add that tag.
     
    Toopac, Jul 22, 2007 IP
    samantha pia likes this.
  14. samantha pia

    samantha pia Prominent Member

    Messages:
    4,639
    Likes Received:
    482
    Best Answers:
    0
    Trophy Points:
    310
    #14
    i agrea too, tags are a strange thing and some work and others dont
    :D
     
    samantha pia, Jul 22, 2007 IP
  15. trichnosis

    trichnosis Prominent Member

    Messages:
    13,780
    Likes Received:
    333
    Best Answers:
    0
    Trophy Points:
    300
    #15
    it will not hurt you seo.

    that tag tell search tell search to index the page and follows the links on that page
     
    trichnosis, Jul 22, 2007 IP
  16. extenddigital

    extenddigital Peon

    Messages:
    9
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #16
    I agree. They do not hurt your site. They basically let the robots know where to go. I added it and the site got indexed by google the next day. I use google webmaster tools.

    If it doesn't hurt why not keep it?

    Good Luck,
     
    extenddigital, Jul 26, 2007 IP
  17. Dan Schulz

    Dan Schulz Peon

    Messages:
    6,032
    Likes Received:
    434
    Best Answers:
    0
    Trophy Points:
    0
    #17
    Whoever told you that they will hurt your rankings or prevent your site from being spidered (and indexed) needs to get out of SEO and take up something far less dangerous, like getting a McJob. Seriously.

    Think about it for a second. What is a search engine spider's default action? To crawl Web pages and index them into the search engine's database. So why would you bother telling the spider to do what it's going to do anyway?

    Now, if you want to prevent a search engine spider from crawling and indexing a page, use a robots.txt file and save yourself the unnecessary weight in the (X)HTML code.
     
    Dan Schulz, Jul 26, 2007 IP
  18. nicolat

    nicolat Peon

    Messages:
    13
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #18
    take a look at what Matt Cutts says about ways to manage what the bots crawl and don't crawl, you should find it imformative.

    http://googlewebmastercentral.blogspot.com/2008/01/remove-your-content-from-google.html

    :D
     
    nicolat, Feb 5, 2008 IP
  19. Dr@gon

    Dr@gon Peon

    Messages:
    66
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #19
    Do these tags need to be there on a site to get indexed??

    <META NAME="ROBOTS" CONTENT="INDEX, FOLLOW"><META NAME="GOOGLEBOT" CONTENT="INDEX, FOLLOW">
     
    Dr@gon, Jan 6, 2009 IP
  20. Toopac

    Toopac Peon

    Messages:
    4,451
    Likes Received:
    166
    Best Answers:
    0
    Trophy Points:
    0
    #20
    No not at all.
     
    Toopac, Jan 6, 2009 IP