Indexing problems! - to robots or not to robots

Discussion in 'Google' started by Kingdom85, Mar 30, 2011.

0
  1. #1
    Hi,

    I've got a huge number of URL's that are currently being indexed by Google in particular that I want to remove from the index. They create a huge concern from a duplicate content perspective and some are having an impact on my users journey via specific keywords.

    My thoughts were to implement "noindex" meta tags on all these pages to first remove them from the index and then to add the directory that a majority of these pages sit in to the robots.txt file.

    Another course of action would be to simply add the directory to the robots.txt file and hope that the URL's would drop out of the index that way.

    Which is the best way to do this so that I can be sure all these URL's will drop out of the index?

    Thanks,
     
    Kingdom85, Mar 30, 2011 IP
  2. tjsocr22

    tjsocr22 Active Member

    Messages:
    1,251
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    80
    #2
    The "noindex" meta tags will work regardless of if you also add the links to your robots.txt file as well. As soon as Google Re-Crawls your website they will immediately see the meta tags and take those pages out of the index.
     
    tjsocr22, Mar 30, 2011 IP
  3. dorothykinder

    dorothykinder Peon

    Messages:
    295
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Do you want to make the pages not to be crawled by Google, then no index can be implemented, but will Gogole remove it from its own indexed servers??
     
    dorothykinder, Mar 30, 2011 IP
  4. bogs

    bogs Active Member

    Messages:
    2,142
    Likes Received:
    16
    Best Answers:
    0
    Trophy Points:
    80
    #4
    i prefer using robots.txt.. the shorter the codes on your pages more easy for bots to crawl and index.. compiling them in one robots.txt is more easier to recognize those pages that you don't want to index..
     
    bogs, Mar 30, 2011 IP
  5. John_Marshal

    John_Marshal Peon

    Messages:
    221
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #5
    update your robot.txt file for tracking.... its easy to get the crawling disable by help of that...
     
    John_Marshal, Mar 31, 2011 IP
  6. aileenwuornos

    aileenwuornos Peon

    Messages:
    288
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #6
    robot.txt is more beneficial then nofollow meta tags. your pages will be deindexed by search engines.
     
    aileenwuornos, Mar 31, 2011 IP
  7. SimoneJ3

    SimoneJ3 Peon

    Messages:
    26
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Robots give better results.
     
    SimoneJ3, Mar 31, 2011 IP
  8. vladthebest

    vladthebest Member

    Messages:
    83
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    28
    #8
    I'd advise to just add the robot.txt, its a lot easier and effective..
     
    vladthebest, Mar 31, 2011 IP
  9. Kingdom85

    Kingdom85 Peon

    Messages:
    178
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Thanks for the feedback guys, this is really helpful.
     
    Kingdom85, Mar 31, 2011 IP
  10. enriquerojas

    enriquerojas Active Member

    Messages:
    312
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    55
    #10
    I laugh when I read people saying - "the shorter the codes on your pages more easy for bots to crawl and index", this no longer truth, stop repeating what you read around. The crawlers are smarter in 2011 than 2005. Crawlers now days respect more the robots.txt rules, they download more bytes in-order to understand and categorized better a page. Crawlers are beginning to understand a real website vs a adsence website.

    You should mark with nofollow,noindex the pages you want to remove, also add the directory on your robots.txt file to ensure that they wont visit again. But only add the directory once they remove the pages other wise they will keep the pages for a while since the robots.txt file is not letting the crawlers get in to that folder.
     
    enriquerojas, Mar 31, 2011 IP
  11. C.Rebecca

    C.Rebecca Active Member

    Messages:
    1,401
    Likes Received:
    11
    Best Answers:
    1
    Trophy Points:
    65
    #11
    C.Rebecca, Mar 31, 2011 IP