Robiots.txt will not stop Google from listing your site in the index.

Discussion in 'Google' started by Barre Tire, Oct 5, 2009.

  1. #1
    After watching the new video from Google below you will see why. I also got from the video that the examples given are rock solid proof of how much Google relies on links and anchor text

    http://www.youtube.com/watch?v=KBdEwpRQRD0

    What are your thoughts?
     
    Barre Tire, Oct 5, 2009 IP
  2. sarirejo

    sarirejo Well-Known Member

    Messages:
    878
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    108
    #2
    I have block all bot and google still list my site on their engine. But not cached.
     
    sarirejo, Oct 6, 2009 IP
  3. MarketerMac

    MarketerMac Peon

    Messages:
    35
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    I'd guess that what sarirejo is doing is the only solution for keeping them from indexing any part of your site..
     
    MarketerMac, Oct 6, 2009 IP
  4. maltadude

    maltadude Peon

    Messages:
    24
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    If you have your urls already in Google's index ... using a robots.txt to block them will not remove them from the index, but you're simply instructing Google not to crawl them again. That's why you don't see the cache any more!

    To remove a url from the index, there are other methods.
     
    maltadude, Oct 6, 2009 IP
  5. rizwanrajput

    rizwanrajput Active Member

    Messages:
    51
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    91
    #5
    video not playing
     
    rizwanrajput, Oct 6, 2009 IP
  6. banless

    banless Peon

    Messages:
    1,745
    Likes Received:
    217
    Best Answers:
    0
    Trophy Points:
    0
    #6
    The only way to get the url's removed is to add a noindex tag to the page that you want removed, and make sure that the page is still live on your server. But even when you do this it can still take forever for google remove those pages, I have pages still showing in the index even though they have a noindex tag, in other words google will remove them when they feel like it.
     
    banless, Oct 6, 2009 IP
  7. theapparatus

    theapparatus Peon

    Messages:
    2,925
    Likes Received:
    119
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Just to clarify, that's a meta noindex tag as discussed in the video, not a noindex tag in the link. It was stated in the video that even with the disallow in the robots.txt file, the page may still be listed in Google. The only way to remove it completely from the index was either through Google Webmaster Tools or via a meta noindex tag on that page.
     
    theapparatus, Oct 6, 2009 IP