Google's Robots.txt Tip

Discussion in 'robots.txt' started by SearchBliss, May 30, 2010.

  1. #1
    Google's John Mueller has stated that Google crawlers rely on the cache of the robots.txt file. This cache is updated approximately every 24 hours.

    So before you add new content that is "disallowed", you need to add it to your robots.txt file for 24 hours before adding the disallowed content to get crawled. This will insure Googlebot has the updated cache of your robots.txt file and obeys the "Disallow" command.
     
    SearchBliss, May 30, 2010 IP
  2. pr0t0n

    pr0t0n Well-Known Member

    Messages:
    243
    Likes Received:
    10
    Best Answers:
    10
    Trophy Points:
    128
    #2
    Interesting... good to know that.
     
    pr0t0n, Jun 3, 2010 IP
  3. assertivemagazine.com

    assertivemagazine.com Greenhorn

    Messages:
    71
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    16
    #3
    How to Disallow from Alexa bot?
     
    assertivemagazine.com, Jun 3, 2010 IP
  4. Victor.Bacon

    Victor.Bacon Guest

    Messages:
    8
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    I had disallowed on my site search pages Disallow: search/ also same way admin pages but Google still managet to index some of them....
     
    Victor.Bacon, Jun 4, 2010 IP
  5. drajat_biantoro

    drajat_biantoro Peon

    Messages:
    27
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Nice info . .still google index my post !!!
     
    drajat_biantoro, Jun 5, 2010 IP
  6. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #6
    This is really great update from John Mueller. This will motivate the webmasters to be more organized and planned in website updates.
     
    manish.chauhan, Jun 13, 2010 IP