Why Google ofent crawl robots.txt

Discussion in 'Google' started by vietlk, Jan 3, 2011.

  1. #1
    I dont know why GG always crawl robots.txt every day ? This file rarely change.
     
    vietlk, Jan 3, 2011 IP
  2. DoDo Me

    DoDo Me Peon

    Messages:
    2,257
    Likes Received:
    27
    Best Answers:
    0
    Trophy Points:
    0
    #2
    because if it's changes, Google need follow it, and follow it quick

    so, they have to check it everyday to make sure they do not miss it.
     
    DoDo Me, Jan 3, 2011 IP
  3. echovme

    echovme Peon

    Messages:
    65
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    According to google robots.txt is very important file. so it update the information available in this robots .txt file
     
    echovme, Jan 3, 2011 IP
  4. rahu_l_

    rahu_l_ Well-Known Member

    Messages:
    535
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    175
    #4
    i don't have robots.txt :(
     
    rahu_l_, Jan 3, 2011 IP
  5. avpower

    avpower Peon

    Messages:
    434
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #5
    it's ok
    you need to have robots.txt if you need to do something with robots.txt help
     
    avpower, Jan 3, 2011 IP
  6. Balendra Singh

    Balendra Singh Active Member

    Messages:
    109
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    55
    #6
    A robots.txt file restricts access to your site by search engine robots that crawl the web. for more click here
     
    Balendra Singh, Jan 3, 2011 IP
  7. Alfonso_Montenegro

    Alfonso_Montenegro Peon

    Messages:
    158
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #7

    Hmm... I see. I learned another thing today. This forum rocks!
     
    Alfonso_Montenegro, Jan 3, 2011 IP
  8. link@logicspice.com

    link@logicspice.com Active Member

    Messages:
    35
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    91
    #8
    I have a site logicspice.com and some day ago goolge don't crawl my site home page. google remove my site from Google SEPRs . Then I check in webmaster google show robots.txt unreachable. Please help me to get back in google
     
    link@logicspice.com, Jan 3, 2011 IP