what is this mean?

Discussion in 'robots.txt' started by saqistation, Jan 31, 2009.

0
  1. #1
    what is this mean?

    User-agent: *
Disallow:
    Code (markup):

     
    saqistation, Jan 31, 2009 IP
  2. NickR25

    NickR25 Peon

    Messages:
    394
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    0
    #2
    It will disallow access (to whatever directories/pages are listed below the "Disallow:") to whatever bot/crawler obeys the robots.txt file.

    If I wanted to instruct all bots to not crawl the /forum directory on my site, I would have this in my robots.txt file:
    If I wanted to block just Google's crawler, I would put this in:

    Read a lot more here: http://sitemaps.blogspot.com/2006/02/using-robotstxt-file.html
     
    NickR25, Feb 1, 2009 IP
  3. shailendra

    shailendra Peon

    Messages:
    1,225
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    0
    #3
    this means nothing and all the pages will be crawled because one thing is missing in it and that is forward slash (/). Had there been a forward slash after Disallow: then no pages would have been crawled
     
    shailendra, Feb 1, 2009 IP
  4. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #4
    This means you have not restricted any web page of your website from search engine's crawlers.
     
    manish.chauhan, Feb 2, 2009 IP