Is this a good robots.txt ?

Discussion in 'robots.txt' started by dmi, Jul 16, 2009.

  1. #1
    My robots.txt file goes like this:

    
    User-agent: *
    Disallow:
    
    
    Code (markup):
    Is that okay? Does it block anything?
     
    dmi, Jul 16, 2009 IP
  2. pictureboarduk

    pictureboarduk Well-Known Member

    Messages:
    551
    Likes Received:
    26
    Best Answers:
    0
    Trophy Points:
    140
    #2
    That is telling all the bots to block nothing.

    In this case you typically don't have a robots.txt file at all.
     
    pictureboarduk, Jul 18, 2009 IP
  3. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #3
    You don;t need a robots.txt file if you wish to have these settings.
     
    manish.chauhan, Oct 28, 2009 IP
  4. jasn009

    jasn009 Active Member

    Messages:
    244
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    80
    #4
    if you really want to disallow then try

    User-agent: *
    Disallow: /
    it will disallow all pages to all search engine to crawl,

    if you want to disallow any particular page then put your page name instead of /
     
    jasn009, Nov 4, 2009 IP