Is Robots.txt Necessary?

Discussion in 'Search Engine Optimization' started by deltron, Jan 4, 2009.

0
  1. #1
    Happy New Year All!

    So I have a site, and on AwStats I always get Robots.txt as a 404 error (file not found).

    Robots.txt is used to restrict google from certain pages correct? There are no pages I want restricted, so is there any harm in having no robots.txt file?
     
    deltron, Jan 4, 2009 IP
  2. silvastar

    silvastar Active Member

    Messages:
    449
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    58
    #2
    Pretty much spot on with your comment regarding pages you want restricted. There isn't a lot
    else you can do with it.

    I can't see any harm in not having a robots txt file if you don't want to restrict pages.
     
    silvastar, Jan 4, 2009 IP
  3. Colbyt

    Colbyt Notable Member

    Messages:
    3,224
    Likes Received:
    185
    Best Answers:
    0
    Trophy Points:
    210
    #3
    The only advantage to having one is that it greatly reduces the 404s that you have to look at.

    BTW, if you do have something you don't want the bad spiders to look at or for, Never place a deny all in the robots.txt file. That is where the bad spiders start probing. Mostly applies to site with a paid download folder.
     
    Colbyt, Jan 4, 2009 IP
  4. deltron

    deltron Active Member

    Messages:
    397
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    73
    #4
    Thanks guys, that helps a lot.
     
    deltron, Jan 4, 2009 IP