Should I deny access to my site forum using googlebot?

Discussion in 'robots.txt' started by Yukio, Jun 21, 2005.

  1. pwaring

    pwaring Well-Known Member

    Messages:
    846
    Likes Received:
    25
    Best Answers:
    0
    Trophy Points:
    135
    #21
    As I've already mentioned (and I believe other people have as well), I don't like seeing those 404 errors because it makes me think there's a broken link somewhere, either on my site or someone else who is linking to me. Same goes for that stupid favicon.ico file that everyone requests - I uploaded a very boring icon just to stop hundreds of 404s clogging up my logs.

    Because it's quicker to type 'touch robots.txt'. :p
     
    pwaring, Jun 29, 2005 IP
  2. nevetS

    nevetS Evolving Dragon

    Messages:
    2,544
    Likes Received:
    211
    Best Answers:
    0
    Trophy Points:
    135
    #22
    I stick at least a "allow everything" robots.txt file up to slim down on my 404 count in my stats. The annoying bots don't follow the rules, but I have a few uses for robots.txt - like stay out of my includes directory, leave my cgi dir. alone, etc.
     
    nevetS, Jun 29, 2005 IP
    minstrel likes this.