1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Should I deny access to my site forum using googlebot?

Discussion in 'robots.txt' started by Yukio, Jun 21, 2005.

0
Page 2 of 2
  1. pwaring

    pwaring Well-Known Member

    Messages:
    846
    Likes Received:
    25
    Best Answers:
    0
    Trophy Points:
    135
    #21
    As I've already mentioned (and I believe other people have as well), I don't like seeing those 404 errors because it makes me think there's a broken link somewhere, either on my site or someone else who is linking to me. Same goes for that stupid favicon.ico file that everyone requests - I uploaded a very boring icon just to stop hundreds of 404s clogging up my logs.

    Because it's quicker to type 'touch robots.txt'. :p
     
    pwaring, Jun 29, 2005 IP
  2. nevetS

    nevetS Evolving Dragon

    Messages:
    2,544
    Likes Received:
    211
    Best Answers:
    0
    Trophy Points:
    135
    #22
    I stick at least a "allow everything" robots.txt file up to slim down on my 404 count in my stats. The annoying bots don't follow the rules, but I have a few uses for robots.txt - like stay out of my includes directory, leave my cgi dir. alone, etc.
     
    nevetS, Jun 29, 2005 IP
    minstrel likes this.
Page 2 of 2