Bad?

Discussion in 'robots.txt' started by swiftmed, Feb 25, 2007.

Thread Status:
Not open for further replies.
  1. #1
    Isnt disallowing bots to crawl your website a bad thing? Or do people do it to conserve bandwidth? I have a forum and my robots.txt is empty, causing no problem, of course a lot of bandwidth gets eaten but I can afford that.
     
    swiftmed, Feb 25, 2007 IP
  2. Katy

    Katy Moderator Staff

    Messages:
    3,490
    Likes Received:
    513
    Best Answers:
    7
    Trophy Points:
    355
    #2
    If people don't want their images to be listed in google for example, they disallow the images/ directory. I know many people disallow their cgi-bin too.
     
    Katy, Feb 27, 2007 IP
  3. swiftmed

    swiftmed Peon

    Messages:
    1,070
    Likes Received:
    26
    Best Answers:
    0
    Trophy Points:
    0
    #3
    ah thats a good idea. i always wondered how i could disallow my images for image search's. if i was to disallow my images folder, would tha totally stop my images from appearing in image search engines or not?
     
    swiftmed, Feb 27, 2007 IP
  4. dotbeyond.com

    dotbeyond.com Well-Known Member

    Messages:
    212
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    110
    #4
    dotbeyond.com, May 1, 2007 IP
  5. kgrad

    kgrad Well-Known Member

    Messages:
    2,414
    Likes Received:
    82
    Best Answers:
    0
    Trophy Points:
    130
    #5
    It actually depends on your purpose, I have had a lot of sites made, for internal training purposes, so there is no need for search engines to crawl them.
     
    kgrad, May 2, 2007 IP
Thread Status:
Not open for further replies.