Robots txt file

Discussion in 'Google' started by ashish123, Jan 30, 2009.

0
  1. #1
    We use Robot txt file , so that google knows these pages need not be crawled...
    But i dont understand:- do we really need that google do not crawl our pages.. i mean everyone wants to be crawled then for which type of pages anyone can mention inRobot.txt file..

    Can anyanybody help me on this?
     
    ashish123, Jan 30, 2009 IP
  2. Kirsty H

    Kirsty H Banned

    Messages:
    178
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Well I use robots.txt to keep google out of the places where I do not want it crawling, for example on a forum you might not want it to keep crawling admincp.php as it's just a waste of bandwidth. You can use it to allow or decline all the bots from different areas. :)
     
    Kirsty H, Jan 30, 2009 IP
  3. zodiac

    zodiac Peon

    Messages:
    2,661
    Likes Received:
    82
    Best Answers:
    0
    Trophy Points:
    0
    #3
    it doesn't tell the bot to not crawl the pages,it tells them not to index them.
     
    zodiac, Jan 30, 2009 IP