robots.txt?

Discussion in 'robots.txt' started by pzn, Apr 19, 2006.

0
  1. #1
    I am interested in this. I was wondering what robots.txt does and how to use it on my site?

    Thanks
     
    pzn, Apr 19, 2006 IP
  2. BILZ

    BILZ Peon

    Messages:
    1,515
    Likes Received:
    62
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Google it. Get some basic info and then ask some more specific questions.
     
    BILZ, Apr 19, 2006 IP
  3. INV

    INV Peon

    Messages:
    1,686
    Likes Received:
    101
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Robot.txt is a file that the major search engine bots read from your directory.

    This file has allow and disallow options to tell the bot what pages or folders not to crawl and index.

    All you have to do is make a blank .txt file in your root folder and name it robots.

    You can use this http://www.mcanerin.com/EN/search-engine/robots-txt.asp to generate yourself the stuff to put in the robots.txt. You dont have to allow every bot though as they welcome themselfs but its good to disallow the robot from crawling stuff you dont wish it to.
     
    INV, Apr 20, 2006 IP
  4. Jean-Luc

    Jean-Luc Peon

    Messages:
    601
    Likes Received:
    30
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Jean-Luc, Apr 20, 2006 IP
  5. Mystique

    Mystique Well-Known Member

    Messages:
    2,579
    Likes Received:
    94
    Best Answers:
    2
    Trophy Points:
    195
    #5
    Definitely Jean-Luc, that site can be considered the authority on robots.txt and where you can find everything about it ;)
     
    Mystique, Apr 20, 2006 IP