robot.txt

Discussion in 'Search Engine Optimization' started by ravensoft, Feb 21, 2011.

0
  1. #1
    How does one go about creating a robots.txt file and placing it in a web site?
     
    ravensoft, Feb 21, 2011 IP
  2. clive2

    clive2 Peon

    Messages:
    343
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Default robots.txt file contains
    User-agent: *
    Disallow: /cgi-bin/

    sitemap: abc.com/sitemap.xml

    Place the above info in notepad and save the file as robots.txt and upload it at root level.
     
    clive2, Feb 21, 2011 IP