Robots text File?

Discussion in 'Search Engine Optimization' started by hst, Mar 23, 2008.

0
  1. #1
    How do I create a Robots text file?
     
    hst, Mar 23, 2008 IP
  2. 5starpix

    5starpix Notable Member

    Messages:
    6,380
    Likes Received:
    83
    Best Answers:
    0
    Trophy Points:
    235
    #2
    5starpix, Mar 23, 2008 IP
  3. choicefreedom2000

    choicefreedom2000 Peon

    Messages:
    374
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #3
    My dear Friend

    You can generate free of cost here "http://webtools.live2support.com/se_robots.php"


    Cool Mate and Enjoy your day
     
    choicefreedom2000, Mar 24, 2008 IP
  4. Tugeza

    Tugeza Well-Known Member

    Messages:
    208
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    128
    #4
    Simplest robots.txt file contains:

    User-agent: *
    Disallow:

    That's it.
     
    Tugeza, Mar 24, 2008 IP
  5. seoservice1

    seoservice1 Well-Known Member

    Messages:
    1,401
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    130
    #5
    robot.txt file contain
    User-agent: *
    Disallow: / (To exclude all robots from the entire server)

    To exclude all robots from part of the server

    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /tmp/
    Disallow: /junk/

    Thanks and enjoy for the day
     
    seoservice1, Mar 24, 2008 IP