Robots.txt file, do i even need it?

Discussion in 'robots.txt' started by ASCFulfillment, Dec 14, 2006.

0
  1. #1
    If the title didn't already give it away...yes I am a noob...

    My question is if I think 99% of the content on my site is relevant, and my understanding of the robots.txt file is to keep bots out...do i even need it?

    TIA:eek:
     
    ASCFulfillment, Dec 14, 2006 IP
  2. weknowtheworld

    weknowtheworld Guest

    Messages:
    306
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #2
    weknowtheworld, Dec 14, 2006 IP
  3. imgtube

    imgtube Well-Known Member

    Messages:
    562
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    110
    #3
    I have never used one.
     
    imgtube, Dec 16, 2006 IP
  4. explorer

    explorer Well-Known Member

    Messages:
    463
    Likes Received:
    40
    Best Answers:
    0
    Trophy Points:
    110
    #4
    No, you don't need it.

    It's helpful to include one though because bots will look for one and you'll get errors in your logs when they can't find it.

    Just make a file in a text editor, paste these two lines into it:

    User-agent: *
    Disallow:

    and save this file as it as robots.txt. Then all you need to do is upload your robots.txt to your site's root directory (often called public_html) and you'll have a robots.txt file that allows bots to spider all of your site.
     
    explorer, Dec 26, 2006 IP