Which is the best Robots.txt file

Discussion in 'Search Engine Optimization' started by shashank4u, Sep 8, 2007.

0
  1. #1
    Hi guys

    There are a lot of Robots.txt files available around ....could you guys discuss what is the best robots.txt and why...with samples ..

    Useful answers will be appreciated.

    Thanks
     
    shashank4u, Sep 8, 2007 IP
  2. qazu

    qazu Well-Known Member

    Messages:
    1,834
    Likes Received:
    62
    Best Answers:
    0
    Trophy Points:
    165
    #2
    There is only one robots.txt file that will be specific to your web site. It's the one you make. Check robotstxt.org for information on making a robots.txt file.
     
    qazu, Sep 8, 2007 IP
  3. WPOSolutions

    WPOSolutions Active Member

    Messages:
    433
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    78
    #3
    A robots.txt file is one the you create to instruct robots and crawlers as to were within your domain they can access.

    Its a simple Permit or disallow structure.
     
    WPOSolutions, Sep 8, 2007 IP
  4. speda1

    speda1 Well-Known Member

    Messages:
    374
    Likes Received:
    10
    Best Answers:
    0
    Trophy Points:
    108
    #4
    You don't need a robots.txt file at all. Only use one if you need to block a bot from part of your site.
     
    speda1, Sep 8, 2007 IP
  5. Zippo

    Zippo Peon

    Messages:
    311
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #5
    robots.txt its a file to have "the work finished", but isnt very important in SEO
    I own aprox 20 websites and i am not using robots.txt at all. With or without, thats not a point in SEO
     
    Zippo, Sep 9, 2007 IP