Should I even have a robots.txt?

Discussion in 'robots.txt' started by dealtoplay, Jun 3, 2008.

0
  1. #1
    From what I understand, the robots.txt is a very bad thing to have....

    I would very much appreciate feedback about this.

    Thanks!
     
    dealtoplay, Jun 3, 2008 IP
  2. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #2
    Totally wrong....who said robots.txt is wrong.
    Get some information about robots.txt from here....:)
     
    manish.chauhan, Jun 4, 2008 IP
  3. rudikeys

    rudikeys Guest

    Messages:
    56
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Depends on whether you want make a link 'nofollow' or not.
     
    rudikeys, Jun 4, 2008 IP
  4. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #4
    :confused::confused:
     
    manish.chauhan, Jun 5, 2008 IP
  5. DrAnujGupta

    DrAnujGupta Peon

    Messages:
    203
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #5
    it is best for site LOL
     
    DrAnujGupta, Jun 10, 2008 IP
  6. Trusted Writer

    Trusted Writer Banned

    Messages:
    1,370
    Likes Received:
    52
    Best Answers:
    0
    Trophy Points:
    160
    #6
    Whether you want no-follow links or not, to meet web standards you need to have a robots.txt regardless if the only content of the file is

    User-agent: *
    Disallow:

    Meaning all robots and crawlers are welcome.

    Other robots.txt settings can be found through the link provided by manish.chauhan
     
    Trusted Writer, Jun 12, 2008 IP
  7. Dayton Internet Marketing

    Dayton Internet Marketing Peon

    Messages:
    55
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #7
    You should have a robots meta tag for sure... and a robots txt file if you do not want certain things on your server indexed by Google... i have contracts that display on one of my sites so i had to disallow them on a robots txt file because Google indexed them. It is not ban in any way... It is good and if you need one just use this as an example

    http://www.daytoninternetmarketing.com/robots.txt
     
    Dayton Internet Marketing, Jun 12, 2008 IP