Is robots.txt required?

Discussion in 'robots.txt' started by surekhaweb, Dec 24, 2006.

  1. #1
    Is robots.txt required?
     
    surekhaweb, Dec 24, 2006 IP
  2. Monty

    Monty Peon

    Messages:
    1,363
    Likes Received:
    132
    Best Answers:
    0
    Trophy Points:
    0
    #2
    No, it's not.
    But it's good practice to have one (even empty) because when bots come on your site, they look for it, and if they don't find it, you got a 404 in your logs.
     
    Monty, Dec 24, 2006 IP
  3. mani

    mani Peon

    Messages:
    679
    Likes Received:
    16
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Almost every website have their robots.txt because it provides the flexibility to control the search engine spiders...

    You can dissalow to index some directories..
    you can disallow your whole site if it is under construction ...

    many more advantages...

    But robots.txt is not REQUIRED , your website will run without it
     
    mani, Dec 24, 2006 IP
  4. surekhaweb

    surekhaweb Banned

    Messages:
    504
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    if i will add blank "robots.txt" then is it harm my ranking or any dis advantage?
     
    surekhaweb, Dec 24, 2006 IP
  5. slayer

    slayer Well-Known Member

    Messages:
    1,926
    Likes Received:
    85
    Best Answers:
    0
    Trophy Points:
    165
    #5
    well it can harm your ranking if you add important pages or directories to it for robots not to crawl, but if you know what folders/files to add than everything will be running good and your site will be safer
     
    slayer, Dec 24, 2006 IP
  6. surekhaweb

    surekhaweb Banned

    Messages:
    504
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    i add blank www.mysite.com/robots.txt file then?
     
    surekhaweb, Dec 24, 2006 IP
  7. mani

    mani Peon

    Messages:
    679
    Likes Received:
    16
    Best Answers:
    0
    Trophy Points:
    0
    #7
    do whatever you like to do with robots.txt but never ever try to insert the following line if your site is online

    Dissallow: /

    This will stop the search engines to index any part of your site :D

    This is useful for under construction sites..
     
    mani, Dec 24, 2006 IP
  8. surekhaweb

    surekhaweb Banned

    Messages:
    504
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    What Is 404? and What is the role of 404?
     
    surekhaweb, Dec 24, 2006 IP
  9. Monty

    Monty Peon

    Messages:
    1,363
    Likes Received:
    132
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Monty, Dec 24, 2006 IP