robots.txt

Discussion in 'Search Engine Optimization' started by steliosp, Dec 17, 2006.

0
  1. #1
    Hi guys i would like to ask if I really need to have a robots.txt I dont have any pages that i dont want the robots to crawl. So do i need to have one or shall I have a meta like <meta name="robots" content="index, follow" />??

    Many thanks!
     
    steliosp, Dec 17, 2006 IP
  2. stOx

    stOx Notable Member

    Messages:
    6,426
    Likes Received:
    130
    Best Answers:
    0
    Trophy Points:
    230
    #2
    <meta name="robots" content="index, follow" /> will serve no purpose.

    If you don't want to dissalow any page you don't need a robots.txt. But uploading a blank text file called robots.txt will reduce the number of 404 errors you get and keep your server stats nice and neat.
     
    stOx, Dec 17, 2006 IP
  3. whatmanual

    whatmanual Peon

    Messages:
    32
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #3
    whatmanual, Dec 17, 2006 IP
  4. Bondat

    Bondat Peon

    Messages:
    2,397
    Likes Received:
    217
    Best Answers:
    0
    Trophy Points:
    0
    #4
    It's always a must to put a robots.txt if you dont want to crawl some of your pages. ;)
     
    Bondat, Dec 17, 2006 IP