Robots.txt file vs robots tag

Discussion in 'robots.txt' started by vinug, May 21, 2008.

0
  1. #1
    Hi DP memmbers,

    Which is the best practise ?
    Using robots.txt file or using robots tag between head tags.
     
    vinug, May 21, 2008 IP
  2. shacow

    shacow Active Member

    Messages:
    339
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    60
    #2
    I would suggest robots.txt
     
    shacow, May 21, 2008 IP
  3. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #3
    I too prefer using robots.txt as you can allow or disallow the folder/files or crawler from the robots.txt. But if you use robot tags, you need to put this on individual pages in order to disallow pages and you can't disallow any folder using robots tag....:)
     
    manish.chauhan, May 21, 2008 IP