robots.txt correct or not?

Discussion in 'robots.txt' started by sumitsaurabhmca, Apr 3, 2012.

0
  1. #1
    Hi DP Members,

    I have a website: http://www.neuroseoservices.com/. Please check its robots.txt file. is it correct or not? if it is incorrect, please suggest me how to write it correctly.

    Thanks in advance
     
    sumitsaurabhmca, Apr 3, 2012 IP
  2. ryan_uk

    ryan_uk Illustrious Member

    Messages:
    3,983
    Likes Received:
    1,022
    Best Answers:
    33
    Trophy Points:
    465
    #2
    You're not denying anything, so you don't really need one.

    You're offering SEO services, so shouldn't you know how to do this yourself?
     
    ryan_uk, Apr 3, 2012 IP
  3. Birbalsingh

    Birbalsingh Peon

    Messages:
    17
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Yes Robots.txt is right.

    Thanks
    Birbal
     
    Birbalsingh, Apr 3, 2012 IP
  4. rgba

    rgba Peon

    Messages:
    12
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Here is some useful information: robotstxt.org/robotstxt.html
     
    rgba, Apr 3, 2012 IP
  5. henrywilliams

    henrywilliams Peon

    Messages:
    119
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #5
    yes that's right but you don't have to add allow command because that is what a crawler does by default. Use this code for your robots.txt

    User Agent *
    Disallow
     
    henrywilliams, Apr 6, 2012 IP
  6. Artuurs

    Artuurs Peon

    Messages:
    24
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Incorrect :D
     
    Artuurs, Apr 7, 2012 IP
  7. tiffanywilliams12i2

    tiffanywilliams12i2 Peon

    Messages:
    164
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    whats incorrect about it?
     
    tiffanywilliams12i2, Apr 10, 2012 IP
  8. ryan_uk

    ryan_uk Illustrious Member

    Messages:
    3,983
    Likes Received:
    1,022
    Best Answers:
    33
    Trophy Points:
    465
    #8
    User-agent: *
    Disallow:

    Would allow all robots.

    The OP isn't denying anything, so one is not needed, and robots.txt entry is invalid (but not harmful).
     
    ryan_uk, Apr 12, 2012 IP