Please suggest if my robots.text file is right.

Discussion in 'robots.txt' started by claudiajay, Nov 23, 2012.

0
  1. #1
    Hello Frinds:

    Please tell me, is it necessary to put a robots.text file, if yes then what is the simple format?

    Thanks
     
    claudiajay, Nov 23, 2012 IP
  2. johnaustin9

    johnaustin9 Member

    Messages:
    104
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    36
    #2
    Yes Its Necessary Because what is the use of google crawler to visit your signup page of your site. You can block signup page or other necessay pages that you dont want to get crawle by search engine.
    Correct Format of :
    (User-agent:*
    Disallow:)
     
    johnaustin9, Nov 23, 2012 IP
  3. ryan_uk

    ryan_uk Illustrious Member

    Messages:
    3,983
    Likes Received:
    1,022
    Best Answers:
    33
    Trophy Points:
    465
    #3
    No, you don't need it, unless you want to block something specifically.

    A very good website about how to use robots.txt correctly:

    http://www.robotstxt.org
     
    ryan_uk, Nov 23, 2012 IP
  4. kashishjain

    kashishjain Peon

    Messages:
    27
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Dude you always need a robots.txt to stop crawlers from accessing specific pages of your website/blog like login page etc. because these pages create a negative impact on search engines and can lower your rankings.
     
    kashishjain, Nov 24, 2012 IP
  5. claudiajay

    claudiajay Peon

    Messages:
    9
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Thanx dude, i did visit robotstxt.org and learnt the art of robots.xt and a lot more !!;)
     
    claudiajay, Nov 26, 2012 IP