Quick Question on a robots.txt file - help!?

Discussion in 'robots.txt' started by Malmsteen, Feb 29, 2008.

  1. #1
    Im almost totally lost when it comes to these types of things
    but what do I put in my robots.txt file, when I want everythings on www.mysite.com crawled, and also everything in
    www.mysite.com/MyStore crawled except this:

    User-agent: *
    Disallow: /admin/
    Disallow: /cont/
    Disallow: /scripts/
    Disallow: /themes/
    Disallow: /web.config

    What does my final robots file look like then? sorry if its a dumb question guys, but I want to be sure..

    Thinking 2 seconds over it, it would be logical that I should use this robots.txt:
    User-agent: *
    Disallow: Mystore/admin/
    Disallow: Mystore/cont/
    Disallow: Mystore/scripts/
    Disallow: Mystore/themes/
    Disallow: Mystore/web.config

    Is this correct?
     
    Malmsteen, Feb 29, 2008 IP
  2. chris265

    chris265 Peon

    Messages:
    702
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    0
    #2
    I think second one is correct not sure but it seems right to me. first one does not look right for some reason
     
    chris265, Feb 29, 2008 IP
  3. Bohol

    Bohol Peon

    Messages:
    2,680
    Likes Received:
    75
    Best Answers:
    0
    Trophy Points:
    0
    #3
    May I know what software you are using?
     
    Bohol, Feb 29, 2008 IP
  4. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #4
    If you are using second robots.txt file:
    User-agent: *
    Disallow: Mystore/admin/
    Disallow: Mystore/cont/
    Disallow: Mystore/scripts/
    Disallow: Mystore/themes/
    Disallow: Mystore/web.config

    then how it is possible to restrict www.mysite.com/admin and other folders from SE robots..
     
    manish.chauhan, Apr 6, 2008 IP