What this robot.txt will do ? Please help !!

Discussion in 'robots.txt' started by Rajnish357, Jan 23, 2012.

0
  1. #1
    Please Help me understand this content of robot.txt file

    User-agent: *
    Allow: /
    Disallow: /category/education/
    Disallow: /category/jobs/
    Disallow: /category/business/

    I want to know what this will do ??


    I have a WordPress Blog and I want to all robots on first page of category http://www.digmlm.com/category/education but not on /page/2, /page/3, /page/4 and So on

    But I think I will use above code in robot.txt file it will all stop indexing article !!!
     
    Rajnish357, Jan 23, 2012 IP
  2. tiffanywilliams12i2

    tiffanywilliams12i2 Peon

    Messages:
    164
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    tiffanywilliams12i2, Apr 10, 2012 IP
  3. carlhenson

    carlhenson Peon

    Messages:
    45
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    You can use:

    Without the trailing slashes, I think that will do what you want.
     
    carlhenson, Apr 11, 2012 IP