Is this robots.txt blocking Slurp ?

Discussion in 'robots.txt' started by Percept, Feb 12, 2006.

  1. #1
    Hi,

    I got the following robots.txt file from a robots.txt generatoir:

    
    # All robots will spider the domain
    User-agent: *
    Disallow:
    
    # Disallow directory /admanager/
    User-agent: *
    Disallow: /admanager
    
    # Disallow directory /wp-admin/
    User-agent: *
    Disallow: /wp-admin
    
    # Disallow directory /wp-includes/
    User-agent: *
    Disallow: /wp-includes
    
    # Disallow directory /mag/
    User-agent: *
    Disallow: /mag
    
    Code (markup):
    Google has thousands of pages indexed, so does MSN but Yahoo falls behind with only 133 pages in the last week. Could it have something to do with this robots.txt ?
     
    Percept, Feb 12, 2006 IP
  2. Jim bob 9 pants

    Jim bob 9 pants Peon

    Messages:
    890
    Likes Received:
    20
    Best Answers:
    0
    Trophy Points:
    0
    #2
    have a look here ROBOT.TXT VALIDATION

    Worth a try - dont think you need User-agent: * each time though

    Jamie
     
    Jim bob 9 pants, Feb 12, 2006 IP
  3. just-4-teens

    just-4-teens Peon

    Messages:
    3,967
    Likes Received:
    168
    Best Answers:
    0
    Trophy Points:
    0
    #3
    here is all you need

    the above the allows search robots to index the entire site exept from the directories listed.
     
    just-4-teens, Feb 12, 2006 IP
  4. mussolinihitler

    mussolinihitler Peon

    Messages:
    258
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #4
    i would recommend what "just-for-teens" had said too. Why do you have to make this robots.txt so long ??
     
    mussolinihitler, May 7, 2006 IP