does my robots.txt block my URLs?

Discussion in 'robots.txt' started by wijasenna, Apr 9, 2012.

0
  1. #1
    just found this on my webmasters tools -- crawler access:

    Blocked URLs: 140
    Downloaded: 20 hours ago
    Status: 200 (success)

    what does it mean? does my robots.txt block my urls to be crawled by spiders? if so, how to fix it?

    thanks!!
     
    wijasenna, Apr 9, 2012 IP
  2. tiffanywilliams12i2

    tiffanywilliams12i2 Peon

    Messages:
    164
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    its means url not accepting the robots.
     
    tiffanywilliams12i2, Apr 10, 2012 IP
  3. wijasenna

    wijasenna Active Member

    Messages:
    375
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    58
    #3
    how to deal with it? what kind of robots.txt that allows all spiders to index all of my articles?
     
    wijasenna, Apr 11, 2012 IP
  4. carlhenson

    carlhenson Peon

    Messages:
    45
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    You will need to add only:

    and remove some disallows that you do not use.
     
    carlhenson, Apr 11, 2012 IP