1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

does my robots.txt block my URLs?

Discussion in 'robots.txt' started by wijasenna, Apr 9, 2012.

  1. #1
    just found this on my webmasters tools -- crawler access:

    Blocked URLs: 140
    Downloaded: 20 hours ago
    Status: 200 (success)

    what does it mean? does my robots.txt block my urls to be crawled by spiders? if so, how to fix it?

    thanks!!
     
    wijasenna, Apr 9, 2012 IP
  2. tiffanywilliams12i2

    tiffanywilliams12i2 Peon

    Messages:
    164
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    its means url not accepting the robots.
     
    tiffanywilliams12i2, Apr 10, 2012 IP
  3. wijasenna

    wijasenna Active Member

    Messages:
    375
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    58
    #3
    how to deal with it? what kind of robots.txt that allows all spiders to index all of my articles?
     
    wijasenna, Apr 11, 2012 IP
  4. carlhenson

    carlhenson Peon

    Messages:
    45
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    You will need to add only:

    and remove some disallows that you do not use.
     
    carlhenson, Apr 11, 2012 IP