Some posts/pages of wordpress blo blocked from googlebot, but robots.txt allows. How

Discussion in 'Google' started by medinewsdigest, May 10, 2011.

0
  1. #1
    After I test my robots.txt file in the webmaster tools, test says crawling by google bots OK.
    It also says that the site looks like a directory and did not pursue checking within the posts/pages....
    How can I make sure ALL my posts on this wordpress blog are crawlable?

    Are there more than one robots.txt? I only see one in my files from my cpanel
     
    medinewsdigest, May 10, 2011 IP
  2. tipoo

    tipoo Peon

    Messages:
    210
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    there must be only single robots.txt file in root folder.
    past the file here to review.
    try to fetch a post in wmt.
    post your site url here to review.
     
    tipoo, May 10, 2011 IP
  3. saqib889

    saqib889 Peon

    Messages:
    1,291
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Well do one thing to make sure you are true about what you are saying.

    site:posturl

    run this on google and see if your url is indexed or not.
     
    saqib889, May 10, 2011 IP
  4. bogs

    bogs Active Member

    Messages:
    2,142
    Likes Received:
    16
    Best Answers:
    0
    Trophy Points:
    80
    #4
    if you want to make all of your page crawlable then just remove the robot.txt..
     
    bogs, May 11, 2011 IP