How to set right crawler errors?

Discussion in 'robots.txt' started by bluestar, Aug 1, 2011.

0
  1. #1
    In my adsense access and authorization there are 150 crawler errors and the reason is blocked Robots.txt File. Anyone pls tell me how to resolve this issue. 
     
    bluestar, Aug 1, 2011 IP
  2. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #2
    Just add following code just before your robots.txt instructions:

    User-agent: Mediapartners-Google*
    Disallow:

    This would allow your adsense crawlers to crawl each and every section of your website.
     
    manish.chauhan, Aug 2, 2011 IP
  3. bluestar

    bluestar Well-Known Member

    Messages:
    288
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    121
    #3
    Thanks mate... i did as you suggested. Thanks for your advice!
     
    bluestar, Aug 2, 2011 IP
  4. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #4
    Your welcome Bluestar:)
     
    manish.chauhan, Aug 2, 2011 IP
  5. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #5
    Your welcome Bluestar:)
     
    manish.chauhan, Aug 2, 2011 IP
  6. bluestar

    bluestar Well-Known Member

    Messages:
    288
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    121
    #6
    I added the following code but still i get crawling error for http://slurke.com/search

    User-agent: Mediapartners-Google
    Disallow:
     
    bluestar, Aug 9, 2011 IP
  7. bluestar

    bluestar Well-Known Member

    Messages:
    288
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    121
    #7
    I am finding crawling error for webcache.googleusercontent.com. Pls tell me what code i should add in robots.txt file.
     
    bluestar, Aug 10, 2011 IP
  8. guyksf

    guyksf Peon

    Messages:
    2
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Want to see DVD files on your iPhone? Well, I’d like to share my simple way to do this with you. Many people prefer to watch their favorite videos on iPhone. They usually use Iphone Video Converter For Mac to converter various videos to the right format iPhone supports. But the problem is that it can not support DVD input file format. Well, you do not need to worry about that any more. DVD to iPhone Converter For Mac can solve your problem easily. With this magic APP, you can enjoy your videos very much in highly outputting quality. Why not to have a try?
     
    guyksf, Aug 25, 2011 IP
  9. jhon786

    jhon786 Peon

    Messages:
    62
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    thank you very much for the information.
     
    jhon786, Aug 25, 2011 IP