The report in my google adsense account shows that I have some blocked url. This is an instruction on what I should do to enable them to crawl this url. I am using wordpress so I install a plugin to get a robot.txt. My question is what should I write in my robot.txt. Should I all all robots to crawl my url or should I allow only googlebot? I am kinda confused.
This guide will help you. http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress#Robots.txt_Optimization You should allow all robots. Don't worry about allowing bad robots because they are so bad that they will not obey robots.txt