URL Detail Last Calculated http://www.pokeraffiliated.com/blog/ URL restricted by robots.txt [?] Feb 3, 2007 http://www.pokeraffiliated.com/blog/2007/01 URL restricted by robots.txt [?] Feb 4, 2007 http://www.pokeraffiliated.com/blog/ni...g/putting-together-a-marketing-strategy URL restricted by robots.txt [?] Feb 6, 2007 http://www.pokeraffiliated.com/blog/ni.../putting-together-a-marketing-strategy/ URL restricted by robots.txt [?] Feb 4, 2007 http://www.pokeraffiliated.com/blog/po...-to-choose-the-right-affiliate-program/ URL restricted by robots.txt [?] Feb 5, 2007 http://www.pokeraffiliated.com/blog/poker-hysterics/dutch-boyd-is-on-crack/ URL restricted by robots.txt [?] Feb 3, 2007 http://www.pokeraffiliated.com/blog/po...ish-takes-a-bad-beat-this-is-hilarious/ URL restricted by robots.txt [?] Jan 26, 2007 http://www.pokeraffiliated.com/blog/po...2-this-guy-is-the-biggest-donkey-alive/ URL restricted by robots.txt [?] Jan 27, 2007 http://www.pokeraffiliated.com/blog/seo/backlinks URL restricted by robots.txt [?] Feb 6, 2007 http://www.pokeraffiliated.com/blog/seo/backlinks/ URL restricted by robots.txt [?] Feb 3, 2007 http://www.pokeraffiliated.com/blog/seo/why-content-matters-to-poker-affiliates/ URL restricted by robots.txt [?] Feb 5, 2007 http://www.pokeraffiliated.com/blog/vi...inkbaiting-and-link-baiting-strategies/ URL restricted by robots.txt [?] Feb 4, 2007 now is it because my site is under http://www.pokeraffiliated.com/blog and not using the main domain, or is it something wrong with my sitemap, i used a wordpress sitemap generator plugin, and my robots.txt is the basic User-agent: * Disallow: Im not sure why these errors are coming up, if anyone can help it would be appreciated.
i have the same experience with this, but not all my page are restricted, usually only two pages, but next crawl, succeeded
You have User-agent: * Disallow: Code (markup): in your robots.txt file. From what I read once on Google's site add User-agent: Mediapartners-Google* Disallow: Code (markup): to your robots.txt file to make them happy and that might fix the problem hopefully. You might check out How do I use the robots.txt file analysis tool for a bit more information
I made a through search on google and found solution that you may create a robots.txt file with the help of google webmaster tool like this one: User-agent: * Allow: / User-agent: Mediapartners-Google Allow: / User-agent: Googlebot Allow: / User-agent: Adsbot-Google Allow: / User-agent: Googlebot-Image Allow: / User-agent: Googlebot-Mobile Allow: / and after downloading that file you required to upload that robots.txt file in your top level of the domain. That's fine.