In my adsense account in "revenue optimization" i have crawl errors then when i click "fix crawl errors" then this.. Blocked Urls Error http://www.rechargeoverload.in/search/label Robot Denied My robots.txt: User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: http://www.rechargeoverload.in/atom.xml?redirect=false&start-index=1&max-results=500 Sitemap: http://www.rechargeoverload.in/atom.xml?redirect=false&start-index=501&max-results=500
Go to Blogger Settings > Search preferences > Custom robots.txt > edit > Remove "Disallow: /search" > Save. This should help the crawlers crawl complete site including labels. Hope this helps...
Here is all information about Robots.txt https://moz.com/learn/seo/robotstxt There are a lot of sites available that create the robots.txt file online, just you have to provide the input. Check the example https://en.ryte.com/free-tools/robots-txt-generator/ hope this will help you..