Blogspot.com Now Has Robots.txt File

That feature on webmaster tools is only there to help you check your robots.txt file for errors. I don't think you can change a blogger robots.txt file and since it is a free service there really isn't anything a person can do except get there own domain and host their blog there. Labels are a good thing for bloggers, but probably a terrible thing for search engines.

 

Some good news for us that have adsense on our blogs, Blogger has now updated the robots.txt file once again. Now, adsense ads can be relevant on tagged pages since they are now allowing the media bot to view tagged content. This is a positive move in my opinion.

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search

 

Thanks for this info. No wonder, there has been a great content-blend of adsense ads which match my blog.

 

That's correct and when I checked my webmaster in Google, before my label used to get blocked to crawl in search engine but now it working absolutely fine.

Now no one has to worry about to crawl labels in search engine, to check it you can find at webmaster in google. Just by adding your site in Google webmaster and after some weeks you will find that.

 

I found out recently about robot.txt.file and i did what you guys wrote in your posts

and this is what it showed the text of my robots.txt

User-agent:
Mediapartners-Google
Disallow:

User-agent:
*Disallow: /search

Allow: /Sitemap:feeds

 

I have such robotsUser-agent: Mediapartners-GoogleDisallow: User-agent: *Disallow: /searchAllow: /Sitemap:my site.com/feeds/posts/default?orderby=updatedIf I understand correctly this is the prohibition of search and indexing Sitemap.?

 

Thanks for useful information....