Blogspot.com Now Has Robots.txt File

I found out by accident tonight that Blogspot.com has added an entry to everyone's blog excluding /search files from all search engines. When most of us were forced to upgrade to the new Blogspot a few months ago, blog labels seemed like a good way to group our content into appropriate categories. Now with this new forced entry into robots.txt those new labels have become worthless since they will not be crawled by search engines. Oh well, I guess all these new labels have caused Googlebot to crawl millions of extra pages each day and push there search engine to the limit.

I really hope this is not the first step in limiting other content on blogs owned by Google, like stuff they don't agree with or content they might consider to be too old for indexing. For the past few years when I have check my blogspot robot.txt file I have always seen a blank page, but today there are restrictions on my blog and several other blogs I have randomly checked.

New robots.txt file on blogspot.com blogs

 

Where did you find this. I cannot seem to find it on my Blogger blog.

 

Type in your URL followed by /robots.txt and you will find the file. They are not only blocking Googlebot, but everyone else as well. My guess is that search engines are eating up bandwidth from blogspot.com blogs since they introduced labels to the mix in mass several months ago. Even Google's own blog on blogspot.com has the same language in it's robots.txt file.

 

When I typed that in the result was "Not found Error 404"

 

yeah it their, nicely found

 

that is there

 

robot.txt is something new for me. what do you mean by They are not only blocking Googlebot, but everyone else as well.? thanks!

 

This is now updated by blogger and it now includes


sitemap: if you have submitted

 

Anyone figure out a work around for this? My blog is entirely organized into label categories. there's no way to ftp into blogger? Also, the site map they direct to seems to only be the most recent page of posts. does that mean it won't crawl older posts? Is it wise to set posts per page as max? I feel like this is going to be really bad for rank.

 

i am wondering the same. They now filter all labels and only check updated posts.

but then it means your old post wont be indexed again.

 

I don't think this will effect regular posts unless there is no other way to find your inter pages except via "labels". Most blog templates are set up with all kinds of built in links for previous posts and a smart bot like Google will be able to figure that out. Not to mention every time you post a blog entry on blogger they broadcast that update to hundreds of sites worldwide and those sites and the previous post feature on most templates should keep your pages from becoming orphans in the eyes of Google and other SE's.

 

Right I get that direct links to posts get indexed, but a single post is not nearly as keyword rich as landing on a category. for instance, in addition to a handful of labels I use, every post I make contains a label that corresponds to one of three categories. at the top of my page, I have a little menu bar that offers those three categories. each category is very focused with lots of keyword rich posts and displays all corresponding posts, not just the last ten. so previously, a google search may yield that category instead of my main index. since those categories are no longer indexing, i fear that my pagerank will drop because the main index is not as focused and is limited to 10 posts per page. considering the new robots.txt, is it wise to increase how many posts are displayed per page? to 20 perhaps? By the way, if i'm just completely misunderstanding how search works, please let me know.

 

I need your help.

2 days ago in my robots.txt has appeared following code on my blogspot:
Disabled: /
Means it is not spidered at all. Web is not against google guidline. One week ago I used new blogger functionality to switch all my feed to Feedburner and maybe sitemap is not clear for Googlebot so they put this code. I don't know....

Do you have any idea???

 

you sure its disable or disallow: /

 

ya correct info
following code find in robots.txt

User-agent: *
Disallow: /search
Sitemap:

can someone guide how to edit this robots.txt file.I mean where i find option to edit robots.txt file when i log into my blogger account.

 

This is what I have in robots.txt of my blogspot blog:

User-agent: *
Disallow: /

DO you have any idea how to change it and why google assign this formula?

 

read google.com/webmasters
sitemap using yourblog.blogspot.com/atom.xml

 

This will filter all "tags" for your post. Google is planning to ignore tags in SE. I dont know any way to edit the robot.txt for blogspot.

 

you can edit the robot.txt in google.com/webmasters (CMIIW)

 

you can't edit the file there. I tried doing the same. Saved it.

checked again, it was back to what it was.