|
|
|
|
#1
Jul 13th 2007, 9:44 pm
|
|||
|
|||
|
Blogspot.com Now Has Robots.txt File
I found out by accident tonight that Blogspot.com has added an entry to everyone's blog excluding /search files from all search engines. When most of us were forced to upgrade to the new Blogspot a few months ago, blog labels seemed like a good way to group our content into appropriate categories. Now with this new forced entry into robots.txt those new labels have become worthless since they will not be crawled by search engines. Oh well, I guess all these new labels have caused Googlebot to crawl millions of extra pages each day and push there search engine to the limit.
I really hope this is not the first step in limiting other content on blogs owned by Google, like stuff they don't agree with or content they might consider to be too old for indexing. For the past few years when I have check my blogspot robot.txt file I have always seen a blank page, but today there are restrictions on my blog and several other blogs I have randomly checked. New robots.txt file on blogspot.com blogs Quote:
Last edited by markhutch; Jul 13th 2007 at 10:44 pm. 
|
|
#2
Jul 13th 2007, 10:33 pm
|
||||
|
||||
|
Where did you find this. I cannot seem to find it on my Blogger blog.
__________________
Sharpe Investing
|
|
#3
Jul 13th 2007, 10:37 pm
|
|||
|
|||
|
Type in your URL followed by /robots.txt and you will find the file. They are not only blocking Googlebot, but everyone else as well. My guess is that search engines are eating up bandwidth from blogspot.com blogs since they introduced labels to the mix in mass several months ago. Even Google's own blog on blogspot.com has the same language in it's robots.txt file.
|
|
#4
Jul 14th 2007, 9:35 am
|
||||
|
||||
|
When I typed that in the result was "Not found Error 404"
__________________
Sharpe Investing
|
|
#5
Jul 14th 2007, 9:47 am
|
||||
|
||||
|
yeah it their, nicely found
|
|
#7
Jul 14th 2007, 4:57 pm
|
|||
|
|||
|
Quote:
|
|
#8
Jul 17th 2007, 12:43 am
|
|||
|
|||
|
This is now updated by blogger and it now includes
sitemap: if you have submitted
|
|
#9
Jul 17th 2007, 3:29 pm
|
|||
|
|||
|
Anyone figure out a work around for this? My blog is entirely organized into label categories. there's no way to ftp into blogger? Also, the site map they direct to seems to only be the most recent page of posts. does that mean it won't crawl older posts? Is it wise to set posts per page as max? I feel like this is going to be really bad for rank.
|
|
#10
Jul 17th 2007, 8:16 pm
|
|||
|
|||
|
i am wondering the same. They now filter all labels and only check updated posts.
but then it means your old post wont be indexed again.
|
|
#11
Jul 17th 2007, 9:46 pm
|
|||
|
|||
|
I don't think this will effect regular posts unless there is no other way to find your inter pages except via "labels". Most blog templates are set up with all kinds of built in links for previous posts and a smart bot like Google will be able to figure that out. Not to mention every time you post a blog entry on blogger they broadcast that update to hundreds of sites worldwide and those sites and the previous post feature on most templates should keep your pages from becoming orphans in the eyes of Google and other SE's.
|
|
#12
Jul 18th 2007, 1:28 pm
|
|||
|
|||
|
Right I get that direct links to posts get indexed, but a single post is not nearly as keyword rich as landing on a category. for instance, in addition to a handful of labels I use, every post I make contains a label that corresponds to one of three categories. at the top of my page, I have a little menu bar that offers those three categories. each category is very focused with lots of keyword rich posts and displays all corresponding posts, not just the last ten. so previously, a google search may yield that category instead of my main index. since those categories are no longer indexing, i fear that my pagerank will drop because the main index is not as focused and is limited to 10 posts per page. considering the new robots.txt, is it wise to increase how many posts are displayed per page? to 20 perhaps? By the way, if i'm just completely misunderstanding how search works, please let me know.
|
|
#13
Jul 19th 2007, 12:51 pm
|
|||
|
|||
|
I need your help.
2 days ago in my robots.txt has appeared following code on my blogspot: Disabled: / Means it is not spidered at all. Web is not against google guidline. One week ago I used new blogger functionality to switch all my feed to Feedburner and maybe sitemap is not clear for Googlebot so they put this code. I don't know.... Do you have any idea???
|
|
#14
Jul 19th 2007, 7:37 pm
|
|||
|
|||
|
you sure its disable or disallow: /
|
|
#15
Jul 19th 2007, 10:53 pm
|
|||
|
|||
|
ok where is option to edit robots.txt
ya correct info
following code find in robots.txt User-agent: * Disallow: /search Sitemap: can someone guide how to edit this robots.txt file.I mean where i find option to edit robots.txt file when i log into my blogger account.
|
|
#17
Jul 20th 2007, 12:14 am
|
||||
|
||||
|
read google.com/webmasters
sitemap using yourblog.blogspot.com/atom.xml
|
|
#18
Jul 20th 2007, 12:41 am
|
|||
|
|||
|
Quote:
|
|
#19
Jul 20th 2007, 12:49 am
|
||||
|
||||
|
you can edit the robot.txt in google.com/webmasters (CMIIW)
|
|
#20
Jul 20th 2007, 2:28 am
|
|||
|
|||
|
you can't edit the file there. I tried doing the same. Saved it.
checked again, it was back to what it was.
|
 |
| Bookmarks |
| Thread Tools | |
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Help with robots.txt file | compindustries | robots.txt | 13 | Nov 1st 2007 1:31 pm |
| Robots.txt file | Shazz | AdSense | 7 | Apr 27th 2007 11:31 am |
| robots don't obey the whole robots.txt file | serban | Search Engine Optimization | 0 | Mar 26th 2007 4:10 am |
| What about robots.txt file? | 3POWER | Search Engine Optimization | 7 | Mar 25th 2007 5:33 am |
| robots.txt file | abhinavspoint | Search Engine Optimization | 6 | Dec 5th 2006 7:36 am |
