1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Blogspot.com Now Has Robots.txt File

Discussion in 'Blogging' started by markhutch, Jul 13, 2007.

  1. #1
    I found out by accident tonight that Blogspot.com has added an entry to everyone's blog excluding /search files from all search engines. When most of us were forced to upgrade to the new Blogspot a few months ago, blog labels seemed like a good way to group our content into appropriate categories. Now with this new forced entry into robots.txt those new labels have become worthless since they will not be crawled by search engines. Oh well, I guess all these new labels have caused Googlebot to crawl millions of extra pages each day and push there search engine to the limit.

    I really hope this is not the first step in limiting other content on blogs owned by Google, like stuff they don't agree with or content they might consider to be too old for indexing. For the past few years when I have check my blogspot robot.txt file I have always seen a blank page, but today there are restrictions on my blog and several other blogs I have randomly checked.

    New robots.txt file on blogspot.com blogs
    SEMrush
     
    markhutch, Jul 13, 2007 IP
    ameyjah likes this.
    SEMrush
  2. mwgr5

    mwgr5 Peon

    Messages:
    230
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Where did you find this. I cannot seem to find it on my Blogger blog.
     
    mwgr5, Jul 13, 2007 IP
  3. markhutch

    markhutch Peon

    Messages:
    357
    Likes Received:
    22
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Type in your URL followed by /robots.txt and you will find the file. They are not only blocking Googlebot, but everyone else as well. My guess is that search engines are eating up bandwidth from blogspot.com blogs since they introduced labels to the mix in mass several months ago. Even Google's own blog on blogspot.com has the same language in it's robots.txt file.
     
    markhutch, Jul 13, 2007 IP
  4. mwgr5

    mwgr5 Peon

    Messages:
    230
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #4
    When I typed that in the result was "Not found Error 404"
     
    mwgr5, Jul 14, 2007 IP
  5. ameyjah

    ameyjah Peon

    Messages:
    2,596
    Likes Received:
    93
    Best Answers:
    0
    Trophy Points:
    0
    #5
    yeah it their, nicely found
     
    ameyjah, Jul 14, 2007 IP
  6. sat123

    sat123 Banned

    Messages:
    1,600
    Likes Received:
    45
    Best Answers:
    0
    Trophy Points:
    0
    #6
    that is there
     
    sat123, Jul 14, 2007 IP
  7. wijasenna

    wijasenna Active Member

    Messages:
    375
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    58
    #7
    robot.txt is something new for me. what do you mean by They are not only blocking Googlebot, but everyone else as well.? thanks!
     
    wijasenna, Jul 14, 2007 IP
  8. godmode

    godmode Well-Known Member

    Messages:
    4,458
    Likes Received:
    156
    Best Answers:
    0
    Trophy Points:
    190
    #8
    This is now updated by blogger and it now includes


    sitemap: if you have submitted
     
    godmode, Jul 17, 2007 IP
  9. viralrootsdotcom

    viralrootsdotcom Peon

    Messages:
    2
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Anyone figure out a work around for this? My blog is entirely organized into label categories. there's no way to ftp into blogger? Also, the site map they direct to seems to only be the most recent page of posts. does that mean it won't crawl older posts? Is it wise to set posts per page as max? I feel like this is going to be really bad for rank.
     
    viralrootsdotcom, Jul 17, 2007 IP
  10. godmode

    godmode Well-Known Member

    Messages:
    4,458
    Likes Received:
    156
    Best Answers:
    0
    Trophy Points:
    190
    #10
    i am wondering the same. They now filter all labels and only check updated posts.

    but then it means your old post wont be indexed again.
     
    godmode, Jul 17, 2007 IP
  11. markhutch

    markhutch Peon

    Messages:
    357
    Likes Received:
    22
    Best Answers:
    0
    Trophy Points:
    0
    #11
    I don't think this will effect regular posts unless there is no other way to find your inter pages except via "labels". Most blog templates are set up with all kinds of built in links for previous posts and a smart bot like Google will be able to figure that out. Not to mention every time you post a blog entry on blogger they broadcast that update to hundreds of sites worldwide and those sites and the previous post feature on most templates should keep your pages from becoming orphans in the eyes of Google and other SE's.
     
    markhutch, Jul 17, 2007 IP
  12. viralrootsdotcom

    viralrootsdotcom Peon

    Messages:
    2
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #12
    Right I get that direct links to posts get indexed, but a single post is not nearly as keyword rich as landing on a category. for instance, in addition to a handful of labels I use, every post I make contains a label that corresponds to one of three categories. at the top of my page, I have a little menu bar that offers those three categories. each category is very focused with lots of keyword rich posts and displays all corresponding posts, not just the last ten. so previously, a google search may yield that category instead of my main index. since those categories are no longer indexing, i fear that my pagerank will drop because the main index is not as focused and is limited to 10 posts per page. considering the new robots.txt, is it wise to increase how many posts are displayed per page? to 20 perhaps? By the way, if i'm just completely misunderstanding how search works, please let me know.
     
    viralrootsdotcom, Jul 18, 2007 IP
  13. vld2czech

    vld2czech Peon

    Messages:
    3
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #13
    I need your help.

    2 days ago in my robots.txt has appeared following code on my blogspot:
    Disabled: /
    Means it is not spidered at all. Web is not against google guidline. One week ago I used new blogger functionality to switch all my feed to Feedburner and maybe sitemap is not clear for Googlebot so they put this code. I don't know....

    Do you have any idea???
     
    vld2czech, Jul 19, 2007 IP
  14. godmode

    godmode Well-Known Member

    Messages:
    4,458
    Likes Received:
    156
    Best Answers:
    0
    Trophy Points:
    190
    #14
    you sure its disable or disallow: /
     
    godmode, Jul 19, 2007 IP
  15. christinabob

    christinabob Banned

    Messages:
    31
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #15
    ya correct info
    following code find in robots.txt

    User-agent: *
    Disallow: /search
    Sitemap:

    can someone guide how to edit this robots.txt file.I mean where i find option to edit robots.txt file when i log into my blogger account.
     
    christinabob, Jul 19, 2007 IP
  16. vld2czech

    vld2czech Peon

    Messages:
    3
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #16
    This is what I have in robots.txt of my blogspot blog:

    User-agent: *
    Disallow: /

    DO you have any idea how to change it and why google assign this formula?
     
    vld2czech, Jul 19, 2007 IP
  17. yudhis97

    yudhis97 Peon

    Messages:
    149
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #17
    read google.com/webmasters
    sitemap using yourblog.blogspot.com/atom.xml
     
    yudhis97, Jul 20, 2007 IP
  18. godmode

    godmode Well-Known Member

    Messages:
    4,458
    Likes Received:
    156
    Best Answers:
    0
    Trophy Points:
    190
    #18
    This will filter all "tags" for your post. Google is planning to ignore tags in SE. I dont know any way to edit the robot.txt for blogspot.
     
    godmode, Jul 20, 2007 IP
  19. yudhis97

    yudhis97 Peon

    Messages:
    149
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #19
    you can edit the robot.txt in google.com/webmasters (CMIIW)
     
    yudhis97, Jul 20, 2007 IP
  20. godmode

    godmode Well-Known Member

    Messages:
    4,458
    Likes Received:
    156
    Best Answers:
    0
    Trophy Points:
    190
    #20
    you can't edit the file there. I tried doing the same. Saved it.

    checked again, it was back to what it was.
     
    godmode, Jul 20, 2007 IP