Blogger Robots.txt Problem- Google is not crawling

Discussion in 'robots.txt' started by nikmahajan, Aug 5, 2008.

0
  1. #1
    For one of my blog on Blogger, robots.txt file is:
    User-agent: Mediapartners-Google
    Disallow:

    User-agent: *
    Disallow: /search

    Sitemap: http://softwarengameslinkshere.blogspot.com/feeds/posts/default?orderby=updated    ​

    It seems to be alright. All of blog URLs except those with /search are allowed. But when I checked Google Webmasters Tools, it showed 183 URLs restricted by robots.txt file and other 21 URLs as having problem with crawling. True to the point, my traffic has fallen adversely in Google.

    Now the question is how can I restore my robots.txt file in Google Webmasters Tools as normal. I know I can't edit it but there should be some way using which I can unrestrict my around 200 valid URLs from the Google spiders.

    Instead of default feeds URL, I am showing feeds with Feedurner. Is it the cause of the problem?
     
    nikmahajan, Aug 5, 2008 IP
  2. lspence

    lspence Peon

    Messages:
    61
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #2
    I'm not sure that I understand your question. Did you want Google to be able to crawl /search? If so, make these changes...

    User-agent: *
    Disallow: /search

    User-agent: Mediapartners-Google
    Allow: /search

    Sitemap: http://softwarengameslinkshere.blogs...rderby=updated

    Also you're using User-agent: Mediapartners-Google that is the bot that deals with Ads, and is not the User-Agent: Googlebot which is the Google bot. Is that what you wanted in your robots.txt?
     
    lspence, Aug 6, 2008 IP
  3. catanich

    catanich Peon

    Messages:
    1,921
    Likes Received:
    40
    Best Answers:
    0
    Trophy Points:
    0
    #3
    I would suggest you try this:


    User-agent: Mediapartners-Google # will block them
    Disallow: /

    User-agent: * # will allow everyone else​
     
    catanich, Aug 21, 2008 IP
  4. bloggerboy86

    bloggerboy86 Guest

    Messages:
    131
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #4
    I have the same problem as well.

    As you guys suggest its not possible to change or edit the robots.txt since my blog is hoted with blogger ( Domain is my my own ). So i dont have an option to add robots.txt to the root directory.

    Is there any other way ?:confused:
     
    bloggerboy86, Sep 10, 2008 IP
  5. chadgarber

    chadgarber Peon

    Messages:
    123
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Any answer to this question?
     
    chadgarber, Jul 29, 2009 IP
  6. chadgarber

    chadgarber Peon

    Messages:
    123
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #6
    "I have the same problem as well.

    As you guys suggest its not possible to change or edit the robots.txt since my blog is hoted with blogger ( Domain is my my own ). So i dont have an option to add robots.txt to the root directory.

    Is there any other way ? "

    Anyone?
     
    chadgarber, Jul 29, 2009 IP
  7. Vinil Mehta

    Vinil Mehta Member

    Messages:
    190
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    48
    #7
    yes i think i need what you are saying .I want google bot instead of anything else .suggest me how to do this
     
    Vinil Mehta, Sep 8, 2011 IP
  8. ankhanh

    ankhanh Peon

    Messages:
    19
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    That is what I also want to know! Any one has experience about this?
     
    ankhanh, Sep 17, 2011 IP
  9. g4ngst3r

    g4ngst3r Active Member

    Messages:
    7
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    86
    #9
    Can we edit Robots.txt file in blogger?
     
    g4ngst3r, Sep 25, 2011 IP