1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

How to edit robots.txt in blogspot blog

Discussion in 'Google Sitemaps' started by rajanforums, Aug 1, 2007.

  1. #1
    In recent times many of my blogspot blogs are de-indexed from google. From google webmaster tools i learnt that

    urls restricted robots.txt[148]
    SEMrush
    According to google robots.txt had restricted access to 148 urls in my blog. Then i used their robots.txt analysis from their tools section.

    it showed the text of my robots.txt as follows

    "
    User-agent: *
    Disallow: /search
    Sitemap: http://self-development-meditation.blogspot.com/feeds/posts/default?orderby=updated
    "

    It clearly shows that term "disallow" is used. This is the problem most of my blogspot blogs in recent times. I am not sure whether you face a similar problem.

    Please help me in this regard. How can i edit that robot.txt file.

    advanced thanks
    rajanmr
     
    rajanforums, Aug 1, 2007 IP
    SEMrush
  2. BlindCat

    BlindCat Banned

    Messages:
    415
    Likes Received:
    34
    Best Answers:
    0
    Trophy Points:
    0
    #2
    You cant change the robots.txt file in Blogger unless you host your own blog in your server.

    Disallow: /search --- Is alright. Its just not going to index the search results pages which are duplicate by nature.

    So the current robots.txt @ blogger is just fine...but ofcourse its not perfect. You cant modify it the way you like.
     
    BlindCat, Aug 3, 2007 IP
  3. trichnosis

    trichnosis Prominent Member

    Messages:
    13,785
    Likes Received:
    333
    Best Answers:
    0
    Trophy Points:
    300
    #3
    if i were , i remove my sitemap url from my robots.txt and submit my url directly from my webmaster tools panel
     
    trichnosis, Aug 6, 2007 IP
  4. webgk.com

    webgk.com Peon

    Messages:
    96
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Why would you do that?
    And does it really matters to submit feed or url as a sitemap?
    As per BlindCat, the current Robots.txt of Rajan's blog is fine, but then what about those 148 urls showing as restricted.
     
    webgk.com, Sep 10, 2007 IP
  5. shaun_ru_th

    shaun_ru_th Active Member

    Messages:
    106
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    71
    Digital Goods:
    1
    #5
    You cannot upload the robots.txt file, but you can use the robots meta tag to control the crawling of bots on particular files.
     
    shaun_ru_th, Aug 25, 2008 IP
  6. coloriteman

    coloriteman Peon

    Messages:
    669
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #6
    you can't change the robots.txt in blogger...try editing your blog header for indexing the other pages...don't use labels..use texts and link your posts on sidebar
     
    coloriteman, Aug 26, 2008 IP
  7. laostravel

    laostravel Peon

    Messages:
    326
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Thank for your suggestion
     
    laostravel, Aug 30, 2008 IP
  8. anujbux

    anujbux Active Member

    Messages:
    235
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    73
    #8
    google is indexing my site like this

    Hi-Quality desktop wallpapers
    Archive. ▼ 2008 (47). ▼ August (47). ▼ Aug 31 (14). Reptiles 2 · Paris at Night · Charming Girls · Perfect Bedrooms · Designer Bathroom · Home Decoration ...
    colordesktop.blogspot.com/ - 113k - Cached - Similar pages


    but there are other texts to show but google is not showing that.

    what should i do?

    thanks in advance
     
    anujbux, Sep 1, 2008 IP
  9. atreyyu

    atreyyu Peon

    Messages:
    124
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Could someone explain better this issue ?
    I have "categories" in my blog, each one is actually a label, I can't figure out another way of grouping together related articles.
     
    atreyyu, Nov 15, 2008 IP
  10. Dr Googler

    Dr Googler Banned

    Messages:
    321
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #10
    is it a private domain "www.(*"(*£(*".com or )"£$").blogspot.com , i could re-code this for you if you PM me, "for free"
     
    Dr Googler, Nov 15, 2008 IP
  11. atreyyu

    atreyyu Peon

    Messages:
    124
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #11
    Blog I am talking about is in my signature :)
     
    atreyyu, Nov 15, 2008 IP
  12. 3bkreno

    3bkreno Peon

    Messages:
    8
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #12
    hi
    i dont know good but you must make it archiving
     
    3bkreno, Nov 17, 2008 IP
  13. asanka000

    asanka000 Peon

    Messages:
    782
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #13
    i have the same problem. since robots.txt blocks crawling urls, my google traffic is fluctuating highly. some time it is 150 per day and some time it is 10-20. this is really annoying!! any expert help me please!
     
    asanka000, Apr 26, 2009 IP
  14. abuviqar

    abuviqar Member

    Messages:
    1
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    36
    #14
    add this meta tag

    <meta content='noindex,follow' name='Labels'/> for block your label or
    <meta content='noindex,follow' name='Archive'/> for block archieve
     
    abuviqar, May 18, 2009 IP
  15. realfan

    realfan Peon

    Messages:
    9
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #15
    i have nearly the same problem. someone know why google wouldnt want to index my web? i have been waiting more than a month
     
    realfan, May 19, 2009 IP
  16. alamest

    alamest Well-Known Member

    Messages:
    278
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    133
    #16
    Is there not really any why to change blogger robot.txt file because there are more than hundreds of bots to follow and in blogger we have only one which is media partner.

    Is media partner means all the bots should follow the website or it's only google search or it includes all search engines.
     
    alamest, Aug 15, 2009 IP
  17. nhocikaka

    nhocikaka Peon

    Messages:
    8
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #17
    Thanks You ::)
     
    nhocikaka, Aug 21, 2009 IP
  18. alamest

    alamest Well-Known Member

    Messages:
    278
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    133
    #18
    Why did you said me thank you, I haven't asked a question, I need answer for the question.

    But you welcome..and I have searched many place to change robot.txt file but there is not way to change, only one thing to do is move to wordpress and buy your own domain and host some where than you can change robot.txt file.

    I hope this will help.
     
    alamest, Aug 21, 2009 IP
  19. avi

    avi Well-Known Member

    Messages:
    200
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    130
    #19
    hey, can you place meta tags in blogger to exclude a directory from google? like Disallow: /2010/01/ ?? how can you do it?
     
    avi, Feb 5, 2010 IP
  20. Dhvanesh

    Dhvanesh Peon

    Messages:
    356
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #20
    Hi,

    Could you please explain it in brief?
     
    Dhvanesh, Feb 18, 2010 IP