1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Robots.txt is blocking my labels

Discussion in 'robots.txt' started by Mrinal_Roy, Feb 5, 2015.

  1. #1
    In my adsense account in "revenue optimization" i have crawl errors then when i click "fix crawl errors" then this..

    Blocked Urls Error
    http://www.rechargeoverload.in/search/label Robot Denied


    My robots.txt:

    User-agent: Mediapartners-Google
    Disallow:
    SEMrush
    User-agent: *
    Disallow: /search
    Allow: /

    Sitemap: http://www.rechargeoverload.in/atom.xml?redirect=false&start-index=1&max-results=500
    Sitemap: http://www.rechargeoverload.in/atom.xml?redirect=false&start-index=501&max-results=500
     
    Mrinal_Roy, Feb 5, 2015 IP
    SEMrush
  2. vgopalsujith

    vgopalsujith Active Member

    Messages:
    57
    Likes Received:
    3
    Best Answers:
    1
    Trophy Points:
    63
    #2
    Go to Blogger Settings > Search preferences > Custom robots.txt > edit > Remove "Disallow: /search" > Save.
    This should help the crawlers crawl complete site including labels. Hope this helps...
     
    vgopalsujith, Feb 5, 2015 IP