1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Robots.txt is blocking my labels

Discussion in 'robots.txt' started by Mrinal_Roy, Feb 5, 2015.

  1. #1
    In my adsense account in "revenue optimization" i have crawl errors then when i click "fix crawl errors" then this..

    Blocked Urls Error
    http://www.rechargeoverload.in/search/label Robot Denied


    My robots.txt:

    User-agent: Mediapartners-Google
    Disallow:

    User-agent: *
    Disallow: /search
    Allow: /

    Sitemap: http://www.rechargeoverload.in/atom.xml?redirect=false&start-index=1&max-results=500
    Sitemap: http://www.rechargeoverload.in/atom.xml?redirect=false&start-index=501&max-results=500
     
    Mrinal_Roy, Feb 5, 2015 IP
  2. vgopalsujith

    vgopalsujith Active Member

    Messages:
    58
    Likes Received:
    3
    Best Answers:
    1
    Trophy Points:
    88
    #2
    Go to Blogger Settings > Search preferences > Custom robots.txt > edit > Remove "Disallow: /search" > Save.
    This should help the crawlers crawl complete site including labels. Hope this helps...
     
    vgopalsujith, Feb 5, 2015 IP
  3. Gideon Duah

    Gideon Duah Greenhorn

    Messages:
    3
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #3
    After removing Disallow:/search should we also remove Disallow: at the top?
     
    Gideon Duah, Sep 30, 2019 IP
  4. MayurKawale

    MayurKawale Well-Known Member

    Messages:
    221
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    170
    #4
    MayurKawale, Jan 1, 2021 IP
  5. Sumon Rheza

    Sumon Rheza Peon

    Messages:
    12
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    1
    #5
    What is the best site for generating robots.txt files also sitemap??
     
    Sumon Rheza, Jul 15, 2021 IP
  6. Gideon Duah

    Gideon Duah Greenhorn

    Messages:
    3
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #6
    Don't know
     
    Gideon Duah, Jul 21, 2021 IP