Need your help regarding Robot.txt file as I am getting errors after changing it.

Discussion in 'Search Engine Optimization' started by mhamidch, Nov 24, 2011.

  1. #1
    Hi friends,

    Recently I have changed my sites robot.txt file for my wordpress site to optimize it so that search engines should not crawl those areas which are not required. After the optimization I have got the following error in Google Webmaster tools.

    a red mark with the message saying: " Some important page is blocked by robots.txt".

    My website is www.idesisms.com

    Can anyone guide me is that the correct way to configure my robot.txt. Any suggestions please.

    The following is my robot.txt file data:
    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /wp-login.php
    Disallow: /*wp-login.php*
    Disallow: /trackback
    Disallow: /feed
    Disallow: /comments
    Disallow: /author
    Disallow: */trackback
    Disallow: */feed
    Disallow: */comments
    Disallow: /z/j/
    Disallow: /z/c/
    Disallow: /stats/
    Disallow: /dh_
    Disallow: /tag/*
    Disallow: /login/
    Disallow: /wget/
    Disallow: /httpd/
    Disallow: /i/
    Disallow: /f/
    Disallow: /t/
    Disallow: /c/
    Disallow: /j/
    Disallow: /*.php$
    Disallow: /*?*
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /*.gz$
    Disallow: /*.wmv$
    Disallow: /*.cgi$
    Disallow: /*.xhtml$
    Disallow: /*?*
    Disallow: /*?
    Allow: /wp-content/uploads

    # alexa archiver
    User-agent: ia_archiver
    Disallow: /

    # disable duggmirror by Digg
    User-agent: duggmirror
    Disallow: /

    # allow google image bot to search all images
    User-agent: Googlebot-Image
    Disallow: /wp-includes/
    Allow: /*

    # allow adsense bot on entire site
    User-agent: Mediapartners-Google*
    Disallow:
    Allow: /*
    Sitemap: http://www.idesisms.com/sitemap.xml.gz
     
    mhamidch, Nov 24, 2011 IP
  2. mhamidch

    mhamidch Greenhorn

    Messages:
    6
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #2
    any one there
     
    mhamidch, Nov 24, 2011 IP
  3. pearl17

    pearl17 Banned

    Messages:
    727
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #3
    have never used this txt file is it important
     
    pearl17, Nov 25, 2011 IP