wordpress robots.txt

Discussion in 'WordPress' started by burumcmtop66, Jul 23, 2011.

0
  1. #1
    i have made a robots.txt for my wordpress, let me know if i have to remove or add anything which is neccessary thanks

    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /trackback
    Disallow: /feed
    Disallow: /comments
    Disallow: /category/*/*
    Disallow: */trackback
    Disallow: */feed
    Disallow: */comments
    Disallow: /*?*
    Disallow: /*?

    # Google Image
    User-agent: Googlebot-Image
    Disallow:
    Allow: /*

    # Google AdSense
    User-agent: Mediapartners-Google*
    Disallow:
    Allow: /*

    # digg mirror
    User-agent: duggmirror
    Disallow: /
     
    burumcmtop66, Jul 23, 2011 IP
  2. Aman Arora

    Aman Arora Active Member

    Messages:
    100
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    78
    #2
    You can remove this Disallow: /category/*/* its really not needed
     
    Aman Arora, Jul 23, 2011 IP
  3. rohitink

    rohitink Active Member

    Messages:
    370
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    60
    #3
    Oh Boy. Thanks a lot. Now I am going to add this to my robots.txt..

    I never knw about it.
     
    rohitink, Jul 23, 2011 IP
  4. mohsince

    mohsince Peon

    Messages:
    33
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    i suggest , disallow- tag alow to avoid duplicated content

    Disallow: /tag/*/*
     
    mohsince, Jul 24, 2011 IP
  5. Aman Arora

    Aman Arora Active Member

    Messages:
    100
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    78
    #5
    Aman Arora, Jul 25, 2011 IP
  6. aliviamallan

    aliviamallan Active Member

    Messages:
    163
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    53
    #6
    Thanks for your good information. I want to ask you 1 question that how we can upload robot.text in wordpress.
     
    aliviamallan, Jul 25, 2011 IP
  7. Aman Arora

    Aman Arora Active Member

    Messages:
    100
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    78
    #7
    Just upload it to the root of your website by using FTP or Cpanel
     
    Aman Arora, Jul 25, 2011 IP
  8. kajol

    kajol Well-Known Member

    Messages:
    523
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    108
    #8
    Or use this plugin

    http://wordpress.org/extend/plugins/kb-robotstxt/
    Code (markup):
    You can create & edit the robots.txt from the wp-admin
     
    kajol, Jul 26, 2011 IP