Feeds restricted by Robots.txt

Discussion in 'Google Sitemaps' started by citrine, Aug 21, 2010.

  1. #1
    I just saw 190 links restricted by robots.txt out of which 180 are my feed links. How do I fix it? I use wordpress

    My KB robots page shows this-

    Disallow: */feed
     
    Last edited: Aug 21, 2010
    citrine, Aug 21, 2010 IP
  2. MirVB

    MirVB Peon

    Messages:
    70
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Just remove that line? You probably have an SEO plugin that is causing this. There should be a setting to turn this option on/off.
     
    MirVB, Aug 21, 2010 IP
  3. citrine

    citrine Greenhorn

    Messages:
    18
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #3
    Oh Okie, but can feeds and comments be indexed by robots.txt? Hope google doesn't ban me for that
     
    citrine, Aug 21, 2010 IP
  4. MirVB

    MirVB Peon

    Messages:
    70
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    I'm not sure what you're trying to say. Yes, feeds and comments can be indexed by Google, as long as they are allowed by your robots.txt.

    Google won't ban you for having comments or feeds.
     
    MirVB, Aug 21, 2010 IP
  5. citrine

    citrine Greenhorn

    Messages:
    18
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #5
    Ohh! Thanks for clarifying, I will remove that line from the plugin:)
     
    citrine, Aug 21, 2010 IP
  6. citrine

    citrine Greenhorn

    Messages:
    18
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #6
    This is what I can see inside the plugin-

    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /trackback
    Disallow: /feed
    Disallow: /comments
    Disallow: /category/*/*
    Disallow: */trackback
    Disallow: */feed
    Disallow: */comments
    Disallow: /*?*
    Disallow: /*?
    Allow: /wp-content/uploads

    # Google Image
    User-agent: Googlebot-Image
    Disallow:
    Allow: /*

    # Google AdSense
    User-agent: Mediapartners-Google*
    Disallow:
    Allow: /*

    # Internet Archiver Wayback Machine
    User-agent: ia_archiver
    Disallow: /

    # digg mirror
    User-agent: duggmirror
    Disallow: /


    So do I remove these 2 lines?

    Disallow: /feed
    Disallow: /comments
     
    citrine, Aug 21, 2010 IP
  7. MirVB

    MirVB Peon

    Messages:
    70
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Did you add these settings manually? As I said earlier, you probably have a plugin (not KB Robots.txt) that is causing this. When you find the plugin, there should be settings to turn these options on/off.
     
    MirVB, Aug 21, 2010 IP
  8. citrine

    citrine Greenhorn

    Messages:
    18
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #8
    I am using a plugin called Robots.txt Editor, I never changed anything. I don't see any setting to turn it off :(
     
    citrine, Aug 21, 2010 IP
  9. MirVB

    MirVB Peon

    Messages:
    70
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Log into WordPress, and click Plugins. Make a list of all of the plugins that you have, and paste them here.

    Did you add the contents of robots.txt yourself? (did you copy it from somewhere else, and paste it into KB Robots.txt)
     
    MirVB, Aug 21, 2010 IP
  10. citrine

    citrine Greenhorn

    Messages:
    18
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #10
    In the plugin list it says-

    KB Robots.txt

    Manage your robots.txt file from within Wordpress. Especially useful for WP-MU installations.

    Deactivate | Edit
    Version 1.0.1 | By Adam R. Brown | Visit plugin site

    But when I click on it it shows-

    Robots.txt Editor
     
    citrine, Aug 21, 2010 IP