Prevent Search Engines Following Your XML/RSS Feeds

Discussion in 'Google' started by twiztedX3, Jul 3, 2010.

0
  1. #1
    It is not good for search engines to index and list your XML/RSS feeds on search engines. So I have prepared the actual robots.txt code that prevents search engines from following those unnecessary pages... It doesn't look good for anyone and can hurt your ranking down the road.

    CODE:

    User-agent: *
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /trackback
    Disallow: /cgi-bin
    Disallow: /search
    Disallow: /feed
    Disallow: /rss
    Disallow: /comments/feed
    Disallow: /feed/$
    Disallow: /*/feed/$
    Disallow: /*/feed/rss/$
    Disallow: /*/trackback/$


    :)
     
    twiztedX3, Jul 3, 2010 IP
  2. AirForce1

    AirForce1 Peon

    Messages:
    1,325
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Thanks for your nice code but I don't think there would be any reasons to prevent search engines from indexing or following our RSS Feeds. :)

    Have a nice day,
     
    AirForce1, Jul 3, 2010 IP
  3. haveseo

    haveseo Well-Known Member

    Messages:
    995
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    103
    #3
    These are .htaccess codes and by using them we can prevent spiders from spidering everything from websites. But preventing them from spidering RSS feed is not necessary at all.
     
    haveseo, Jul 4, 2010 IP
  4. Abhik

    Abhik ..:: The ONE ::..

    Messages:
    11,337
    Likes Received:
    606
    Best Answers:
    0
    Trophy Points:
    410
    Digital Goods:
    2
    #4
    Why'd anyone want to stop spiders crawling RSS Feeds? I mean, what's the point doing so?
     
    Abhik, Jul 4, 2010 IP
  5. lightlysalted

    lightlysalted Active Member

    Messages:
    2,067
    Likes Received:
    32
    Best Answers:
    0
    Trophy Points:
    90
    #5
    if anything i think that I would want search engines following RSS feeds. Could you explain a little more why you think it would be a bad idea for google to follow these?
     
    lightlysalted, Jul 4, 2010 IP
  6. touchserv

    touchserv Guest

    Messages:
    65
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Thanks a lot for imp link
     
    touchserv, Jul 4, 2010 IP
  7. Grimm

    Grimm Peon

    Messages:
    3,072
    Likes Received:
    57
    Best Answers:
    0
    Trophy Points:
    0
    #7
    LOL. Why would I want to block RSS or feeds? Google won't consider it as duplicate content silly. It also helps get your new posts get indexed quicker.
     
    Grimm, Jul 4, 2010 IP
  8. WishBone

    WishBone Peon

    Messages:
    2,566
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #8
    RSS and Feeds, in fact is one good way to represent your site's content to search engines. Would you care to elaborate how those part of blogs/sites being indexed would hurt your site?
     
    WishBone, Jul 5, 2010 IP