Is it a good idea to block /rss/feed in Robots.txt?

Discussion in 'Search Engine Optimization' started by janmars, Jun 6, 2007.

  1. #1
    I have noticed that I am starting to have a number of bots related to Rss Feed sites (Google Feed, Feedster, YahooFeed etc.)

    Currently I have the RSS feeds blocked in my Robots.txt to stop content duplication problems / penalties.

    I assume this would be causing the Feed bots to not be able to crawl my feeds?

    Should I allow these directories to be crawled - and if so could I face duplicate content issues?
     
    janmars, Jun 6, 2007 IP
  2. DanInManchester

    DanInManchester Active Member

    Messages:
    116
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    53
    #2
    I allow them. it's not really content duplication and I wouldn't concern yourself about it. the whole point of RSS is that provides a summary of existing content and by its very nature will therefore duplicate some but i wouldn't expect all your content.

    Anyway, if google hasn't considered this I'd be VERY surprised
     
    DanInManchester, Jun 6, 2007 IP
  3. ajsa52

    ajsa52 Well-Known Member

    Messages:
    3,426
    Likes Received:
    125
    Best Answers:
    0
    Trophy Points:
    160
    #3
    If you don't want your RSS to be crawled, don't include in your site the reference to the feed or XML.
    Example, remove this kind of lines from your .html:
    
    <link rel="alternate" type="application/rss+xml" title="Your feed title" href="./your_feed_file.xml">
    
    Code (markup):
    That way the bots won't try to download (because they don't know the existence of that feed).
     
    ajsa52, Jun 6, 2007 IP
  4. Anghus

    Anghus Peon

    Messages:
    510
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    0
    #4
    It's no problem getting them crawled, but the problem arises when your RSS feed gets higher position then your website. If you are running a blog, I would recommend to block comment RSS feed.
     
    Anghus, Jun 6, 2007 IP