Best way to Block same content...but millions of links

Discussion in 'Search Engine Optimization' started by azn_romeo_4u, Oct 26, 2009.

  1. #1
    I got a forum right? The forum shares certain sections of it with other forums across different domains. For example

    www.example1.com/forum
    www.example2.com/forum
    www.example3.com/forum

    Each of these forums share a section like "Chit Chat, General Discussion" When someone posts in Chit Chat, it will appear in those other forums. Like if they start a new thread, if will also appear in all the sites.

    I want to use robots.txt file. Can I exclue something like

    http://example1.com/forumdisplay.php?f=12

    And it won't index any of the threads in that forum? There's like 20,000 threads per forum with like 200,000+ posts...I don't want to create a 100mb robots txt file to exclude these shared forums on every site...and I probally have to do it manually.

    Is there way to tell the search engines not to index any of the forumdisplay.phpf=12 url's and any links, pages, posts, threads? Without having to do it for every URL?
     
    azn_romeo_4u, Oct 26, 2009 IP
  2. addlinkurl

    addlinkurl Well-Known Member

    Messages:
    2,409
    Likes Received:
    129
    Best Answers:
    0
    Trophy Points:
    155
    Digital Goods:
    1
    #2
    this can work:

    this means: do not index the urls that include forumdisplay.
     
    addlinkurl, Oct 26, 2009 IP
  3. azn_romeo_4u

    azn_romeo_4u Peon

    Messages:
    590
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    0
    #3
    The only problem is there are some forums that have unique content to the domain that is also under forumdisplay. The other forums, title community, chitchat, games, blah blah, all have forumdisplay in their url. It's how vbulletin works. http://example1.com/forumdisplay.php?f=xx

    I have about 20 http://example1.com/forumdisplay.php?f=xx that are shared across the other forums.

    Is there no other methods to tell the bots to not crawl anything from those 20 different URL's? But also crawl other?
     
    azn_romeo_4u, Oct 26, 2009 IP
  4. Slincon

    Slincon Well-Known Member

    Messages:
    1,320
    Likes Received:
    44
    Best Answers:
    0
    Trophy Points:
    180
    #4
    use htaccess and have a php file serve a specific robots.txt depending on the domain being visited. If domain1.com then the php file will show a particular robots that allows indexed, if domain2.com or domain3.com then show a robots that prevents those specific files from being indexed.
     
    Slincon, Oct 26, 2009 IP
  5. azn_romeo_4u

    azn_romeo_4u Peon

    Messages:
    590
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Yea but the robot txt file is going to contain millions of links. That's not smart is it?

    Say the content is already index, can i just do nofollow on those other sites....will that work?
     
    azn_romeo_4u, Oct 26, 2009 IP
  6. Feathers

    Feathers Peon

    Messages:
    209
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    0
    #6
    To make it simple, just use the noindex meta tags for those specific pages you don't want indexed on thus said site(s). This can be done by your forums template system without any issues, however that question is for another thread.
     
    Feathers, Oct 26, 2009 IP