1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robots.txt for phpBB forum

Discussion in 'robots.txt' started by Googles76, Dec 3, 2005.

  1. #1
    If I'm running the latest phpBB forum, which parts should I have included in robots.txt so they don't get spidered? I don't want all that junk being in the index, just the useful posts. If anyone has an actual example of a robots.txt with that, it would be great:)
     
    Googles76, Dec 3, 2005 IP
  2. mdvaldosta

    mdvaldosta Peon

    Messages:
    4,079
    Likes Received:
    362
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Here's one of mine:

    User-agent: *
    Disallow: /forums/admin/
    Disallow: /forums/images/
    Disallow: /forums/includes/
    Disallow: /forums/language/
    Disallow: /forums/templates/
    Disallow: /forums/common.php
    Disallow: /forums/config.php
    Disallow: /forums/groupcp.php
    Disallow: /forums/memberlist.php
    Disallow: /forums/modcp.php
    Disallow: /forums/posting.php
    Disallow: /forums/profile.php
    Disallow: /forums/privmsg.php
    Disallow: /forums/viewonline.php
    Disallow: /forums/search.php
    Disallow: /forums/faq.php
     
    mdvaldosta, Dec 3, 2005 IP
  3. Googles76

    Googles76 Peon

    Messages:
    664
    Likes Received:
    26
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Thanks! thats what I was looking for:cool:
     
    Googles76, Dec 3, 2005 IP
  4. vic@VKIStudios

    vic@VKIStudios Member

    Messages:
    39
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    41
    #4
    robots.txt is great for controlling the indexing of bots, for your site, if you have duplicate content elsewhere on the web. Duplicate content is bad, bad bad!
     
    vic@VKIStudios, Dec 6, 2005 IP
  5. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #5
    Here's what mine looks like:

     
    minstrel, Dec 11, 2005 IP
  6. unitechy

    unitechy Peon

    Messages:
    350
    Likes Received:
    11
    Best Answers:
    0
    Trophy Points:
    0
    #6
    should i just put what minstrel said on robots.txt file????
     
    unitechy, Dec 14, 2005 IP
  7. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #7
    Change the "/phpbb/" part to whatever your forum folder is named, of course.
     
    minstrel, Dec 14, 2005 IP
  8. unitechy

    unitechy Peon

    Messages:
    350
    Likes Received:
    11
    Best Answers:
    0
    Trophy Points:
    0
    #8
    yup thanku very much..
     
    unitechy, Dec 14, 2005 IP
  9. Jac0bsnn

    Jac0bsnn Peon

    Messages:
    5
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Mine is more compact:
    
    User-agent: * 
    Disallow: /forum/admin/ 
    Disallow: /forum/includes/ 
    Disallow: /forum/common.php 
    Disallow: /forum/config.php 
    Disallow: /forum/groupcp.php 
    Disallow: /forum/memberlist.php 
    Disallow: /forum/modcp.php 
    Disallow: /forum/profile.php 
    
    Code (markup):
     
    Jac0bsnn, Jan 18, 2006 IP
  10. Martindale

    Martindale Well-Known Member

    Messages:
    440
    Likes Received:
    11
    Best Answers:
    0
    Trophy Points:
    108
    #10
    Sorry for the ancient bump, but what how would I go about blocking the "viewtopic.php?p=<postnumber>" links?

    Mainly to avoid duplicate content penalties.
     
    Martindale, Apr 30, 2007 IP
  11. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #11
    There is no duplicate content penalty. There is a duplicate content filter which will index one page and ignore other pages with identical content. Unless Google indexes the "wrong" page, you don't need to worry.

    How does "viewtopic.php?p=<postnumber>" create a duplicate content issue?
     
    minstrel, Apr 30, 2007 IP
  12. Martindale

    Martindale Well-Known Member

    Messages:
    440
    Likes Received:
    11
    Best Answers:
    0
    Trophy Points:
    108
    #12
    A topic is usually "viewtopic.php?t=<threadID>" - however, each individual post on that page, the first page of the thread, can be linked to using "viewtopic.php?p=<postID>#<postID>". Google of course ignores the #postID, referring only to a specific area of the page. However, with probably 20 posts per page, there's no content difference (at ALL) on the post pages versus the thread page itself.
     
    Martindale, May 2, 2007 IP
  13. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #13
    I think you're worrying way too much about the duplicate content issue here. Remember, it's a filter, not a penalty.

    You don't really want to eliminate the individual post URLs, since other sites will legitmately want to link to either a specific thread OR to a specific post.

    And you don't NEED to eliminate individual post URLs - all links will take you to the thread one way or another, which is after all the whole idea.
     
    minstrel, May 2, 2007 IP
  14. Martindale

    Martindale Well-Known Member

    Messages:
    440
    Likes Received:
    11
    Best Answers:
    0
    Trophy Points:
    108
    #14
    Wow, thanks for the info. That helps a ton. :)
     
    Martindale, May 8, 2007 IP
  15. trichnosis

    trichnosis Prominent Member

    Messages:
    13,785
    Likes Received:
    333
    Best Answers:
    0
    Trophy Points:
    300
    #15
    this looks good but i think you must add $ to end of the file names

     
    trichnosis, May 11, 2007 IP
  16. gbonnet

    gbonnet Peon

    Messages:
    84
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #16
    Do not forget this one :
    It's probably the most important rule actually since the same message can be accessible with /forum/viewtopic.php?p=<id> and /forum/viewtopic.php?t=<id>
     
    gbonnet, Mar 23, 2009 IP
  17. teen_titans

    teen_titans Peon

    Messages:
    3
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #17
    where to put the robot txt?
     
    teen_titans, Apr 4, 2009 IP
  18. yoursilk29

    yoursilk29 Peon

    Messages:
    99
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #18
    on forum seeting can i put this code?
     
    yoursilk29, May 2, 2009 IP
  19. linux-lover

    linux-lover Peon

    Messages:
    45
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #19
    you have to create the robots.txt in where your forum is located and then put the codes.
     
    linux-lover, May 14, 2009 IP