1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

is robot.txt very important for a forum?

Discussion in 'robots.txt' started by kong707, Dec 18, 2007.

  1. #1
    need some advise how important this file is
     
    kong707, Dec 18, 2007 IP
  2. _Ken

    _Ken Peon

    Messages:
    27
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Very important, you can keep them from spending time on stupid pages you don't want indexed.

    Here's an example of my robots.txt:

    That keeps them from spending time looking at (and indexing) stuff you don't need them to, like for example, the registration page, the new topic / thread page, and so forth.
     
    _Ken, Dec 19, 2007 IP
  3. madk

    madk Peon

    Messages:
    141
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #3
    I thought I'd share my robots.txt for a phpbb2 forum:

    
    User-agent: *
    Disallow: /phpBB2/admin/
    Disallow: /phpBB2/faq.php
    Disallow: /phpBB2/groupcp.php
    Disallow: /phpBB2/login.php
    Disallow: /phpBB2/memberlist.php
    Disallow: /phpBB2/modcp.php
    Disallow: /phpBB2/posting.php
    Disallow: /phpBB2/privmsg.php
    Disallow: /phpBB2/profile.php
    Disallow: /phpBB2/search.php
    Disallow: /phpBB2/viewonline.php
    
    Code (markup):
     
    madk, Jan 3, 2008 IP
  4. SwapsRulez

    SwapsRulez Peon

    Messages:
    32
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Yeah... you can always keep them out from the crawling the pages which are not necessary. But you can also leave it as is, as the size of those files is just a few MB's & nothing a big deal for the crawlers... ;) Peace!
     
    SwapsRulez, Jan 13, 2008 IP
  5. Bohol

    Bohol Peon

    Messages:
    2,680
    Likes Received:
    75
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Yes, it is very important so that the search bots would be properly guided what and what no to index.
     
    Bohol, Jan 16, 2008 IP
  6. Markas

    Markas Well-Known Member

    Messages:
    710
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    128
    #6
    If you have a phpbb forum and SEO mods are installed, it is very important to have robots.txt because Google does not like double content.

    If you have IPB, you have to disable Lo-Fi version with robots.txt because it's still double content :eek:

    And I am not sure about vbullettin never used it ;)
     
    Markas, Jan 16, 2008 IP
  7. Bohol

    Bohol Peon

    Messages:
    2,680
    Likes Received:
    75
    Best Answers:
    0
    Trophy Points:
    0
    #7
    There are discussions about this issue in simplemachines.org/community.
    You'd hear words right from the horses mouth.
     
    Bohol, Jan 16, 2008 IP
  8. rich_uk

    rich_uk Peon

    Messages:
    321
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Ideal so long as members' forums are installed in a subdirectory oh phpbb2 madk ;) I have my forum installed in / and by clever use of .htaccess file, I ensure that the home page is the portal and not the index.php.

    I use the excellent phpbbSEO which ensures I have 0 dupes and THE best, free SEO for phpbb2 (and now 3) forums that one can possibly have.
     
    rich_uk, Jan 23, 2008 IP
  9. Chris-777

    Chris-777 Peon

    Messages:
    18
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #9
    You should also toss a pointer to your sitemap in there, if applicable.

     
    Chris-777, Feb 2, 2008 IP