Robots.txt & SMF forum help

Discussion in 'Google' started by Synchronium, Jul 31, 2007.

0
  1. #1
    Hi,

    How would I configure my robots.txt file to prevent google from indexing my users profiles, and possibly other stuff like print view?

    Thanks.
     
    Synchronium, Jul 31, 2007 IP
  2. jetwings

    jetwings Active Member

    Messages:
    61
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    98
    #2
    To block profiles from bots: Disallow: /forum/index.php?action=profile*
    print view: Disallow: /forum/index.php?action=printpage*


    Below is a sample one. You can add/remove your choices.

    # for SMF
    Disallow: /forum/index.php?action=search*
    Disallow: /forum/index.php?action=calendar*
    Disallow: /forum/index.php?action=login*
    Disallow: /forum/index.php?action=register*
    Disallow: /forum/index.php?action=profile*
    Disallow: /forum/index.php?action=stats*
    Disallow: /forum/index.php?action=arcade*
    Disallow: /forum/index.php?action=printpage*
    Disallow: /forum/index.php?PHPSESSID=*
    Disallow: /forum/index.php?*rss*
    Disallow: /forum/index.php?*wap*
    Disallow: /forum/index.php?*wap2*
    Disallow: /forum/index.php?*imode*
    Disallow: /forum/Themes/*
    Disallow: *msg*
     
    jetwings, Jul 31, 2007 IP
    Synchronium likes this.
  3. Synchronium

    Synchronium Active Member

    Messages:
    463
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    58
    #3
    Thanks a lot, that's awesome!
     
    Synchronium, Jul 31, 2007 IP
  4. whoamifooling

    whoamifooling Peon

    Messages:
    98
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Thanks I have been wondering this aswell.
     
    whoamifooling, Jul 31, 2007 IP
  5. SEOpaw

    SEOpaw Peon

    Messages:
    437
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #5
    thanks! i needed this one also
     
    SEOpaw, Feb 6, 2008 IP
  6. virkalurk

    virkalurk Peon

    Messages:
    30
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    This was just what I am looking for. But I have one question, is there a way to keep it from indexing all the topics in a certain board without having to manually add each topic?

    ex. Board = /index.php?board=27.0
    ex. Topic = /index.php?topic=15.0
    ex. Topic = /index.php?topic=16.0

    The Topic url's don't include the board URL, so I am kinda stuck.
     
    virkalurk, Mar 7, 2009 IP
  7. wp-themes

    wp-themes Banned

    Messages:
    230
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Thanks 'virkalurk' for bringing an almost TWO YEARS OLD thread back in discussion!!!
    However, you can always use the * char to widen your rules:

    ex. Board = /index.php?board=27.0*
    ex. Topic = /index.php?topic=15.0*
    ex. Topic = /index.php?topic=16.0*

    And by the way, consider using URL Rewriting ;)
     
    wp-themes, Mar 7, 2009 IP
  8. wp-themes

    wp-themes Banned

    Messages:
    230
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #8
    am really sorry for the duplicated post, kindly remove this one ;)
     
    wp-themes, Mar 7, 2009 IP
  9. virkalurk

    virkalurk Peon

    Messages:
    30
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Thanks for your reply.

    I decided to go a completely different route that didn't require me to hide certain info from search engines that would interfere with showing other certain info.

    I didn't see that it was year old.

    Also, I do appreciate your comment immensely about URL redirection and SEF. I have been looking into that for SMF. You just made me realize how much more important it is.

    Kudos
     
    virkalurk, Mar 8, 2009 IP