Search engines and your forum

Discussion in 'Forum Management' started by ThreeGuineaWatch, Nov 18, 2009.

  1. #1
    What percentage of your forum's traffic/bandwidth is consumed by search engine spiders? They seem to banging on mine all the time. Not just googlebot and Slurp but a whole range of others too. I'd even go as far as saying that close to 30-40% of traffic is SE spiders.

    Is it worth looking into was to cut this down? I notice some of the bots revisit months old threads which seems as though it is a bit of a waste when you have thousands of them.
     
    ThreeGuineaWatch, Nov 18, 2009 IP
  2. RectangleMan

    RectangleMan Notable Member

    Messages:
    2,825
    Likes Received:
    132
    Best Answers:
    0
    Trophy Points:
    210
    #2
    If they want to be there that much let them I say. Are you hurting from bandwidth charges? Is your site loading slow because of bots? If not then let them crawl.

    Google Webmaster Tools allows you to slow down the crawl if you are registered.

    You can also do sitemaps that may help direct the bots to pages you want. And consider using nofollows and/or a robots.txt to restrict them from viewing pages that are useless for them. Bots will click every link on the page and normally that's a waste. Example is the "reply" button which will take it to a register or no permission page. Easier to just restrict it from crawling or adding a nofollow to those type of links.

    Lots you can do IF you feel they are negatively impacting your site or results.
     
    RectangleMan, Nov 18, 2009 IP
  3. Stingerftw

    Stingerftw Well-Known Member

    Messages:
    860
    Likes Received:
    31
    Best Answers:
    0
    Trophy Points:
    195
    #3
    They don't use 1GB a month when they index my forum and they are pretty much always there ;)
     
    Stingerftw, Nov 19, 2009 IP
  4. -MGS-

    -MGS- Well-Known Member

    Messages:
    283
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    130
    #4
    Let them be! I also get crawled a lot by them but I also get a lot of Google traffic so it pays off. Never try to limit a search engine on your site, as it could affect you rankings
     
    -MGS-, Nov 19, 2009 IP
  5. webcosmo

    webcosmo Notable Member

    Messages:
    5,840
    Likes Received:
    153
    Best Answers:
    2
    Trophy Points:
    255
    #5
    Less then 5% I would say.
     
    webcosmo, Nov 25, 2009 IP