Bandwidth problem due to bots

Discussion in 'robots.txt' started by Molon Lave, Apr 10, 2008.

0
  1. #1
    Hello guys, I'm not sure if this is the correct section to post this problem in. Mods, please move to the appropriate section if this is the wrong place to be in.

    Ok, here it is, ever since I have changed from 2.1.7 to 2.3.3 IPB boards, my bandwidth has gone thru the roof. I used to go thru about 30 gigs a month, know I go thru 120 gig, and the google bot eats up about 85 gig, the yahoo bot another 10 gig.

    My limit is 40 gig and each gig after that is $4 a pop.

    Now the problem, I rank very high on each search engine, and dont want to lose it, but I need to control the amount of bandwidth the bots go thru. I dont want to ban them altogether either as i will lose my search engine ranking.

    I found this in regards to the yahoo bot
    http://help.yahoo.com/l/us/yahoo/search/webcrawler/slurp-03.html

    Now, I have a VPS server for my site, so my questions are as follows.

    What value should I use, and is there is a similar thing I can do for a google bot?

    Or is there another way to reduce the amount of bandwidth the bots consume?
     
    Molon Lave, Apr 10, 2008 IP
  2. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #2
    1- If you want that Google bot should not consume too much bandwidth from your website,you can adjust your website crawl rate to Slower from 'set crawl rate' section under tools tab in the Google webmaster tools. By doing this you website will be crawled regularly but the traffic generated by Google bots will be reduced hence less bandwidth usage by Google bots.

    2- There are some unwanted bots , that crawl your web pages for email address to spam you with advertisements. So it is a good practice to block these spammy robots to reduce your bandwidth usage..

    I hope this should help you...:)
     
    manish.chauhan, Apr 10, 2008 IP
  3. Molon Lave

    Molon Lave Peon

    Messages:
    2
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Thank You. Will give it a try.
     
    Molon Lave, Apr 11, 2008 IP
  4. webcont247365

    webcont247365 Peon

    Messages:
    143
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Hi Manish,

    I am new to SEO and internet marketing too. When I check statistics of my website I see some domain like web.usafootball.com visited my website and eaten up a lot of bandwidth. I tried blocking the domain but I think I failed -- they are still doing the same. I have few questions.

    a) Is there any way to stop them?
    b) My website is doing good on google and other search engines, how they affect my ranking?

    Thanks,
     
    webcont247365, May 6, 2008 IP
  5. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #5
    If that domain is eating up your website too much bandwidth, you can block them using .htaccess. For that you need to track domain's IP address first, from where he is visiting your website that you can find from your traffic log, then disallow that IP or series of IP(in case if that domain using several IP addresses) from your htaccess file.

    As you mentioned you are doing good on search engines, there is no way that such kind of visits can alter those rankings.....It can only increase your bandwidth usage hence more charges to your hosting company, It'll not affect your search engine rankings.....:)
     
    manish.chauhan, May 7, 2008 IP