50 GB of bandwidth used by 400 MB site!!! Please help me detecting where problem lies

Here are the current stats of my site right now:

Size: 500 MB
Bandwidth used this month: 50 GB
Daily unique visitors: 100-200 visitors


I have been allowed just 100 GB of bandwidth per month and since there are other sites hosted too, I'm getting worried where all this bandwidth is used. There are no flash, video, audio or software at the site. The site is in Joomla with articles, forum, picture gallery etc.

One thing I want to mention is that the site was covered recently by media and the average visitors per day have shot from 20-30 to 100-200. I never had this problem before.

Can anyone recommend me how I detect this issue? I ma using Google Analytics and I have access to raw Apache log too. There are also 20 email users (each has 5 MB space) but I don't see any option in Plesk to know how much bandwidth each of them is using. I have a feeling there is some problem. Either some email user is misusing or hackers are doing something to the site.

 

Here are ths statistics from December 2007 to April 2008 :

Pageviews:



Bandwidth:



The site has not been changed since December 2007 and the site size is the same. Notice the month of April in these statistics. I'm so worried because only 1/3rd of April has passed and bandwidth consumed is over 49 GB. Please help me!!!

 

My Site is 2.3 gb approx... it needs 400gb/ month... dont be surprised by the bandwidth usage of your site... its pretty normal.

However, last month it did have a jump... maybe now you have lots of downloads/images and stuff? Or, people are using up your downloads by linking to your site without you knowing it....

To prevent it: Simply log in Cpanel, and select the menu called Hotlink Protection and assign yourdomain.com and www.yourdomain.com to be the allowed hotlinkers... and hopefully it will jump down again.

Cheers!

 

You can also reduce your bandwidth usage by simply blocking some spammy bots using robots.txt

 

Thanks a lot for pointing out. I didn't think about that before. From the traffic I can see that robots are consuming most of the bandwidth. For instance, today 5490 pageviews are by the robots and only 482 pageviews by the normal visitors. Moreover I installed a new tracking code and there is a strange thing. Apart from Google, Yahoo, MSN and Alexa robots, the top two entries are used by other robots:

1. Twiceler (+http://www.cuill.com/twiceler/robot.html)

2. psbot/0.1 (+http://www.picsearch.com/bot.html)

This Twiceler has about 70% of the robots traffic. Anyone has idea about it? It seems to be an upcoming search engine but why is it so aggressively indexing my site, as compared to big players like Yahoo and Google?

 

This list is very outdated.

The mjaority of the harvesters listed there are not robots compliant any way.
These bots are best denied access via htaccess, which controls rather than requests.

 

I've had twiceler "denied" since it first appeared.

psbot has been denied for more than nine years.

 

You can find some related info in this thread:
http://forums.digitalpoint.com/showthread.php?t=377143

 

i denied all the bots in robot.txt when i faced this type problem

 

All bots means...do you also block google, yahoo and msn...??

 

you given the suggestion buddy

its like

*
disallow /

why anything wrong ?

 

You've denied nothing, rather, you've suggested to compliant bots to honor your requests.

 

ok thanks

thats what i made with robots.txt

 

Twiceler seem to have been an "upcoming" search engine for a very long time now. They also request that you tell them if their bot is getting out of control as they are still working the kinks out of it. Hopefully they will have a way of letting the bot know to slow down a bit on your site.

On a related note, I got my first ever hit from baidu.com ! After the Baidu spider has been using twice the bandwidth of any other bot for months and months, I finally actually got someone who used their search engine and visited my site. (I'm still not sure whether the bandwidth was worth it...)

 

maybe people are using your images, pages or downloads for their own sites

 

You can also save your bandwidth by avoiding hot linkings of your images used by other websites. To avoid this, you can add a .htaccess file in your image folder..that will restrict others to use your images...

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com/.*$ [NC]
RewriteRule .*\.(gif|GIF|jpg|JPG|bmp|BMP)$ - [F]

 

Sorry, can issue outdated ... I encountered the same problem .. The same BOT Twiceler. You have denied it in the robots.txt? Maybe those who still confronted with this?

 

It is better to do block via htaccess.

 

Thats interesting cause i still havent seen cuill bots on my server and ive submitted to them several times..

can i ask you.. did you submit any sites manualy or did thay just come?

 

I added a site to directories (used service). How to define it now, with a catalog apparently ..