I don;t know if it's a good place to post this one We have been victim of a hacker. The firewall blocked 20 tries but not the 21 it seems that the guy was trying to get in using a new/unknown security holes in Apache. We will know better when we will analyze log files. According to my engineer this is not a hacker but a cracker, looking to install worms so they can hack or dos other servers and collect CC information, send spam. When they can’t install their worms, they destroy and trash everything not enough to stop the server. And leave blocking access from outside. So the box was down around 9pm, just before the daily backup that may be contaminated. To restore everything will cost 2-3 days between re-imaging with Fedora, re-installing applications from RPMs, reloading and testing backup. Our websites are crawled by Google, Yahoo, MSN and others every single day. Do you know what will be the collateral damage having the sites down for 3 days? Thank you
You should be fine as long as you get back up in 3 days, then when google checks you are there again. We had a couple of down days that didn't effect any ranking.
What kind of disaster recovery time is that? 2hrs for a full restore is our deal. Can't you re-route the domain to show at least a 'temporarily unavailable' response code?
A normal restore costs less than an hour when not done through FTP. When the Kernel is dead and the backup ran during the hack... it takes days! you have to rebuild the box from scratch as new. When it's a single CPU and single HD it takes time when you have a dual CPU and 2 HD you multiply tasks by 10. Looking at the logs, the guy/gale is good... as soon as he/she (let say he) got in, he stopped all reporting applications then tried to load crap of his own under root from his IP. Unfortunatly the box accepts only root load from 2 IPs... and root run programs from inside the box, not outside he couldn't stop the firewall as it accepts modification from 1 IP only. It saved us from attacking other servers. He then killed part of the kernel, killed MySQL, damaged conf files and let backup run, saving damaged material. Soemtime life is just tough.
Merlin, Can you tell us what the security hole this hacker used was and why he might be interested in your site? Also, do you have a web host, or do you serve from your home, office or a server bank? Just wondering, as this is a big concern for everyone in the industry.
Damn that sucks for sure. I'm glad you were not hit any worse than that and I hope you get it back online soon.
I had a box hacked once due to an unknown (at the time) security hole in Apache. It was annoying as hell. The last command the person ran was: rm -R /etc Code (markup): Which made recovery (without formatting) even more annoying. hehe
It seems that the security issues is in Fedora Core 3 and spamassassin-3 a release was posted on June 16th hours before we update. We have a root server from a server bank. These people are not interested by websites they are interested by servers with dedicated IPs they'll use as proxy for DOS attacks, spamming... I read that now it's a commodity you can purchase to make huge spam campaign. I'm sure you have seen ads about mailing to 15,000,000 recipients. In our case we were lucky they couldn't use the box, that's why they destroyed it. I can't imagine the FBI calling asking me why did I sent 5,000,000 spam emails... They let horses and worms sleep for 6 month... so good luck to defend yourself.