minstrel
Jun 6th 2007, 10:12 am
Unsettling (http://mezzoblue.com/archives/2007/06/05/unsettling/)
Dave Shea's Mezzoblue
Wed, Jun 6 2007
A bit more than a week ago, tired and jetlagged, I received an email that my site had a bunch of hidden links at the footer of every page. Links that looked suspiciously like spam. My immediate reaction was incredulity, and sure enough when I looked, no such thing existed. I wrote off that complaint as someone getting caught in a frame, or browsing with a spyware-infested PC, and forgot about it.
Then a few days later, I received a similar message from someone else. One was easy to dismiss, two was curious. I viewed source again, but this time, there they were. A series of a few dozen links to otherwise innocuous sites, with obviously spammy drug-related keywords. All wrapped up in a <u style=display:none>. Presumably the invalid code is how people were stumbling across them.
After some digging, here are the bits I've managed to piece together:
The links the first person saw were different from those the second person and myself saw afterward. That they changed, combined with the fact they only appeared to be visible at certain times of the day, leads me to believe that the links were inserted multiple times. It seems like someone had a script doing this automatically, because...
The links were inserted directly into the files sitting on the server. There was a clear pattern: any file named index.php or index.html had them appended to the end, after everything else in the file. The script started from the very root of my user account on my web server, and went four levels deep. Since I host multiple sites on the same server, every single site on this server was affected.
That left two questions. One, how did this happen? And two, how do I fix it? The second question was easy enough: manually recurse through every file affected and delete the links one by one. A huge time-waster, and it killed an entire afternoon last week, but it worked. That bought me some temporary relief, but given the already-observed recurrence of these links, I had to dig deeper and get to the root of the problem.
After a few emails with my host, they were pointing the finger at unspecified PHP security holes. There's probably no way to tell specifically how this happened, they say; the log files would be a nightmare to sift through.
Since this site does use PHP in a few spots, I only had a few guesses about where I should be looking to fix the problem:
First thing I did was change my password, naturally.
Then I upgraded to the latest version of Movable Type. Previously I was on 3.2, now I'm up to 3.35
And then, armed with tutorials and books I started guessing at where my PHP might be in error. This felt like stabbing in the dark, especially because the user-facing scripts I've written already had a bunch of data-checking built in. But there are a few other low impact uses of PHP along the way which didn't necessarily seem to need the extra scrutiny at the time; those I've gone back and tightened up quite a bit.
Cumulatively, that may have worked. A week later I haven't seen another recurrence. I don't feel out of the woods quite yet, but I'm crossing my fingers.
So let this be a cautionary tale for anyone using any sort of dynamic code or CMS on their site: security matters. Stay on top of updates. Change passwords regularly. All that stuff they tell you to do, it's for good reason.
Update: From the comments, it appears that numerous people are experiencing this problem. It also appears that there are two common threads amongst those who have suffered it: they host with Dreamhost, and/or they use Wordpress. It's not exclusive; some who host with Dreamhost don't use Wordpress, and some host elsewhere but do use Wordpress.
For those who host with Dreamhost: I received a confirmation email from them at 8:27pm PST on June 5th that yes indeed, something in the neighbourhood of 3,500 FTP accounts have been compromised. If you're on Dreamhost, time to change all your passwords, and check your sites for mischief. They haven't posted an official announcement anywhere so I can't link to it, however they recommend keeping an eye on their status blog for updates as they come in.
So in the end, it looks like there was nothing I could have done to prevent this one aside from host elsewhere. Just goes to show how important backup is, local and remote. Here's how to use rsync to back up your entire web server, from a few years back.
Dave Shea's Mezzoblue
Wed, Jun 6 2007
A bit more than a week ago, tired and jetlagged, I received an email that my site had a bunch of hidden links at the footer of every page. Links that looked suspiciously like spam. My immediate reaction was incredulity, and sure enough when I looked, no such thing existed. I wrote off that complaint as someone getting caught in a frame, or browsing with a spyware-infested PC, and forgot about it.
Then a few days later, I received a similar message from someone else. One was easy to dismiss, two was curious. I viewed source again, but this time, there they were. A series of a few dozen links to otherwise innocuous sites, with obviously spammy drug-related keywords. All wrapped up in a <u style=display:none>. Presumably the invalid code is how people were stumbling across them.
After some digging, here are the bits I've managed to piece together:
The links the first person saw were different from those the second person and myself saw afterward. That they changed, combined with the fact they only appeared to be visible at certain times of the day, leads me to believe that the links were inserted multiple times. It seems like someone had a script doing this automatically, because...
The links were inserted directly into the files sitting on the server. There was a clear pattern: any file named index.php or index.html had them appended to the end, after everything else in the file. The script started from the very root of my user account on my web server, and went four levels deep. Since I host multiple sites on the same server, every single site on this server was affected.
That left two questions. One, how did this happen? And two, how do I fix it? The second question was easy enough: manually recurse through every file affected and delete the links one by one. A huge time-waster, and it killed an entire afternoon last week, but it worked. That bought me some temporary relief, but given the already-observed recurrence of these links, I had to dig deeper and get to the root of the problem.
After a few emails with my host, they were pointing the finger at unspecified PHP security holes. There's probably no way to tell specifically how this happened, they say; the log files would be a nightmare to sift through.
Since this site does use PHP in a few spots, I only had a few guesses about where I should be looking to fix the problem:
First thing I did was change my password, naturally.
Then I upgraded to the latest version of Movable Type. Previously I was on 3.2, now I'm up to 3.35
And then, armed with tutorials and books I started guessing at where my PHP might be in error. This felt like stabbing in the dark, especially because the user-facing scripts I've written already had a bunch of data-checking built in. But there are a few other low impact uses of PHP along the way which didn't necessarily seem to need the extra scrutiny at the time; those I've gone back and tightened up quite a bit.
Cumulatively, that may have worked. A week later I haven't seen another recurrence. I don't feel out of the woods quite yet, but I'm crossing my fingers.
So let this be a cautionary tale for anyone using any sort of dynamic code or CMS on their site: security matters. Stay on top of updates. Change passwords regularly. All that stuff they tell you to do, it's for good reason.
Update: From the comments, it appears that numerous people are experiencing this problem. It also appears that there are two common threads amongst those who have suffered it: they host with Dreamhost, and/or they use Wordpress. It's not exclusive; some who host with Dreamhost don't use Wordpress, and some host elsewhere but do use Wordpress.
For those who host with Dreamhost: I received a confirmation email from them at 8:27pm PST on June 5th that yes indeed, something in the neighbourhood of 3,500 FTP accounts have been compromised. If you're on Dreamhost, time to change all your passwords, and check your sites for mischief. They haven't posted an official announcement anywhere so I can't link to it, however they recommend keeping an eye on their status blog for updates as they come in.
So in the end, it looks like there was nothing I could have done to prevent this one aside from host elsewhere. Just goes to show how important backup is, local and remote. Here's how to use rsync to back up your entire web server, from a few years back.