Mellowhost crash nightmare: just lost all my databases, forums & blogs!!

Next time be sure you pick a web host service which does daily/weekly/monthly backups... that way you are sure you won't loose your data...

 

read above - i do have old backups for some of the more established sites. i have over 30 domains, at least 15 of those sites have content and are active, so i don't sit everyday to backup my databases, esp when i just launched one 3 days ago and invested over $500 into that launch.

But yes in hindsight I should have. So as much as i appreciate your wise words, i already have learnt this lesson.

I'm trying to move forwards and decide how to go from here. I do have hosting still with hostgator but my mistake was moving all my domains to mellowhost. I do have to say their support response is very fast but the rest has been pretty disappointing, but if they do manage to restore some content (as I know they are reading this) then I'll certainly do the right thing and inform everyone of that on here. I've already asked for a public_html restore for my key sites, but nothing's back yet.

re hostgator addon/parked domains - i just prefer having cpanels individually on the sites. but i may look at this again. i think my hostgator account is still live, so i'll try it again.

 

This is affected for the server INFERNO, all other servers named the following are not affected with this:

1. Sniper
2. Driver
3. Beret
4. Spy
5. Semi-BoxN
6. Thief
7. Lupin

There is nothing else other than asking for apology regarding the matter. Hard drive is something like out of anyones control. The situation most likely announced and explained here:
http://www.mellowhost.com/file-system-corruption-on-inferno

We use Softlayer Evault Backup service to backup each data on daily basis. The server was using 215 GB of data as far i know from last stat and evault is having 210 GB of those data. Our Evault backup takes only the home directory backups as we never thought the hard drive can crash including the backup drives. Thats the reason we were storing the cpanel configuration and databases using rsync on a backup drive. But as the whole situation got interrupted suddenly on a journal crash. Right now, i m speechless, there is nothing really i can say on behalf of my support team as we couldn't recover that 5GB data anyway.

I really hate to find once we pay for the backup service, but not be able to provide the part when both are needed

Client should make the backups, but yet, i still agree about the mis-techniques we applied to backup the data while we should have everything in our evault.

A few minutes ago, i received a ticket response like this:

Looks true to me, but definitely, there are more than 3 times users on all of our other 7 servers and that won't really be wise enough to leave them in dust.

 

ASO does have an add-on for WHM at $5/month. So you could have separate cPanels if you add the add-on which gives you a reseller account basically.

 

We all do, we just don't all guarantee it. The insurance premiums are insane. If you really want to insure something is backed up.

1. Back it up yourself.
2. Pay for the backup services.

If the drive goes on you personal computer and you lose all your data and you did not back it up, do then go look for a hard drive company that does daily/weekly/monthly backups of your drive?

You really cannot lay all the blame on the hosting company here. You've got to take some personal responsibility with your own data, regardless of whether your host is or is not backing up data for you. If its that important to you, back it up.

 

If my memory serves me right each partition has it's own journal and since the backup drive would be on it's own partition it should be fine. Actually I am pretty positive on this one since you could format the first disk and the second disk would still be in working order. So losing the journal file for one partition would not knock out all the others or a backup drive.

So format the main drive then mount the backup drive back in and it should still be in working order.

 

According to NOC speech, Journal for both partition was corrupted. It happened while rsync was doing the move to the backup drives automatically yesterday. We were running ext3, so we were trying to convert this to ext2 and recover the journal using IPMI, but in the middle we ran fsck and it put all the /var partition and backup drive under lost+found for inode mis-match. Few of our techs were studying the whole day to recover the data from lost+found, but couldn't find any suitable way to recover all the data. Most the ways are available to recover, is for a gentle amount of data. But the amount of files under those partitions caused all the issues to screw up. Its practically hopeless now and thats why we announced this after trying the best we could do.

 

By the sounds of it someone ran a fsck on a mounted partition and well doing it on a mounted partition essentially wrecks the partition and dumps it all in lost+found. Quite a huge blunder for someone to do considering it tells you before attempting it that doing it on a mounted partition will destroy it.

Not the first time I've heard of someone doing it and probably will not be the last.

 

Nope, fsck didn't run on mounted partition. fsck sends the data to lost+found if and only if there are inodes errors. Partitions were unmounted before fsck using IPMI.

 

Are you so sure someone unmounted the partition? Because it seems very odd the entire partition would be garbage. I mean even if it was there are the following partitions in a standard softlayer cpanel server:

/
/boot
/home
/usr
/var
/tmp


I doubt all of them had journal issues unless as I said someone ran a fsck on them when mounted.

Oh and good reference for drive crashes: http://www.webhostingtalk.com/archive/index.php/t-232162-p-1.html

All the locations of cPanel related files

 

Well, sure, of course you guys won't quit the business- most of the ones who have been affected will probably leave, and if you have the 3x users on all your other 7 servers leave also = no income for Mellow Host = the end of the business!

But instead of thinking on how this will affect the MH business, may I pls suggest as a first point of call to helping the current people who are in this situation? This means helping us who have lost lots of data to restore what you can.

I already accepted the fate that you guys have lost all my databases stored on your servers... but so far, all that was restored on MH's behalf was the WHM panel. And I've had to manually add each of my domains again, and have emailed support this morning asking to restore the public_html on the key accounts urgently.

And so far, nothing...

I'm sure the 3x other customers have not had any problems yet, but I would say that if you guys want to keep the business running, to attend to those of us in need, as the rest of your passive income I'm sure will be funding the business ok, so don't worry too much about that.

Plenty of MH business was coming from here, so I think it's only wise to try to resolve this. If we remain unhappy for a long time it can't imaginably prove any good to Mellow's continuation of the business.

 

Yes, partitions were unmounted.

Thats the bad happened with us, even i m begging if i could get these partition to rejoin first time. We made only 3 partitions and created /var, /home and /usr on the same parition. Damn! Mysql databases were running on the same journal.

 

too bad . sorry for u guys. Whats the status at the moment?

 

I still luv the mellow host!
ish happens!where ever u R who ever u R! always has always will u just gotta deal with it and make sum dam lemonade with the lemons ur served!live and learn! u live a learn!! if only we'd have our own back ups !
iv had way worse happen with other host!
I have over 50 sites on this server and only backups for some as i just moved a bunch of sites from vibriantstreamsnetwork the worst host ever! if u want problems try them!
and most of my backups were incomplete!
and i just did alot of updates to the stuff that was on inferno that is lost!

just gotta say im glad I haven't resold any packages as this is a nightmare for any host or reseller!
but for the money the service is great support is great! u get what you pay for dont 4get that!

i got almost half of my sites back up already
tho it is work i wish i didnt have to do and i did lose alot of work but getting mad and gonna make any one happy and it mos def wont fix anything!

 

There is at least 5 techs continuously working in setting up 1000 accounts backup. Hope your one should come back asap.

I won't say anything actually as i m quite speechless today. I know you are angry, though, i expect the best to deliver all time as usual.

If someone has closely taken care of mellowhost's interest, support staffs, assistance most likely each of them will help us to encourage under such situation like you did. Hope to do the best for you in this season.

 

1000 accounts on one server??

 

Yeah thats what I thought lol.

- Prilep

 

Yes, its from two Servers of Dual Proc Quad core clustered with Softlayer's Load balancer. There are around 67 resellers i think in that clustered server having cpanel accounts. Most of those accounts are suspended for due payment i believe.

Sometimes i find your posts saying what you can't

 

And what do you mean by that?

- Prilep

 

Your posts get more interesting the more I read them.


Now I'm wondering how exactly you're handling that load balancing. A few issues come to mind;

The Data
You would need to be either replicating the data across or you're mounting the other system on that one. If that is the case there are a slew of issues with regards to file locking. So in order for that to reasonably work you'd think you'd be running a distributed file system. However I have my doubts on that one

Network Issues
One of the machines would most likely be accessing the mysql of the other. I assume you're doing some sort of strange routing of the mysql to make localhost work for the other machine. Although this is highly unlikely and I have not read of anyone doing this. It's far easier to access MySQL via IP.


I honestly do not believe a word you're saying at this point in time.

Unless of course you have no clue what you're talking which is a likely possibility. On your site you claim you had raid to protect against data corruption, but this is not what raid does. It protects against drive failures and in no way can it protect against corruption. If your file system becomes corrupted it will be mirrored across all drives.

Frankly I think your stories make little sense and the likely cause is the system locked up and upon reboot required a manual fsck. The person who was in the IPMI ran the fsck on a mounted partitions and butchered the OS and various important directories (var and usr most likely)