Well that was a pretty craptastic 22 hours. lol The short version of what happened is someone at the data center where the equipment is at took the liberty of moving around some of our PDU plugs... Ultimately putting multiple of them on the same electrical circuit, so we ended up tripping the breakers at the data center because the equipment was put on a single circuit and was drawing too much current for it to handle. That in itself is no big deal to fix, but when you have to restore about 300GB of databases to an in-memory database cluster, it's very, very slow (it's designed to be very fast once it's loaded, but not fast with the actual loading of data).
Oh yeah... the search engine is in the process of being rebuilding all data, so not everything is searchable quite yet. Currently everything is searchable *except* for users with a user ID > 70000 and posts with a post ID > 2,900,000... but it's working on it, and shouldn't be too much longer until the last bits are searchable.
Yeah... it ended up being faster to just restore a backup from a few hours earlier, so I went that route. So anything within a couple hours of the "event" is gone.
ok.. not what I was expecting to read seeing the title "Super Fun..." good you have the backups! That was a long downtime. I just figured you got raided by FBI and that's the last we'll see of DP. jk good to see it's back
lol, i was feeling very bad again and again seeing my social buttons not displaying anyway welcome back DP Good work shawn, you are really a hardworking person, my site has only 2GB of database yet, i can understand how it feels when you have to do something with it
Hopefully this doesn't happen again between July - Dec then we wouldn't have your awesome skills to solve it. Just sayin.
btw, some data got lost. Like there were several pms that I sent/received right before the crash and they disappeared.