1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Catastrophic Cluster Failure

Discussion in 'Support & Feedback' started by digitalpoint, Jun 3, 2013.

  1. #1
    So as you may (or may not have) noticed, we had to roll back to a backup from 2 weeks ago (from May 16).

    This means anything done after that (public messages, private messages, even new user accounts signed up) are all gone.

    I'm sure people are going to piss and moan about it, and trust me... no one is more pissed off about this than me, so nothing you can say is going to make it any better. I've had a hell of a day today, and I'm probably in the mood to ban people who want to mouth off about it. Even being able to roll back to "only" 2 weeks ago was some luck... it was almost far worse.

    So here's the deal... threads, posts, likes, user accounts, private conversations, between May 16 and now are effectively gone.

    Balances for advertisers were *not* wiped out, so advertisers balances/credits are accurate, but reporting for the last 2 weeks was wiped.

    For users that had PayPal transactions that triggered something (premium membership or purchase of something in the digital goods marketplace), send me a private message with the transaction ID, so I can insert/restore those manually.

    Also, if you were a brand new account that had to sign up again, and you had a recurring premium membership, you should cancel the subscription in your PayPal and redo it (if you still want it) because the PayPal recurring subscription would be bound to a now non-existant account.
     
    Last edited: Jun 3, 2013
    digitalpoint, Jun 3, 2013 IP
  2. browntwn

    browntwn Illustrious Member

    Messages:
    8,347
    Likes Received:
    848
    Best Answers:
    7
    Trophy Points:
    435
    #2
    That sucks, but shit happens. Hope its not too much hell for those affected, including you.
     
    browntwn, Jun 3, 2013 IP
    Arick unirow and Devtard like this.
  3. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,333
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #3
    It was very nearly very worse... I thought the 2 week backup was gone for about an hour too, and thought we might have to roll back to a long-term backup from January. I'm glad things are back online, but yeah... worst day of my life ever today. Hopefully in a year I don't remember this.
     
    digitalpoint, Jun 3, 2013 IP
  4. competent123

    competent123 Notable Member

    Messages:
    1,743
    Likes Received:
    69
    Best Answers:
    6
    Trophy Points:
    255
    #4
    Story of a forum owner, always at the mercy of mysql gods.... ;)

    Glad it came back though.

    even 1 warning got wiped out of my account, with a loss of about 200$

    so i am not very happy, but not too sad either i guess..

    Sh*t happens. move on...

    good thing you had a backup though.
     
    competent123, Jun 3, 2013 IP
  5. malky66

    malky66 Acclaimed Member

    Messages:
    3,996
    Likes Received:
    2,248
    Best Answers:
    88
    Trophy Points:
    515
    #5
    What a bummer!!..it just go's to show how fragile the internet can be especially if something like this can happen to DP, can't imagine how you must feel.:(
     
    malky66, Jun 3, 2013 IP
  6. Vvardenfell

    Vvardenfell Member

    Messages:
    17
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    38
    #6
    This almost gave me a heart attack..
    I'm glad nothing worse happened. Hopefully it'll be forgotten in a few days.
     
    Vvardenfell, Jun 3, 2013 IP
  7. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,333
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #7
    Real life friends were checking on me all day to make sure I didn't throw myself off a cliff. lol (not funny really, but yeah...)

    If there's any TINY REMOTE good thing that came of this is I spent the extra time to finally get the giant table into the cluster that I didn't do yet (the post table), so the forum is fully in the cluster now.
     
    digitalpoint, Jun 3, 2013 IP
    Karen May Jones and browntwn like this.
  8. Zoti Media Group

    Zoti Media Group Notable Member

    Messages:
    1,598
    Likes Received:
    113
    Best Answers:
    2
    Trophy Points:
    265
    Digital Goods:
    2
    #8
    I have lost some msg's but thats not a problem. Almost everyday forum is down. I hope this is fixed once forever. Keep it up.
     
    Zoti Media Group, Jun 3, 2013 IP
  9. kamimura

    kamimura Greenhorn

    Messages:
    45
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    16
    #9
    sorry to hear this shawn... thanks for the genius support... always
     
    kamimura, Jun 3, 2013 IP
  10. finweek

    finweek Well-Known Member

    Messages:
    1,733
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    165
    #10
    Sorry to hear that. Wish you more luck in the next days
    How about the VIP membership? Do you have record of the payments and renewals?
     
    finweek, Jun 3, 2013 IP
  11. vistha82

    vistha82 Active Member

    Messages:
    179
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    63
    #11
    I have lost PMs,threads and also i did premium membership on 23rd May which is also lost.I hope you recover it and DP site is very slow for me..It takes few minutes to load it.Please look into it.
     
    vistha82, Jun 3, 2013 IP
  12. mrzyzzthetic

    mrzyzzthetic Peon

    Messages:
    11
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    1
    #12
    Dont worry about it. I only lost about 50 posts and a "like" but there was a post in the SEO section titled something along the lines of "6 reasons why you should get a better pagerank" was something to do with the new update. I think the author of that will be a little dismayed as it was over 1600 words and extremely good! Oh well. Remember the bumper sticker i invented with that slogan?

    [​IMG]
     
    mrzyzzthetic, Jun 3, 2013 IP
  13. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,333
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #13
    The issue with the forum before was that it was partially in the cluster but not fully. It's fully in the cluster now (I took the time to finish that migration while I was restoring the backup today).

    There may be some kinks here and there over the next day or two because all of the cluster database optimizations I did were lost, but yes... it's fully i the cluster now.


    PM me your transaction ID and I can get it restored.
     
    digitalpoint, Jun 3, 2013 IP
  14. karthimx

    karthimx Prominent Member

    Messages:
    4,959
    Likes Received:
    127
    Best Answers:
    2
    Trophy Points:
    340
    #14
    oh important conversation and useful posts gone :( but its good its back online.
     
    karthimx, Jun 3, 2013 IP
  15. GeeOne

    GeeOne Well-Known Member

    Messages:
    3,238
    Likes Received:
    79
    Best Answers:
    0
    Trophy Points:
    125
    #15
    Being a admin of a forum, I can understand sometime this kind of things happen.

    Feel really sorry for the members who lost their posts, and likes.

    G!
     
    GeeOne, Jun 3, 2013 IP
  16. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,333
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #16
    On a side note, running a backup as we speak... the reason the backups weren't more often before is because of our transition into the cluster... Once it's fully in the cluster (which it is now) we can do hot backups without any downtime.
     
    digitalpoint, Jun 3, 2013 IP
  17. imthebanana

    imthebanana Well-Known Member

    Messages:
    484
    Likes Received:
    40
    Best Answers:
    0
    Trophy Points:
    140
    #17
    No big deal mate,

    Server failures happen, Just reposed my competition thread asking for old entries to reenter, I have no doubt that some people may complain though.

    cheers
     
    imthebanana, Jun 3, 2013 IP
  18. puremonopoly

    puremonopoly Active Member

    Messages:
    925
    Likes Received:
    32
    Best Answers:
    0
    Trophy Points:
    70
    #18
    Well, that's a lot more than I accomplished today. I'm still stuck in brainstorming mode at the moment. You're doing an excellent job with this website and things like this happen. All that was lost had been a few days of information. I'm so glad that the whole website wasn't lost. Aaaaahhhhhhhhhhhh... I was able to move my conversations to email. However, I'm sure others (including you) have had an epic day of chaos. Nice catch though! I was already expecting to say goodbye to DP when it was down for so long. I'm happy it's still here though because this place has so much information and knowledgeable users... ooooooooooooooooo... I'm drunk. [ / rant over ] :oops:
     
    Last edited: Jun 3, 2013
    puremonopoly, Jun 3, 2013 IP
  19. amirk

    amirk Well-Known Member

    Messages:
    1,014
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    155
    #19
    So, you had only 2 backups from DP? One for two weeks ago and the other for Jan?
     
    amirk, Jun 3, 2013 IP
  20. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,333
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #20
    Normally we have a daily backup as well, but for whatever reason those were not running on the new servers (I had thought they were).
     
    digitalpoint, Jun 3, 2013 IP