1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Catastrophic Cluster Failure

Discussion in 'Support & Feedback' started by digitalpoint, Jun 3, 2013.

0
Page 10 of 10
  1. Spoiltdiva

    Spoiltdiva Acclaimed Member

    Messages:
    7,758
    Likes Received:
    2,902
    Best Answers:
    53
    Trophy Points:
    520
    #181
    Your timing was *GREAT!* Earlier last month I took 10 days off from DP (too busy with work). I wish I would have chosen my time better and avoided losing 45 trophy points, 30 likes and all my conversations zapped.
     
    Spoiltdiva, Jun 9, 2013 IP
  2. VukasinI

    VukasinI Well-Known Member

    Messages:
    617
    Likes Received:
    117
    Best Answers:
    2
    Trophy Points:
    115
    #182
    Not the big deal for me.. Funny thing is that I didn't even noticed that some likes and posts are missing till one member PM me and said: ''You lost a lot mate, just like me, let's hope this will not happen again''.. I mean, I was totally confused, I didn't know what's happening..
     
    VukasinI, Jun 12, 2013 IP
  3. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #183
    Another upside of being on the cluster... Right now (without needing to take anything down as far as end users are concerned), we are upgrading the database servers to MySQL 5.6.x (from 5.5.x).

    [ndbd(NDB)]    8 node(s)
id=11    @192.168.10.20  (mysql-5.6.11 ndb-7.3.2, starting, Nodegroup: 0)
id=12    @192.168.10.21  (mysql-5.5.30 ndb-7.2.12, Nodegroup: 1, Master)
id=13    @192.168.10.22  (mysql-5.5.30 ndb-7.2.12, Nodegroup: 2)
id=14    @192.168.10.23  (mysql-5.5.30 ndb-7.2.12, Nodegroup: 3)
id=15    @192.168.10.24  (mysql-5.5.30 ndb-7.2.12, Nodegroup: 0)
id=16    @192.168.10.25  (mysql-5.5.30 ndb-7.2.12, Nodegroup: 1)
id=17    @192.168.10.26  (mysql-5.5.30 ndb-7.2.12, Nodegroup: 2)
id=18    @192.168.10.27  (mysql-5.5.30 ndb-7.2.12, Nodegroup: 3)
    Code (markup):
    On "traditional" MySQL, the site would be down for maintenance right now.
     
    digitalpoint, Jun 19, 2013 IP
  4. browntwn

    browntwn Illustrious Member

    Messages:
    8,347
    Likes Received:
    848
    Best Answers:
    7
    Trophy Points:
    435
    #184

    It is always dangerous to poke the "down for maintenance" gods.
     
    browntwn, Jun 19, 2013 IP
    sarahk likes this.
  5. Vision4DVD

    Vision4DVD Well-Known Member

    Messages:
    531
    Likes Received:
    37
    Best Answers:
    0
    Trophy Points:
    135
    #185

    Another brief downtime today. I kept refreshing patiently was this a planned update? Everything seems intact, no data lost. Relieved:)
     
    Vision4DVD, Jun 19, 2013 IP
  6. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #186
    Couple minutes were planned... had to rearrange a few services (namely memcached).
     
    digitalpoint, Jun 19, 2013 IP
  7. holyhermit

    holyhermit Notable Member

    Messages:
    331
    Likes Received:
    22
    Best Answers:
    0
    Trophy Points:
    235
    #187
    Are the privileges for established members not working? or is it only me who is having problem? I don't see the Like button, I can't start a new thread...
     
    holyhermit, Jun 21, 2013 IP
  8. Spoiltdiva

    Spoiltdiva Acclaimed Member

    Messages:
    7,758
    Likes Received:
    2,902
    Best Answers:
    53
    Trophy Points:
    520
    #188
    I thought hermits were anti social. Why would you want to interact with fellow members by creating a thread?:rolleyes:
     
    Spoiltdiva, Jun 21, 2013 IP
  9. holyhermit

    holyhermit Notable Member

    Messages:
    331
    Likes Received:
    22
    Best Answers:
    0
    Trophy Points:
    235
    #189
    Oh Diva! where had you been? I would not have become a hermit had I found you earlier :D
     
    holyhermit, Jun 21, 2013 IP
  10. liamwli

    liamwli Well-Known Member

    Messages:
    174
    Likes Received:
    16
    Best Answers:
    1
    Trophy Points:
    190
    #190
    I walk in and read the first post, then I immediately go and setup automatic remote backups for my forum :p

    Good job I missed this I see, I can understand how bad it would be for all of you! That day must have been horrid :p

    But, no harm done really ;)
     
    liamwli, Jul 12, 2013 IP
  11. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #191
    Related side note - All tables for everything we do are now 100% in the cluster. The forum, tools, WordPress blogs and marketplace have been fully in the cluster for awhile. Digital Point Ads was mostly in the cluster, but 4 tables were not because it had some fairly complex queries against those tables that the cluster didn't like. Those queries have been simplified (well really it was denormalizing some things internally to make the query easier).

    Long story short is those last 4 straggler tables are now in the cluster as well.

    Oh, and backups are still working (haha)... retains all backups for the last 7 days, as well as every backup on the 1st of each month indefinitely.

    twin1:/var/mysql-cluster/BACKUP # ls -al
total 40
drwxr-x--- 10 root root 4096 Jul 15 00:15 .
drwxr-xr-x  4 root root 4096 Jun 26 17:28 ..
drwxr-x---  2 root root 4096 Jul  1 00:03 BACKUP-33
drwxr-x---  2 root root 4096 Jul  9 00:03 BACKUP-41
drwxr-x---  2 root root 4096 Jul 10 00:03 BACKUP-42
drwxr-x---  2 root root 4096 Jul 11 00:03 BACKUP-43
drwxr-x---  2 root root 4096 Jul 12 00:04 BACKUP-44
drwxr-x---  2 root root 4096 Jul 13 00:03 BACKUP-45
drwxr-x---  2 root root 4096 Jul 14 00:02 BACKUP-46
drwxr-x---  2 root root 4096 Jul 15 00:02 BACKUP-47
    Code (markup):
     
    digitalpoint, Jul 15, 2013 IP
    liamwli and Arick unirow like this.
  12. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #192
    Have I mentioned how cool it is that we can now upgrade the database server to new versions of MySQL without any downtime? :)

    In the middle of upgrading all DB servers to MySQL 5.6.14 (MySQL Cluster 7.3).

    [ndbd(NDB)]    8 node(s)
id=11    @192.168.10.20  (mysql-5.6.14 ndb-7.3.3, starting, Nodegroup: 0)
id=12    @192.168.10.21  (mysql-5.5.31 ndb-7.2.13, Nodegroup: 1)
id=13    @192.168.10.22  (mysql-5.5.31 ndb-7.2.13, Nodegroup: 2, *)
id=14    @192.168.10.23  (mysql-5.5.31 ndb-7.2.13, Nodegroup: 3)
id=15    @192.168.10.24  (mysql-5.5.31 ndb-7.2.13, Nodegroup: 0)
id=16    @192.168.10.25  (mysql-5.5.31 ndb-7.2.13, Nodegroup: 1)
id=17    @192.168.10.26  (mysql-5.5.31 ndb-7.2.13, Nodegroup: 2)
id=18    @192.168.10.27  (mysql-5.5.31 ndb-7.2.13, Nodegroup: 3)
    Code (markup):
     
    digitalpoint, Nov 18, 2013 IP
    ryan_uk and FreeFun4Every1 like this.
Page 10 of 10