MPEx status update

Thursday, 10 July, Year 6 d.Tr. | Author: Mircea Popescu

As you probably know, MPEx went down yesterday. Let's get up to date with an IRC summary :

July 9

13:35:37 davout mpex.co down for everyonei ?
13:36:12 fluffypony http://isup.me/mpex.co
13:41:07 davout isup.me is down for me :D

17:48:54 mircea_popescu so as a mpex update : all 8 drives + the raid controller are going to be cut up. replacements are going in. mpex won't be back today, but i expect it to be back tomorrow.

20:12:14 jurov wow is this second longest mpex downtime ever?ii

July 10

04:18:42 mike_c once a year the mpex raid array goes on summer vacation.

05:57:49 mircea_popescu as an aside, mpex update : hardware changed and tested, os installed and failed. os reinstalled and failed. hardware retested and failed (!!!). hardware rechanged. tested, passed, tested, passed, os resintalled, tested.
05:57:57 mircea_popescu things are progressing smothly in the manner of fml.
06:00:54 decimation re: upgrades http://xkcd.com/349/

Affected was the core of MPEx, ie the trade engine. Out of the eight drives involvediii six are currently bricked, two garbled. The server in question is currently available, and has been since the early morning, on spare gear. However...

However it doesn't really happen that a large number of new, premium quality drives just brick together of their own accord. There's no such thing as a HDD union, blessfully. There are a few main angles which I can see that'd explain the situation. One is that someone tried to do something or the other to the server - I mean, in physical proximity - and unaware of the particular arrangements in placeiv manged to brick the drives. This could be anything from a confused tech lockpicking the wrong box to some nefarious agent lockpicking the right box, inept handling of RF surveillance equipment and so on and so forth. The other is some obscure bug we're not aware of. I very much doubt this is the case, but then again the very definition of "obscure bug" is that I would. Or whatever, maybe Kali squirted her tits the wrong way, who's to know.

At any rate this circumstance has soured the relationship with the datacenter in question, and it looks like we'll be seeing each other in court so as to get to the bottom of the matter, or at least as reasonably close to a bottom as at all possible. I remain generally unconvinced by the "it just so happened" theory. If my admittedly paranoidv suspicion is correct Bitcoin is yet again ahead of schedule, as is by now customaryvi - I was expecting something like this (and even said in chan I'm expecting something like this) eventually, but not quite July 2014.

In any case, I am escalating MPEx infrastructure to a significantly larger, better, hardier and consequently significantly more expensive arrangement. This will take a little time to deploy and check, which is how it works when things take the initiative out of your hand. I expect a fully functional, significantly more powerful and reliable MPEx back online no later than Monday, July 14th.

Owing to specific countermeasures implemented after the unpleasant experience last year it is not conceivable that any data has been recovered or will be recovered from the drives in question. That notwithstanding, we do have full backups (and obviously, so do you). No Bitcoin is or was at risk throughout this situation, nor has any key material been compromised. Nevertheless, I will probably generate new keys for MPEx on the new system just for the sake of clarity (and after all, there's nothing wrong with changing keys once a year anyway). If you require emergency BTC withdrawals, please get in touch here or on irc, I can and will process your request.

Other than that... whatever doesn't kill us makes us stronger, as they say. That'd be all.

———
  1. It had just happened, minutes prior. This is not exceptional either, IRC is on the ball. []
  2. It is, yes. Or rather, was. []
  3. 4+4 in a Raid 1+0 array []
  4. Such as custom firmware. []
  5. Just because they're after you doesn't mean you're paranoid, or how did the ditty go ? []
  6. For what it's worth, the first conference was supposed to happen just before, not just after the climb to 100. []
Category: MPEx
Comments feed : RSS 2.0. Leave your own comment below, or send a trackback.

13 Responses

  1. "and after all, there’s nothing wrong with changing keys once a year anyway" except for customers?

  2. Bitcoin has a schedule nao?

  3. Mircea Popescu`s avatar
    3
    Mircea Popescu 
    Thursday, 10 July 2014

    @pankkake Customers can change their keys as often as they wish. The more responsible actually do.

    @bitcoinpete Everything has a schedule. Even the Cardano.

    How well that schedule is kept however...

  4. Without buying a new account?

  5. Mircea Popescu`s avatar
    5
    Mircea Popescu 
    Friday, 11 July 2014

    About as often as people get new beers without buying a new beer.

  6. when will i can use MPEx again?

  7. Mircea Popescu`s avatar
    7
    Mircea Popescu 
    Sunday, 13 July 2014

    Hopefuly as the article says,

    I expect a fully functional, significantly more powerful and reliable MPEx back online no later than Monday, July 14th.

  8. still cant open MPEx,and I cant send you email.

  9. Mircea Popescu`s avatar
    9
    Mircea Popescu 
    Wednesday, 16 July 2014

    Email works as far as I can tell (I've been emailing with people back and forth). MPEx is still down, yes, still being worked on.

  10. okay,thank you,hope everything is going to be fine.

  11. any good news of MPEx?

  12. Mircea Popescu`s avatar
    12
    Mircea Popescu 
    Monday, 21 July 2014

    Yes, it's back up. http://btcbase.org/log/?date=21-07-2014#763155

  1. [...] major news item for the period would of course be the downtime around June 10th. There’s little more to add, other than continued comittment that MPEx will [...]

Add your cents! »
    If this is your first comment, it will wait to be approved. This usually takes a few hours. Subsequent comments are not delayed.