MPEx status update

As you probably know, MPEx went down yesterday. Let's get up to date with an IRC summary :

July 9

13:35:37 davout mpex.co down for everyoneⁱ ?
13:36:12 fluffypony http://isup.me/mpex.co
13:41:07 davout isup.me is down for me :D

17:48:54 mircea_popescu so as a mpex update : all 8 drives + the raid controller are going to be cut up. replacements are going in. mpex won't be back today, but i expect it to be back tomorrow.

20:12:14 jurov wow is this second longest mpex downtime ever?ⁱⁱ

July 10

04:18:42 mike_c once a year the mpex raid array goes on summer vacation.

05:57:49 mircea_popescu as an aside, mpex update : hardware changed and tested, os installed and failed. os reinstalled and failed. hardware retested and failed (!!!). hardware rechanged. tested, passed, tested, passed, os resintalled, tested.
05:57:57 mircea_popescu things are progressing smothly in the manner of fml.
06:00:54 decimation re: upgrades http://xkcd.com/349/

Affected was the core of MPEx, ie the trade engine. Out of the eight drives involvedⁱⁱⁱ six are currently bricked, two garbled. The server in question is currently available, and has been since the early morning, on spare gear. However...

However it doesn't really happen that a large number of new, premium quality drives just brick together of their own accord. There's no such thing as a HDD union, blessfully. There are a few main angles which I can see that'd explain the situation. One is that someone tried to do something or the other to the server - I mean, in physical proximity - and unaware of the particular arrangements in place^iv manged to brick the drives. This could be anything from a confused tech lockpicking the wrong box to some nefarious agent lockpicking the right box, inept handling of RF surveillance equipment and so on and so forth. The other is some obscure bug we're not aware of. I very much doubt this is the case, but then again the very definition of "obscure bug" is that I would. Or whatever, maybe Kali squirted her tits the wrong way, who's to know.

At any rate this circumstance has soured the relationship with the datacenter in question, and it looks like we'll be seeing each other in court so as to get to the bottom of the matter, or at least as reasonably close to a bottom as at all possible. I remain generally unconvinced by the "it just so happened" theory. If my admittedly paranoid^v suspicion is correct Bitcoin is yet again ahead of schedule, as is by now customary^vi - I was expecting something like this (and even said in chan I'm expecting something like this) eventually, but not quite July 2014.

In any case, I am escalating MPEx infrastructure to a significantly larger, better, hardier and consequently significantly more expensive arrangement. This will take a little time to deploy and check, which is how it works when things take the initiative out of your hand. I expect a fully functional, significantly more powerful and reliable MPEx back online no later than Monday, July 14th.

Owing to specific countermeasures implemented after the unpleasant experience last year it is not conceivable that any data has been recovered or will be recovered from the drives in question. That notwithstanding, we do have full backups (and obviously, so do you). No Bitcoin is or was at risk throughout this situation, nor has any key material been compromised. Nevertheless, I will probably generate new keys for MPEx on the new system just for the sake of clarity (and after all, there's nothing wrong with changing keys once a year anyway). If you require emergency BTC withdrawals, please get in touch here or on irc, I can and will process your request.

Other than that... whatever doesn't kill us makes us stronger, as they say. That'd be all.

1. It had just happened, minutes prior. This is not exceptional either, IRC is on the ball. [↩]
2. It is, yes. Or rather, was. [↩]
3. 4+4 in a Raid 1+0 array [↩]
4. Such as custom firmware. [↩]
5. Just because they're after you doesn't mean you're paranoid, or how did the ditty go ? [↩]
6. For what it's worth, the first conference was supposed to happen just before, not just after the climb to 100. [↩]