MPEx security breach

Thursday, 26 September, Year 5 d.Tr. | Author: Mircea Popescu

Today in the interval 2 pm - 7 pm GMT MPEx suffered a layered attack which eventually resulted in unexecuted trades being relayed through to the market data feeds, brokers et al. We will go here into the details of what happened, why, how and what measures have been taken as a result.

First off, all funds are safe. No sum of BTC was obtained by the attacker as a result of his efforts. Second off, market data is also safe, in the sense that there exists no ambiguity as to what orders are in fact legitimate and what orders were illegitimately conveyed.

Unfortunately this also means a number of trades that appeared to be made in the respective interval will be rolled back. Third parties can trivially verify which is which, because while the legitimate orders contain tracking information, the illegitimate orders do not. Affected are a few dozen trades in the S.MPOE and S.BBET symbols.

In order to understand what happened, we will have to refer to the earlier tech stuff article. The particular architecture MPEx employs leaves it vulnerable to a situation very similar to what is known in Bitcoin as a 50% attack. That is to say, in some circumstances the proxies caching content from MPEx may find themselves in the situation of incorrectly identifying data as valid.

Such circumstances occur rarely under very heavy load. The first occurrence was on April 12, 2013. At that time one user had a trade erroneously reported as executed when in fact his counterparty had already cancelled his order, resulting in a pretty ruffled user and a sustained effort to refine and improve the proxy synchronisation mechanics so as to avoid such occurrences in the future.

MPEx has been sustaining significant floods since September the 22nd, peaking at around 3.2 Mbi. Nevertheless, outside of transient page timeouts trade was not significantly impaired, and proxy desynchronisation was not an issue at that time.

Today however the attacker also approached the admin of one proxy machine, and managed to convincingly impersonate me. This resulted in him acquiring login privileges, which he spent about three hours trying to use to effect trades without success. Eventually his lucky star and an overloaded router severed the proxy graph in such a manner that he was able to seemingly push trades through. It is my estimate that his window had maybe one hour left, and I would guess the odds of the various events required occuring in any given day to be under 1%. Nevertheless, it is factually correct to say that MPEx didn't have enough safeguards in place for the case of hostile proxy on deck.ii

The attacker may have obtained partial, obfuscated copies of the MPEx databases. While I won't go into details with regards to said obfuscation, it is a fact that breaching one proxy is not, as far as I can see, sufficient to obtain complete trade history. As directed in the FAQ, users are expected to take steps within their own control and at their own convenience to ensure their anonimity, and further,

Internet-facing machines do hold copies of the database. They are reasonably secured, which probably means they are more secure than most any other bitcoin-based application at this time. However, their absolute security is not considered a strategic priority and no extraordinary measures are undertaken to ensure it.

Moving to measures taken, we have reported the attempt with the involved datacenter's FBI contact as a matter of course.iii I do not expect this to yield any particular results or be any kind of silver bullet. We have taken steps to significantly strengthen the protocol used for communication with the proxy sysadmins. This will come at a cost of convenience and perhaps in time may delay efforts to recover from other types of problems, but nevertheless, I judge it prudent at this juncture. We are also making a number of code modifications, and the entire proxy farm will be moved on updated software as a matter of paranoia. This will result in a service interruption of about four to six hours, starting just as soon as we have everything together. I will also be taking this opportunity to create a new, stronger MPEx key, which was something that I had been wanting to do ever since the recent NSA debacle but have been hesitant to actually implement on account of the process requiring everything being taken offline.

I will try my best to answer any questions you may have below (and I would prefer you ask your questions below so they may remain related to their context and easily accessible in the future), nevertheless there are topics which I will not discuss in arbitrary detail for various reasons.

———
  1. The "very heavy" epithet used in the log will have to be rectified at this time. It wasn't actually as heavy as it momentarily seemed. []
  2. I do not however believe the design as it was could be called "bad" or "wrong". It is a fact that security and convenience are inverse functions, and I am and remain firmly convinced that increases in deployed security should follow actual need rather than imagination. []
  3. The exact choice of venue is a matter of convenience. Law enforcement shares information in any case. []
Category: MPEx
Comments feed : RSS 2.0. Leave your own comment below, or send a trackback.

10 Responses

  1. It seems these proxies can independently produce MPEx-signed stuff? But that needs GPG private key, how comes the attacker did not pull it from the machine's memory or so?

  2. Mircea Popescu`s avatar
    2
    Mircea Popescu 
    Thursday, 26 September 2013

    This appears as a theoretical possiblity looking from outside the box. I believe it would not work as such in practice, for reasons I won't go into.

  3. hahahahaha

    good job, hacker.

  4. This morning the National Bank of Panama announced that it was suspending all services until Tuesday the 1st of October. The National Bank of Panama says that the reason is to upgrade systems. The Banking Holiday in Panama was announced this am.

    http://news.goldseek.com/GoldSeek/1380224271.php

    Influencere care esti influencer.

  5. Mircea Popescu`s avatar
    5
    Mircea Popescu 
    Friday, 27 September 2013

    Fucking copycats everywhere. And I bet diamonds to doughnuts they didn't even link to Trilema.

  6. lollergirl`s avatar
    6
    lollergirl 
    Sunday, 29 September 2013

    http://www.salon.com/2013/09/26/nationwide_it_glitch_wipes_balances_and_deletes_accounts_newscred/ loller

  7. Mircea Popescu`s avatar
    7
    Mircea Popescu 
    Sunday, 29 September 2013

    Lol ok.

  1. [...] you ask : no, this is in no way related to the recent breach, nor is it in any significant way related to the recovery work. Obviously when people get burned [...]

  2. [...] into our server, or whatever the hell happened here. Sorry for the downtime! ———MPEx security breach ; MPEx breach - post mortem [↩]If we're not cheapskates content to just format the failed [...]

  3. [...] to specific countermeasures implemented after the unpleasant experience last year it is not conceivable that any data has been recovered or will be recovered from the drives in [...]

Add your cents! »
    If this is your first comment, it will wait to be approved. This usually takes a few hours. Subsequent comments are not delayed.