MPEx security breach

Today in the interval 2 pm - 7 pm GMT MPEx suffered a layered attack which eventually resulted in unexecuted trades being relayed through to the market data feeds, brokers et al. We will go here into the details of what happened, why, how and what measures have been taken as a result.

First off, all funds are safe. No sum of BTC was obtained by the attacker as a result of his efforts. Second off, market data is also safe, in the sense that there exists no ambiguity as to what orders are in fact legitimate and what orders were illegitimately conveyed.

Unfortunately this also means a number of trades that appeared to be made in the respective interval will be rolled back. Third parties can trivially verify which is which, because while the legitimate orders contain tracking information, the illegitimate orders do not. Affected are a few dozen trades in the S.MPOE and S.BBET symbols.

In order to understand what happened, we will have to refer to the earlier tech stuff article. The particular architecture MPEx employs leaves it vulnerable to a situation very similar to what is known in Bitcoin as a 50% attack. That is to say, in some circumstances the proxies caching content from MPEx may find themselves in the situation of incorrectly identifying data as valid.

Such circumstances occur rarely under very heavy load. The first occurrence was on April 12, 2013. At that time one user had a trade erroneously reported as executed when in fact his counterparty had already cancelled his order, resulting in a pretty ruffled user and a sustained effort to refine and improve the proxy synchronisation mechanics so as to avoid such occurrences in the future.

MPEx has been sustaining significant floods since September the 22nd, peaking at around 3.2 Mbⁱ. Nevertheless, outside of transient page timeouts trade was not significantly impaired, and proxy desynchronisation was not an issue at that time.

Today however the attacker also approached the admin of one proxy machine, and managed to convincingly impersonate me. This resulted in him acquiring login privileges, which he spent about three hours trying to use to effect trades without success. Eventually his lucky star and an overloaded router severed the proxy graph in such a manner that he was able to seemingly push trades through. It is my estimate that his window had maybe one hour left, and I would guess the odds of the various events required occuring in any given day to be under 1%. Nevertheless, it is factually correct to say that MPEx didn't have enough safeguards in place for the case of hostile proxy on deck.ⁱⁱ

The attacker may have obtained partial, obfuscated copies of the MPEx databases. While I won't go into details with regards to said obfuscation, it is a fact that breaching one proxy is not, as far as I can see, sufficient to obtain complete trade history. As directed in the FAQ, users are expected to take steps within their own control and at their own convenience to ensure their anonimity, and further,

Internet-facing machines do hold copies of the database. They are reasonably secured, which probably means they are more secure than most any other bitcoin-based application at this time. However, their absolute security is not considered a strategic priority and no extraordinary measures are undertaken to ensure it.

Moving to measures taken, we have reported the attempt with the involved datacenter's FBI contact as a matter of course.ⁱⁱⁱ I do not expect this to yield any particular results or be any kind of silver bullet. We have taken steps to significantly strengthen the protocol used for communication with the proxy sysadmins. This will come at a cost of convenience and perhaps in time may delay efforts to recover from other types of problems, but nevertheless, I judge it prudent at this juncture. We are also making a number of code modifications, and the entire proxy farm will be moved on updated software as a matter of paranoia. This will result in a service interruption of about four to six hours, starting just as soon as we have everything together. I will also be taking this opportunity to create a new, stronger MPEx key, which was something that I had been wanting to do ever since the recent NSA debacle but have been hesitant to actually implement on account of the process requiring everything being taken offline.

I will try my best to answer any questions you may have below (and I would prefer you ask your questions below so they may remain related to their context and easily accessible in the future), nevertheless there are topics which I will not discuss in arbitrary detail for various reasons.

1. The "very heavy" epithet used in the log will have to be rectified at this time. It wasn't actually as heavy as it momentarily seemed. [↩]
2. I do not however believe the design as it was could be called "bad" or "wrong". It is a fact that security and convenience are inverse functions, and I am and remain firmly convinced that increases in deployed security should follow actual need rather than imagination. [↩]
3. The exact choice of venue is a matter of convenience. Law enforcement shares information in any case. [↩]