MPEx DDOSed, an epic.

Since Trilema has been down for the interval I'm a little backed up here. If this thing doesn't run ten thousand words I'd be surprised.

On or about March 1st (possibly earlier) MPEx webservers started receiving ever increasing volumes of broken and out-of-context packets.ⁱ This was not particularly remarkable at the time, the Internet is a messy soup, stuff happens.

However, by the 2nd of March the noise was becoming sufficient that maintenance admin on duty was doing stuff to the iptables.

Mar 02 02:24:36 jurov if anyone desperately needs to place orders, mpex.coinbr.com still works
Mar 02 02:24:45 jurov at leat stat does
Mar 02 02:26:15 jurov mircea_popescu , no comment? it's ddos?
Mar 02 02:26:26 mircea_popescu what is ?!
Mar 02 02:26:34 mircea_popescu site's fine ?
Mar 02 02:26:45 jurov i can't access mpex.co .. times out
Mar 02 02:26:51 jurov only via proxy
Mar 02 02:27:00 mircea_popescu tehn you may have a dns issue locally
Mar 02 02:27:05 mircea_popescu but the site is on the web.

Upon examination later that day I shot a

csf -dr ju.ro.v.ip #This is legitimate traffic

and never thought more of it. Just some junior sysadmin being overzealous, right ?

Wrong, obviously. Floods kept waxing and waning but mostly waxing & creeping up until finally, March 6th,

Mar 06 14:13:03 Bugpowder Alt link for mpex? Getting ddosed?
Mar 06 14:14:06 jurov Bugpowder, mpex.coinbr.com

Trade moved to the various proxies and well... that's that, time for a statement.

Mar 06 18:44:33 mircea_popescu if you're having trouble loading trilema, please flush your dns.
Mar 06 18:44:49 mjr_ yay
Mar 06 18:44:50 mircea_popescu and otherwise, mpex,co is being ddosed by someone too stupid to know that doesn't work.
Mar 06 18:44:56 ZedsterX i never load it no problem
Mar 06 18:44:58 mircea_popescu use the various proxies, such as
Mar 06 18:45:07 ZedsterX :)
Mar 06 18:45:17 error4733 hi mp
Mar 06 18:45:28 smickles damn, i thought your sexdungeon revolted

So that's basically it, wait for script kiddie / neckbeard to get bored and leave, right ?

Wrong, obviously. A few hours later...

Mar 07 00:09:29mircea_popescu lol so our friendly ddosers actually wiped the upstream provider.
Mar 07 00:10:10mircea_popescu NOW mpex is offline.
Mar 07 00:10:25jurov indeed
Mar 07 00:11:01mircea_popescu i guess never scorn a script kiddie.
Mar 07 00:11:19Lyspooner hell hath no fury like a script kiddie scorned
Mar 07 00:12:10mircea_popescu anyway. should be resolved momentarily.
Mar 07 00:12:16 mircea_popescu the worst case scenario is we move over to tor.
Mar 07 00:12:28 mircea_popescu that'd likely be a coupla days of downtime.
Mar 07 00:12:37 mircea_popescu in between these extremes a number of intermediate solutions.
Mar 07 00:13:03 Lyspooner Heaven has no rage like love to hatred turned, Nor hell a fury like a script kiddie scorned
Mar 07 00:13:17 kakobrekla so the proxy idea doesnt work after all
Mar 07 00:13:36 mircea_popescu kakobrekla well anything on the internet is limited by the internet actually existing.
Mar 07 00:14:06 BitHub it is?
Mar 07 00:14:09 BitHub dear god
Mar 07 00:14:42 jurov mircea_popescu can't the engine just get moved to completely different dc?
Mar 07 00:14:48 mircea_popescu jurov yes, it could.
Mar 07 00:15:01 mircea_popescu but i dislike this sort of solutions, just hop around dcs ?
Mar 07 00:15:10 mircea_popescu what if i need to take a piss one day ?
Mar 07 00:15:19 jurov and only proxies would know where.. as long as it stays secret, no hoppin necessary
Mar 07 00:15:43 mircea_popescu this is a small scale garden variety tor implementation.
Mar 07 00:15:55 kakobrekla tor sux
Mar 07 00:15:58 mircea_popescu why ?
Mar 07 00:16:00 Lyspooner Uncertainty and expectation are the joys of life. Security is an insipid thing.
Mar 07 00:16:17 mircea_popescu Lyspooner shush you, you're not the one with the suspended exchange.
Mar 07 00:16:34 Lyspooner that's not me, that's william congreve
Mar 07 00:16:48 kakobrekla tor is not friendly
Mar 07 00:16:57 mod6 for whom?
Mar 07 00:17:01 kakobrekla to ME
Mar 07 00:17:11 mircea_popescu kakobrekla mpex would be on tor, proxies would be on normal web.
Mar 07 00:17:24 mircea_popescu problem is it's slow
Mar 07 00:17:39 kakobrekla what the point then
Mar 07 00:17:55 kakobrekla if proxies hide location
Mar 07 00:17:59 mircea_popescu well look, ddos mitigation is an exercise in moving upstream.
Mar 07 00:18:12 mircea_popescu what i had here was enough to make 10 Gb attacks ineffectual.
Mar 07 00:18:37 mircea_popescu someone taking down all tor nodes is a coupla degrees of magnitude over that.
Mar 07 00:19:23 kakobrekla ya but they still know your current location
Mar 07 00:19:34 kakobrekla and you need to change that
Mar 07 00:19:47 kakobrekla if they are fuckin your isp
Mar 07 00:19:51 kakobrekla or what
Mar 07 00:20:00 mircea_popescu half of gnax atlanta is dark atm.

At this point in the game the brilliant idea of redirecting mpex.co to whitehouse.gov popped up.

Mar 07 00:33:10 mircea_popescu try mpex.co nao
Mar 07 00:33:10 jcpham nope
Mar 07 00:33:10 Pucilowski what happened to mpex.co?
Mar 07 00:33:27 pigeons OMG surely there are no alternative sites using twitter bootstrap that will buy bitcoins for us off an exchange for an extra fee
Mar 07 00:34:11 jurov btw, didn't anyone try to compare coinlab sales with pirate's loot? whether the amount isn't approximately similar, perchance
Mar 07 00:34:16 pigeons why'd you wait till june, japan had gained a lot by then
Mar 07 00:34:27 Bugpowder lol
Mar 07 00:34:33 Bugpowder very patriotic
Mar 07 00:34:56 iz jurov: how much did pirate actually make off with? i think it's unknown, b/c everyone won't fess up about how much they got paid back in interest
Mar 07 00:35:04 pigeons it is unknown
Mar 07 00:35:05 mircea_popescu hey, people wanna show off their muscle, let them show it off to people who care.
Mar 07 00:35:17 pigeons i doubt he had much left
Mar 07 00:35:20 iz pigeons: same
Mar 07 00:35:37 Bugpowder is that going to redirect the ddos to wh?
Mar 07 00:35:41 Bugpowder no right?
Mar 07 00:35:52 mircea_popescu i imagine yes.
Mar 07 00:36:01 jurov mpex.co -> http://www.whitehouse.gov LMAO
Mar 07 00:36:22 mircea_popescu i figure i got a 1% shot of making us national news.
Mar 07 00:36:26 Bugpowder heh
Mar 07 00:36:36 Bugpowder its a very clever idea
Mar 07 00:36:43 mod6 lol, that actually worked.
Mar 07 00:37:03 mod6 WE GET SIGNAL
Mar 07 00:37:05 mircea_popescu mod6 what ?!
Mar 07 00:37:10 mircea_popescu don't tell me it's down
Mar 07 00:37:21 mod6 redirect from mpex.co to whitehouse.gov
Mar 07 00:37:24 mircea_popescu a a
Mar 07 00:37:25 mircea_popescu yeah
Mar 07 00:37:26 mod6 i luld
Mar 07 00:37:37 mod6 and needed to subsequently quote zero-wing
Mar 07 00:37:59 jcpham that's terrible mircea_popescu
Mar 07 00:38:16 jurov everyone, now try to submit post requests to whitehouse
Mar 07 00:38:28 jurov with gpg encrypted mpex stuff
Mar 07 00:38:49 mircea_popescu lol it has no hole
Mar 07 00:39:08 Bugpowder not working

To be sure, the outage was momentary, and within minutes the bot was again streaming trades

Mar 07 00:52:55 assbot [MPEX] [O.BTCUSD.C450T] 120 @ 0.22222082 = 26.6665 BTC [+]

and for maximal hybris,

Mar 07 01:07:48 assbot [MPEX] [S.MPOE] 19250 @ 0.00072389 = 13.9349 BTC [+]
Mar 07 01:07:49 assbot [MPEX] [S.MPOE] 9076 @ 0.00073711 = 6.69 BTC [+]
Mar 07 01:08:01 assbot [MPEX] [S.MPOE] 5247 @ 0.00074548 = 3.9115 BTC [+]
Mar 07 01:08:03 assbot [MPEX] [S.MPOE] 41427 @ 0.00074999 = 31.0698 BTC [+]
Mar 07 01:08:43 mircea_popescu o hey. "mpoe surges on news of failed ddos"

At about this time the Bitcoin network was starting to heat up. The various MtGox API livecharts and other charting services (bitcoincharts.com, bitcoinity.com) were going up and down, a little later blockchain.info took an extended break, unconfirmed transactions were piling up by the thousands and the BTC/USD started tumbling (from almost 50 to under 35 within that day).

Chaos, in short. But MPEx was still trading, and ~50k open option contracts were closed in the next hour or so. A few hours later the MPOE bot mostly stopped quoting because it couldn't make sense of the feeds, but by then orders had already stopped.

So, basically... the worst was behind us, right ?

Wrong. The attacker had spent all this time methodically taking the proxies out, one by one. Soon enough,

Mar 07 12:09:40 Namworld both mpex.coinbr.com and mpex.co seem unavailable now.
Mar 07 12:09:53 Namworld Someone is really DDoS'ing everything
Mar 07 12:10:22 maximian yeah looks like coinbr.com is getting the DDoS treatment

Coinbr.com was at the time hosted by BitVPS. Apparently the attack took out their Switzerland DC or whatever slice thereof, and being threatened with expulsion they sent jurov packing. For shame, eh ? Perhaps it's time for an announcement.

Mar 07 12:19:05 mircea_popescu hey all.
Mar 07 12:19:30 mircea_popescu mpex servers will be offline for a few hours. we are reconfiguring the router system, putting in new optics
Mar 07 12:19:37 mircea_popescu and doing a lot of other overhauling.
Mar 07 12:20:00 mircea_popescu no need to panic, all data is safe, by the time this is done mpex will have 10gb dedicated.
Mar 07 12:20:16 mircea_popescu and a few other perks.
Mar 07 12:20:29 Namworld 10 Gbps?
Mar 07 12:20:38 mircea_popescu yes.
Mar 07 12:20:45 Namworld Seems massive.
Mar 07 12:20:50 Namworld Good thing.
Mar 07 12:20:53 mircea_popescu hey. 1gb x2 seemed massive.

The ugly truth behind this is : some people had private proxy access to MPEx. With the fall of jurov's last proxy there was however no public way to access MPEx left. Rather than split the users among the elite, with access, and the plebs, without, I judged it most expedient to simply halt trading. There weren't at the time any positions desperately at risk as best I could determine so nobody risked getting his toes caught in the door, I had the measure announced on the forum with a "if you're desperate to close option exposure contact me" for the people not in the know that may be trapped over there and that was that. In retrospect I think this was the correct move under the circumstances.

Originally the plan was to simply wait for a full day, because new MPEx megapipe was supposed to be deployed early on the 8th. However...

Mar 07 20:16:30 imsaguy how about mpex over irc?
Mar 07 20:16:44 imsaguy let freenode get ddos
Mar 07 20:16:47 imsaguy I hear they like that
[...]
Mar 07 20:17:44 mircea_popescu dawg...
Mar 07 20:17:52 mircea_popescu fuck you srsly.
Mar 07 20:17:55 mircea_popescu why didn't i think of that
Mar 07 20:17:57 imsaguy dee oh gee
Mar 07 20:18:05 mircea_popescu people can just paste a pastebin link or w/e
Mar 07 20:18:06 Diablo-D3 oh em gee

This is brilliant, you must admit. The first bot was up within about six hours, and with it the ability to trade on MPEx could be restored without wrangling over fairness issues. Soon after a 2nd was available, and from there on things went a little wild. It's not just MPEx, you see, and it certainly isn't just MP. The support people showed themselves willing to lend was fantastic. Obviously it was imsaguy's idea. Nanotube generously drafted some module interface for supybot. Smickles made mpexbot out of a stock supybot, nanotube's drafts and some pseudocode (which read something like "I dunno dood, pastebin must have an api"). Kakobrekla wrote the same functionality into assbot. It doesn't stop there, either. FabianB wrote a Ruby (not ROR) expansion to pympex, which is irc bot aware. Gabridome wrote her own. There's talk of minecraft MPEx and MPEx GUI and all sorts of stuff. Don't you wonder how Second Life MPOE-PR looks like ?

Lots more people offered - publicly and privately - to contribute hosting resources. These I mostly turned down, not because I didn't appreciate the offers - which I did and do - but because by now it had become apparent that it wouldn't be safe. Most hosting out there happens on shared servers, whether they are actually called "shared" and go for 5-10 dollars a month or are called "VPS" and go for 50 or so. The principle is the same, one physical machine hosts multiple instances which may look and feel more or less like an "actual" machine but nevertheless are not.

The physical machine itself is most likely a "dedicated server"ⁱⁱ, which probably shares a 100 Mbit port with one or even a few other machines. A single rack can have as little as 2 Gbit actually available, and contain dozens of machines. So a very common structure would be something like 100 Gbit optics -> main router which splits it into 10 Gbit pipes -> subrouters which split each into 1Gbit -> rack routers which further split into 100 Mbit and beyond -> physical machine which splits its resources among the hosted VPS/shared accounts.

The early attack (end of January) actually took out the datacenter's main router. The current one did not, but the frames were piling up at that level and the next router down was unresponsive so IPs had to be null routed. I have no idea how things looked for BitVPS's datacenter - the people in charge didn't say much / don't seem like they know what to say, so my best guess would be that it was similar. I've somehow lost a number of Google Cloud and Amazon EC instancesⁱⁱⁱ, and jurov also lost one. This is not the sort of thing that random interwebs omnipotent dweller can commonly omnipote. This is military grade stuff. That doesn't rule out some private person being behind it, seeing how the Internet is the ultimate individualist medium, and indeed one person can accomplish anything that at all can be accomplished. Nevertheless, the resources employed were vast. In these circumstances, accepting someone's offer to run a MPEx proxy off their VPS wouldn't have been something I could in good conscience do, since I knew the future so to speak, and it didn't look bright for the would-be helper.

And that pretty much brings us up to date : trade has been ongoing without problems via the irc bots since then,

Mar 10 20:21:07 gabridome http://mpex.coinbr.com/ on line! Am I drunk?

Third generation proxies are online, an .onion MPEx proxy is ready and going to be released^iv. Unfortunately some of the hardware being delivered for March 8th proved defective, the 9th and the 10th being the weekend saw little movement, and so the new MPEx public website should be arriving sometime this week. If you're reading this post it means it's just happened, you can compare my estimate to actual reality.

That's the whole story. I said before that MPEx is immune to DDOS attacks and I am saying it again now. The fact that two weeks' worth of DDOS^v resulted in half a day's worth of trading being paused^vi supports this notion.

At the end of the day, MPEx finds itself in a much better position to withstand similar attacks in the future (not similar in size, similar in type, irrespective of size). This is exactly the lesson I wish embedded in the skull of any would-be attacker, incidentally (again, irrespective of size - for this application I see no difference between the government of the United States and some random lonely guy sitting among piles of empty pizza boxes) : the more you try the stronger MPEx gets. The only way this story ends is with the offender's abject submission, because for as long as there's anyone who has an inkling of hope that they may cause trouble for MPEx, I will be here and I will humiliate them into oblivion. Again and again and again until either suicide or capitulation, unconditional and heartfelt, releases the would-be attacker from his mindless entrapment.

MPEx is forever. The world may not make it that long.

PS. If you're thinking this article is a little heavy on irc quotes, consider that the channel log is over 1.3 Mb for the current month. You're getting off easy.

PPS. Everyone named in this article for helping gets a block of 10`000 S.MPOE shares. Contact jurov to take possession.

1. These are the bogonic building blocks of any DDOS attack. Abroken packet is simply a packet that either misses some part or for whatever reason isn't quite able to stand on his own. An out-of-context packet is something like a large authoritative DNS response that was never solicited (and boy were we well informed of the state of various DNS records this past week) or a SYN that's never followed by an ACK, or an ACK that never had a SYN etc. [↩]
2. As opposed to "colocated" - the difference is strictly one of ownership, roughly following the distinction between renting and owning. [↩]
3. In fairness I didn't have the time/resources to look into exactly what happened there. [↩]
4. I did not wish to release it before ample alternatives exist because of concerns that the DDOS might actually cause significant problems on TOR - not necessarily by taking it offline but perhaps by separating its graph in undesirable ways. It's a complicated topic. [↩]
5. Which would cost north of $100k to replicate, by the way, if you had to hire the techs and the bots. My costs defending from it were a tenth that or less, and most of it went to provisioning new hardware that's not yet online, so could as well have been eschewed. [↩]
6. Because I didn't want to go to private-only mode which, incidentally, is exactly how finance works IRL, they've long lost this battle I've just won. [↩]