Motto : "how do you spell illiterate ?"
"look it up in the dictionary."
"how am i going to look it up if i don't know how to spell it ?!"

To set the mood, let's start with something light :

All the efforts of their fellow MITtards to embed spellcheckers into everything didn't help either! But let's move on with our show, there's a whole flea circus getting impatient in the box.

~ Part 1 ~

Just at the turn of 2014, the USG's Department of OMFGTGWKU's Czarⁱ was running around with his hair on fire in a spacious if poorly furnished office (hey, what do you want from socialism, plastic "carpeting" and halogen "lighting") trying not to catch his polyester shirt.

As a direct result, assetsⁱⁱ were mobilized and lo and behold :

Looking before the Scaling Up Leap

I've been busy the last few weeks filling up my hard disk with copies of the main Bitcoin blockchain.

My goal is to prove that it is safe to raise the maximum block size from 1MB to at least 20MB, with the existing Bitcoin Core code (no fancy invertible bloom filters, no hyper-optimized-for-the Bitcoin-elliptic-curve verification code).

If the existing code can handle 20 megabyte blocks, then I'll argue we should schedule a hard fork to raise the block size. Ideally, we would never run into the maximum block size limit -- we would always have a situation like we do today, where miners decide how large or small to make their blocks, and people using the blockchain decide whether they're willing to pay miners transaction fees to include their transactions.

So I've been testing the existing code with much larger blocks -- 20MB and 200MB.

I could create artificial tests that fill up blocks with randomly generated transactions, but it would be much better to test with real-world transactions packaged into bigger blocks. I'm running a couple of full Bitcoin nodes, so I have 20 or so gigabytes of real-world transactions.

I just had to figure out how to repackage them into bigger blocks. After a couple of false starts over a couple of frustrating days (wouldn't it be nice if our ideas always worked perfectly the first time?), I came up with the following scheme:

First, I hacked the reference implementation and changed some constants related to the block size (MAX_BLOCK_SIZE, DEFAULT_BLOCK_MAX_SIZE, MAX_BLOCKFILE_SIZE). That part is easy.

The tricky bit is how to take transactions from the 1-MB and repackage them in bigger blocks. For most transactions, it is trivial, because ordinary transactions don't care what block they are in.

There are two exceptions:

1. Transactions that use the 'locktime' feature so they are not valid until a particular block height or date. I dealt with these by simply treating all transactions as 'final'.

2. There are special rules for 'coinbase' transactions: there can be only one coinbase transaction per block, it must be the first transaction, etc. My first couple of false starts went down the path of relaxing those rules, but I gave up that approach because I found myself changing more and more code. I think if I was 22 years old I probably would have just forged ahead, unwilling to let go of a couple of days of programming effort because "just one more change and it will probably all start to work..."

I wonder if old programmers throw away more code than young programmers. I bet we do.

Anyway, after a rethink I came up with a much cleaner solution for handling coinbase transactions from the real blockchain: I write them to a 'coinbasetx.dat' file which the hacked bitcoind reads at startup and stores in memory (there are only 73 megabytes of them). As blocks are added to the big-block chain, those coinbases are added to the "unspent transaction output set" so they are available to be spent (and I set the COINBASE_MATURITY constant to zero instead of 100, so they can be spent right away, in the same big block).

I wrote a tool ("gen_megablocks") that creates the coinbasetx.dat file and blk*.dat files that are valid-but-oversized, -regtest-mode blocks.

Those big blocks are then loaded into the bigger-block-capable bitcoind using the -loadblock command-line option, where they go through full transaction and block validation and indexing.

The end result is a fully-indexed blockchain with real transactions arranged into much larger blocks. I can import some of my main-chain wallet private keys and create transactions that spend them; those transactions are valid on both the hacked big-block chain and the main Bitcoin 1-MB blockchain (because ordinary transactions are completely independent of blocks).

So far, the results are very good-- the time to process a block scales up linearly with the size of the block, there are no hidden O(n^2) gotchas waiting to bite us. My desktop machine (a quad-core, 16GB-memory, "Late 2012" iMac) can easily keep up with a 20-MB-per-block blockchain, and could even keep up with a 200-MB-per-block chain if run with bigger -maxsigcachesize and -dbcache settings.

Today I'm generating alternate chains (I added an option to gen_megablocks to skip one or more transactions and their descendants); I'll be testing long and short blockchain re-orgs, measuring CPU time and memory usage. Once that is done I'll try syncing these 'megablock' chains across the internet and will measure real-world network bandwidth usage and time. And along the way I'll import a few keys and make sure the Bitcoin-Qt wallet works properly with big blocks (it should, it doesn't contain any code that cares about the block size).

Once all that is done, I will write up the results, write up a detailed specification for increasing the block size, and will start working on a patch (and lots more tests) to make it happen.

And I'm bracing myself for endless re-hashing of arguments about why a maximum block size tied to transaction volume or difficulty or exchange rate or total transaction fees or the phase of the moon would be better than some dumb, predictable formula....

History has this fundamental capacity of being forgottenⁱⁱⁱ, so there's a lot of things you probably do not at this point recall. The fact that articles from the period have been liberally deleted^iv doesn't help, either. Let's try a brief list.

One is that up until that point "most everyone" simply followed PRB^v, on the grounds that while "most everyone" (the other end of the everyone spectrum) knew the dev group is about as infested as any other FOSS body, nevertheless "they wouldn't dare" because "a thousand eyes"^vi and all that. This complacency was broken that same month of January 2015, by my dissent and by my dissent alone. Yes, others joined in the breach, much like in the days after MPOE-PRs scathing posts^vii regarding the Pirate scam others joined in the breach and soon thereafter the thing was history.

The second is that the change was misrepresented as a fait accompli. It wasn't a question of whether, to hear the assets talk. Just like it wasn't a question of whether a certain lulzy "offensive" around Tikrit was going to succeed, or even happen at all, if you were dumb enough to listen to the "media" presenting its "fair and balanced" propagandistic narrative.

The third is that the change was presented to have the backing of sugar, spice and everything nice - in this case "science"^viii (just like the "global warming" aka "anthropogenic climate change" aka etc bla bla was supposedly backed by very solid, actual science). That it consisted, then and now, of absolutely nothing at all above and beyond political interest is not important - what's important is that it comes out the right way.

If you are actually a cultivated man, by now you must've recognized the standard homily of Western Christianity throughout its contorted history. The pope would present things as facts. Whence and wherefore, worry not. That's two. The facts as dreamed up then have the backing of "all good things", which is to say the angels - who will surely come and fight the good fight with you. That's three. But the original insanity this all depends upon is that you, and barbarians like you, follow it as a default. Because you gotta follow something, or because it's there, or because this guy has statues and sends you gifts of weird if entirely useless shit, or for no reason whatsoever. That's how you end up butchered in some far away desert - it's not the siren that sung ; it's you, that listened. Remember that.

Fortunately for everyone involved (yes, even the unsuccessful, would-be papacy), this time there was a higher power watching. And so I swiftly negated the "it's done already" side of the charade, rebranding it as Gavincoin instead - yet another wanna-be scamcoin, which happens to be exactly accurate ; sent the cherubs dressed as angels packing with red streak marks on their shameless bottoms, and threatened to bring forth the very word, if things continue.

Things didn't continue. They just morphed.

~ Part 2 ~

Dazed and confused by the "sudden and unexpected" defeat that "nobody could have predicted", the ever-stolid Department of OMFGTGWKU went to work trying to understand what went wrong, while a certain Czar frothed and bubbled quietly in a corner. For their efforts they achieved two firm if incorrect conclusions - one, that the "democratic" astroturf method they had been using is valid and the correct approach ; and two, that failure came from my successful rebranding of their inept efforts. Consequently a new plan was formulated, and assets moved yet again : Bitcoin "XT" was born.

Leaving aside the amusements of how "science" changed meanwhile^ix, I didn't bother to even discuss the matter, leaving some observers a tad unsatisfied. I simply made a token financial commitment, and left it at that.

The defeat was resounding, seeing how the USG troops mostly collapsed upon themselves, under their own, unwarranted and uncontrolled weight. Lulzy.

~ Part 3 ~

Dazed and confused by the "sudden and unexpected" defeat that "nobody could have predicted" yet again, the ever-supine Department of OMFGTGWKU went to work trying to understand what went wrong, while a certain Czar wailed and scratched himself in the basement. For their efforts they achieved two firm and this time also correct conclusions - one, that the "democratic" astroturf method they had been using was invalid^x ; and two, that it's time for fresh assets, the previous names are well and thoroughly burned.

Consequently they activated a group that they had been grooming for a little while to take the public face of NSA's own mining network, and anyone who may be inclined to listen is now regaled with a promise empty of implementation under the name of "Bitcoin Classic". Here's a lulzy bit, that's pretty much all that's worth the mention of this third and doomed approach :

There's a lot more lulz and gomedy cold, if that's your interest, such as how they announced the ever-hysterical

That puts over 50% of the hash rate behind Bitcoin Classic, consensus has been achieved.

on the 15th, which ran into my unyielding

actually i wouldn't go to war over keccak.

on the 17th and within hours the "consensus" participants were coming out to explain that they had said nothing of the sort, merely "welcomed" the effort and basically... "Classic" can get lost. Which is exactly as it should be, seeing how miners have very little freedom of movement and a very large lot to lose (and speaking of people who have a lot to lose - I don't see a future for Bitfury, dear investors in Bitfury, and certainly not under its current management).

All that's left of this third and to date most amusing effort is the hollow shell of a piece of vaporware and some very, very butthurt young eagles over at Ft. Meade. What, darlings, you thought you're immune from being publicly raped by MP because you're "pros" whereas the other two were mere geeks ? Hurr durr, not how it works.

Anyway, here's to eagerly awaiting part four, sometime around July if history is any guide.^xi

———

1. Has everyone else noticed how it's suddenly no longer fashionable to call them that ? Hm ? [↩]
2. That Gavin Andresen was an USG mole had been a matter of public record for a few years at this point. I don't recall anyone ever regarding Mike Hearn as anything else at any point, if not everyone was equally public about the matter. The workings of media, be it "mainstream" or "social" or otherwise as propaganda instruments devoid of other utility or value shouldn't, for sane people, require further explanation. [↩]
3. Here's a juicy bit. [↩]
4. Just as a for instance, "I think I addressed in my Blocksize Economics post" references blog.bitcoinfoundation.org/blocksize-economics/, which of course... [↩]
5. The PR-Bitcoin, a vapid deluge of pointless "improvements" mostly made with a view to give the talkers what to talk about. Heck, TRB didn't even have a public name yet. [↩]
6. You know what else has thousands of eyes upon it ? Rotten meat. [↩]
7. That my agents, rather than "the community", the US SEC or any other authority ended Pirate's Savings and Trust is a historical fact. Yes, it's true that assorted scumbags posing as "respectable and respected community members" such as James Gibson aka gigavps et al found somehow the unmitigated audacity to protest my agents' deeds, and yes it's the joint efforts of that scumbag party that you have to thank for Pirate surviving months rather than days after the indictment (and yes exactly the same scumbags made MtGox drag on for months after being excommunicated at the First Bitcoin Conference). There's actually a very good article from that same source dealing with this. [↩]
8. Which in this case backfired in a particularly amusing way - Gavin tried to package it as "the consensus of economists", a diagnosably insane move.

   For one thing, the "economists" he got to sign off on his insanities were infantile derps a la Oleg Andreev aka Oleganza, barely if at all qualified to sit at the adult table in any field, let alone the very specialised field in question here. For the other thing the very ethos of Bitcoin is that it tried, and succeeded, to show to "economists", especially the consensus-inclined above illustrated bright Harvard minds, that they are not even wrong, and not even economists in the first place.

   It's rather unlikely that Gavin failed to grok the extreme weakness, to a degree of self-parody even, of this move. It is however patently obvious that what Gavin did or did not understand was irrelevant for his owners, constrained as they are by their own bureaucratic world. The Czar didn't understand, if he bothered to even look at all, the people putting nylon fires out didn't understand, if they bothered to even look at all and here we are, with a dead turkey on our hands. Was it worth it ? Well... that's the sort of thing they don't tell you when they bring you those papers to sign. [↩]

9. What's so surprising about all that ? Pseudoscience predicated on political expediency is wont to change as expediency dictates, what of it ?! [↩]
10. Talk about strategic superiority, huh. For as long as I bothered engaging the circus (1, 2, 3), they thought it works. Once I stopped bothering, they decided it doesn't work. Who's who, here ? [↩]
11. Hey, by the way, you know what a timing attack is ? No, it doesn't "just work on computers", security discusses systems not "computers". [↩]