Unsorted collection of various Cardano related updates + spurious pics

I. The question of entropy debit :

The RNG in copper sheet shielding, being tested.

mircea_popescu So one measure quotes entropy at about 1k/sec. You said 100 baud above. Which is the better approximation ?
asciilifeform 2 different engines, 1st was straight von Neumann, 2nd also von Neumann but with multiple iterations xor-ing over a buffer.

mircea_popescu Yes yes, but generally per rng board, what'd you guess the entropy debit is ?
asciilifeform Why guess, let me measure.

mircea_popescu Aite.
asciilifeform 200 baud sharp.

II. On the question of competition, market leadership, intellectual leadership and all the rest o that good stuff :

The RNG in copper sheet shielding, being otherwise tested.

asciilifeform Found out some quite surprising things about how RNG tests are (mis)-used in practice. Seems like many commercial (read: USG-approved) RNG widgets run in batches, and reject buffer if it doesn't pass 'FIPS'. Can you see the problem here, or do I need to draw a picture?
mircea_popescu Here's another half to this problem. Suppose all transformers ever sold in a geographical area for customer use have well known EM-soup form factors.

asciilifeform Beyond the usual 50/60hz?
mircea_popescu Could you, on the basis of 1. the rng widgets running the way you describe and 2. having a pretty good clue as to what EM is going about around the unshielded rngs obtain a narrower keyspace to explore than the math seems to imply ?

asciilifeform You'd need a very special kind of idiot to design an RNG that picks up line noise.
mircea_popescu Nono. It's not line noise. You run your computer next to the TV. The computer is making a GPG key. Meanwhile the TV is pumping out radiation. You know, just like your cellphone.

asciilifeform Well an ordinary Linux PC traditionally uses key/mouse intervals (low bits) for entropy generation.
mircea_popescu But an avalanche diode system (ie, 99% of the "professional" RNG solutions)...

asciilifeform An avalanche diode, if not shielded, will pick up all kinds of fun stuff.
mircea_popescu Right. Fun stuff which is predictable somewhat, as per 2, and then filtered as per 1, becoming a little more predictable. These together probably make RNG-based systems perversely less secure than good old linux on desktop.ⁱ

asciilifeform A, but, the buggers didn't bother with trying to diddle chinese TVs into compliance. They do it in software, the victim himself provides it, enthusiastically.
mircea_popescu Not diddle them, just measure them. All you need to know is "most" of the EM around.

asciilifeform The perverse beauty of what is actually being done is that you could use the Devil's own arse as an entropy source, but if you apply FIPS (or whatever) and reject batches of 20kb based on the result, you're adding pattern. Think about it.
mircea_popescu Yes you are.

asciilifeform Imagine you have all but the last bit. So you have to guess that last bit. You pick the one that... gives better FIPS! And so on, for n-1...n0.
mircea_popescu Exactly. I'm just saying, a little bit of here, a little bit of there, in the end maybe keyspace is like 2^24 or some shit.

asciilifeform Even smaller, if you (like virtually everyone) use straight whitening. E.g. AES or blumblumshub.
mircea_popescu Right. I think few people realise just how weak their crypto is actually.

asciilifeform If they did, they'd shit themselves.

[a little later]

Populated RNG board, top view

asciilifeform People in Schneier's peanut gallery: 'somebody should make hardware to stand up to (classic)ⁱⁱ NSA!' Reply: 'but nobody knows how!'.
mircea_popescu Lol.

asciilifeform The comments are full of 'Ruiu' types (the bugs, they're everywhere! my oven's infected).
mircea_popescu I suspect that's the intended effect. Anyway, I wonder if the "powers that be" in their own imagination realise they're proper fucked yet.

asciilifeform Nah, they're wanking under their blanket, to 'full spectrum dominance!'
mircea_popescu Lulzy.

III. The question of treating your customers right :

asciilifeform Btw, are we going with immersion gold plating on this one? (~$1/board)
mircea_popescu Remind me what this is about.

asciilifeform Ok look at 2 photos. 1) rng board 2) manufacturer's demo board for micro seen in last shot. Notice the colors of the contacts. Gold is an optional extra at most board houses.
mircea_popescu But what does it do ?

asciilifeform Sits there without oxidizing. Also shines and looks spiffy.
mircea_popescu Figure it's worth it ?

asciilifeform I'm asking you. You're the marketing sorcerer here.
mircea_popescu Myeah. 1 dollar is exactly rightly priced. I can see it either way.

asciilifeform I'll get an exact quote from golden turd, with and without.
mircea_popescu Eh what the hell, nobody ever went broke by throwing away gold at their customers. So let's.

asciilifeform Works.

As you can see, I love you all. Next product we add the diamond.

IV. The question of entropy quality :

asciilifeform Entropy = 7.981308 bits per byte.
Optimum compression would reduce the size of this 5788119 byte file by 0 percent.
Chi square distribution for 5788119 samples is 137163.04, and randomly would exceed this value less than 0.01 percent of the times.
Arithmetic mean value of data bytes is 127.3688 (127.5 = random).
Monte Carlo value for Pi is 3.139591535 (error 0.06 percent).
Serial correlation coefficient is 0.017729 (totally uncorrelated = 0.0).

For the sample I sent you earlier, output is from John Walker's 'ent' util.

[later]

asciilifeform Don't publish the dump. Found an artifact in there. Accounts for the less-than-perfect test result.
mircea_popescu Ha! Wasn't going to, but what artefact was it ?

asciilifeform Comes from a null word that the serial tx engine shits out. Fixing now.
mircea_popescu Ha!

asciilifeform See, this is why /nothing/ can replace the Mark One Naked Eye. And a hexdumper.
mircea_popescu You know I'm publishing this.

asciilifeform Sure. If you like. People are gonna laugh, you know it. But if you think this helps the cause... sure. Go ahead. Damn I feel like a moron. 4 bytes of every 512 block were nils. See, this is why I compulsively massage the beast. Even when it 'seems to work'.
mircea_popescu Hahaha.

asciilifeform Classic.
mircea_popescu Totally.

mircea_popescu The difference between this and moronity is that 1. you caught it and 2. you admitted it. Shit's supposed to not work by itself, so merely discovering that it ain't working isn't in any way damning.

[later]

asciilifeform Fixed. Incidentally, passes 'FIPS' now. We can sell to the Fuhrer!
mircea_popescu All of them ?

asciilifeform All of them. Not that the FIPS tests are worthless, but the practice of throwing buckets away is laughable.
mircea_popescu FIPS rejected nothing at all of how many MB ?

asciilifeform So far, 200kB. Remember, thing is slow. Ima leave it running.
mircea_popescu I would like you to do a 1gb sample.

asciilifeform It will piss as long as we like. I'm gonna see if I can shave a few of the xor iterations off and get similar metrics.
mircea_popescu Nah.

asciilifeform Ok.

[later]

asciilifeform Entropy = 7.988682 bits per byte.
Optimum compression would reduce the size of this 64409600 byte file by 0 percent.

Chi square distribution for 64409600 samples is 507404.32, and randomly would exceed this value less than 0.01 percent of the times.

Arithmetic mean value of data bytes is 128.3681 (127.5 = random).
Monte Carlo value for Pi is 3.128231541 (error 0.43 percent).
Serial correlation coefficient is 0.000002 (totally uncorrelated = 0.0).

mircea_popescu Sweet.

[later]

mircea_popescu Ok, we have some nice numbers here.
asciilifeform Maybe, maybe not; I've a sample purporting to be 10 megs of Geiger here. Gives Pi to 0.01%

mircea_popescu Matters not. The 0.000002 serial correlation coefficient is a thing of beauty. And looking at the crossingsⁱⁱⁱ I'd say it's just fine, with all my experience of following financial indicators and resisting the urge to imagine patterns where clearly no patterns exist.^iv
asciilifeform Have you ever wondered how paradoxical, from the engineering point of view, building an RNG is? Normally you want a machine to behave deterministically...

mircea_popescu Mhm.
asciilifeform ... and try to plaster over the noisiness of the physical world. Johnson noise, shot noise, etc. Engineers hate them. But RNG is a walking-on-your head sort of affair where you love and nurture noise and non-determinism. (Johnson noise is us. The fact that a resistor, any time above 0K, has a charge across it at all times. Just sitting on your desk.)

mircea_popescu Mno. Listen, you got a bias in here. For ones.
asciilifeform Yes. Trivial to fix, but i'd like to learn why it's there. It disappears after the von Neumann stage, but reappears upon the xor.

mircea_popescu About .8 which means about 1 in 160.
asciilifeform Interestingly, other RNG builders von Neumann before final output stage, but I never knew why.

mircea_popescu Explain the process to me like I was a 3 yo.
asciilifeform Ok, we have the board. It puts out a logic-level signal, 1 or 0, transitioning whenever it feels like it. No clocking. Micro samples the line. Every time it does so twice, the following is done: '00' or '11' -> nothing ; '01' -> we have a '1' ; '10' -> we have a '0'.

mircea_popescu Why is this ?
asciilifeform This is von Neumann's 'fair coin' algorithm. Given an arbitrarily weighted coin, you get a fair coin (50/50 odds). If we use the stream directly after this, we get fairly good entropy, but we're vulnerable to transients.

mircea_popescu You're implementing vN fair coin backwards. Or w/e, bigendian style.
asciilifeform Inverted fair coin is still fair.

mircea_popescu Yup. But do me the following favour : invert your fair coin algo. Let's see if we end up with 126.9. You know ?
asciilifeform Sure, why not.

[later]

asciilifeform Interesting... about the same result.
mircea_popescu So it's not really to do with the chip. We're fucking it up afterwards.

asciilifeform Wondering if it's to do with rs232 again. (It's how we're hooked up to pc at the moment). There's no error correction...
mircea_popescu What'd it correct for ?

asciilifeform Notice the 'idle' state of the line is a 1. So a frame error would give '1'.
mircea_popescu Ha! Well then! That's it baby. You get 0.6% line errors.

asciilifeform Idea: I'll clock down the baud rate on the tx end, let's see what that does. Right now txmitter is putting out 115200. Easy to test hypothesis.
mircea_popescu Let's see.

asciilifeform 9600 baud; still there, but appears to be lessened. I'm convinced it's to do with the line. Switched off the xor part, to get something exactly like the setup i had with the (polled) FTDI chip. Bias still there.
mircea_popescu Listen. Implement the fair cointoss at this end rathert than at that end.

asciilifeform PC end?
mircea_popescu Yup.

asciilifeform Sure!
mircea_popescu You wanna hear something funny ?

asciilifeform Sure.
mircea_popescu The 4 nils in 512 blocks you fixed.... they would have just about balanced this out

asciilifeform Haha.
mircea_popescu PUT THEM BACK IN!

asciilifeform Ehehe
mircea_popescu :D

asciilifeform I bet some people do just that!
mircea_popescu You got any idea how often this is just how business works ? Yeah.

asciilifeform 'Fucker doesn't wurk, whack it with this here hammer'.
mircea_popescu "We don't have the time for this. Make it do it something."

asciilifeform Don't forget, I live in the place where they mine this kind of thinking and export to the rest of the planet. That's the main U.S. natural resource export, not helium...
mircea_popescu Nah, the Chinese are worse. Way way worse.

Long story short : we've been massaging this thing for weeks. We're looking everywhere we can think of, cutting no corners, discovering in the process a bunch of likely pitfalls you can feel more than free to ask any other vendor about, just for kicks, and reinforcing the whys and wherefores engineering is fun. Because it is fun. Done in the proper environment, for a proper business, with actual management, engineering is the most enjoyable passtime intelligent people with tinkering inclinations could possibly pursue.

I can't guarantee that we've found all the holes, nor do I think anyone else could. We have however found a bunch of different holes, which we have fixed, which was the point of the entire exercise in the first place. And I can definitely guarantee that what we do when we find a hole is fix it, rather than anything else.

1. This point couldn't be overstated enough : the way soft power works isn't that you try and control everything. The way soft power works is that you merely focus on doing a good job controlling that end of the stick which has already self-selected to indicate you should be controlling it.

   In this sense, your going "upmarket" as far as privacy products are concerned hurts your privacy, in that it signals your enemy you have secrets to keep, which is in no way offset by the "advantage" than you now have a "better" (as classified by your very enemy) yet still enemy-approved item.

   The first step towards actual privacy is acquiring devices that weren't issued by the enemy. [↩]

2. He means fiat. [↩]
3. We had the file ent'd every 100kB, precisely to see how the correlation moves. Here's the result, if you're curious. "The crossings" refers to where that indicator crosses 0. [↩]
4. Yes, I think TA is nonsense, generally speaking. [↩]