The sad story of me sniffing varnish

When you're running through hundreds of GB's worth of bandwidth each month, and serving millions of hits, the thought might well occur that maybe you know... some caching would be a good idea. And so you figure hey, varnish worked splendidly in the past, should try it again.

When you're a bright young chap with managerial (but nevertheless!) exposure to IT, and a server that's yours for the breaking... you might figure that hey! Let's have some fun!

Sure enough, varnish installs in five minutes. Then you set apache to listen to port 8080, set varnish to connect on the back end to 8080 and to listen on 80 and you're all set. Except whoops... suddenly now qntra.net returns pages from trilema.com

So you spend a little learning a new domain-speciffic language called vclⁱ, which is pretty retarded as far as languages go, but anyway. You create a config that correctly identifies the respective dedicated IPs and has varnish serve accordingly. Victory!

At which point you notice that... well obviously varnish is a proxy, and so Apache would see its IP rather than the actual users'. This is a problem, for many reasons we won't go into. But if it's a problem, it also can be fixed, right ?

Wrong.

For one thing, there's two Apache modules that do the kindness of translating X-Forwarded-For headers into actual IP addresses : mod_rpaf and mod_remoteip. Not only neither of these is included in apache by default, or in any way handled by cpanel, easyapache or any other automation script, but! - the latter only works for Apache 2.4.x and above, which we don't use for [flimsy] reasonsⁱⁱ, whereas the former... no longer exists. Just like that, fuck you, it used to be on "stderr.net" but is no longer and if you want some random dude's github'd versions you're more than welcome. Signatures ? WoTs ? What is this, the future ?! Download random code off anon derps' social media profiles or go home!11

So what do you do ?

Eventually you find a "backward porting" of mod_remoteip for apache 2.2, because yeah, shit is so broken you actually have stand-alone modules that need to be backported. What is the point, even, why are they stand-alone then ?

At which juncture, what do you do ? Compile random code and throw it on the server ? Or do you take the high road and actually read and review random code ? For free ? Why not put some valuable work and effort and time nobody's going to give you back into something that's never ever going to be useful to anyone !ⁱⁱⁱ

And so you do, and so the day passes, and lo and behold! You now have IPs passed correctly.

Mostly.

Oh joy of having succeeded at a task you set for yourself! Oh... what the shit ?

[Tue Apr 28 17:23:08 2015] [error] [client �_] client denied by server configuration: /home/trilema/public_html/wp-admin/edit-comments.php, referer: http://trilema.com/wp-admin/edit-comments.php
[Tue Apr 28 17:23:04 2015] [error] [client 8�b] client denied by server configuration: /home/trilema/public_html/wp-admin/edit-comments.php, referer: http://trilema.com/wp-admin/edit-comments.php
[Tue Apr 28 17:18:25 2015] [error] [client ] client denied by server configuration: /home/trilema/public_html/wp-admin/images/wp-logo.gif, referer: http://trilema.com/wp-admin/edit-comments.php
[Tue Apr 28 17:18:23 2015] [error] [client @�_] client denied by server configuration: /home/trilema/public_html/wp-admin/js/edit-comments.js, referer: http://trilema.com/wp-admin/edit-comments.php
[Tue Apr 28 17:18:20 2015] [error] [client �a] client denied by server configuration: /home/trilema/public_html/wp-admin/js/common.js, referer: http://trilema.com/wp-admin/edit-comments.php
[Tue Apr 28 17:18:19 2015] [error] [client a] client denied by server configuration: /home/trilema/public_html/wp-includes/js/hoverIntent.js, referer: http://trilema.com/wp-admin/edit-comments.php
[Tue Apr 28 17:18:18 2015] [error] [client ��a] client denied by server configuration: /home/trilema/public_html/wp-includes/js/jquery/jquery.js, referer: http://trilema.com/wp-admin/edit-comments.php

Dude... seriously ?!

Seriously. About one in ten requests, for reasons incomprehensible, has random junk in the IP field. So you spend a few hours trying to debug this shit and give up, because by now a whole rack of servers'd have been cheaper than what you'd be billing yourself if you were going to bill for this wasted day at all.

In conclusion :

jurov There's this saying about inspiration and perspiration, you know.
mircea_popescu Dude, come on, it's ridiculous. Imagine if cars worked like software works.

"Sorry honey, can't come to dinner, I'm apparently unable to find the entry to the Brooklyn Bridge today."
"Yes I know five billion people found it before. And yet..."
"Musta been a different car - and - gasoline combination I guess. Going back (on foot) to gas station trying some more combos."

jurov Oh, they did, until very recently.
mircea_popescu Cars NEVER worked like software works, outside of a Laurel and Hardy short.

jurov I have 14y old Daewoo. We can split "never ever" hairs but the end result and frustration is the same.
mircea_popescu See jurov, the thing is... I have the OPTION to not own obscure exotic cars whose maker went out of business shortly after starting because he sucked. I can buy a fucking BMW. There ISN'T a BMW here. It's varnish or nginx. that's it. Not like I'm running unheard-of-software.

There's "free as in beer", there's "free as in freedom", and then there's "free as in your time's not worth anything". That's FOSS.

There's no value and no utility in "Free and Open Source". It's a pastime, for a particular sort of imbecile (adolescent boys), like romantic comedies are a pastime for a different sort of imbecile (retarded women). It doesn't do anything, it doesn't achieve anything. It solves no problems - merely creates them. It's pure masturbation, broken piled upon broken that "sort-of works" and "kinda does a 90% of a job". Useless, pointless, offensive bullshit through and through.

The only people I hate more than USG employees are... everyone on github. Go die in a fire. All of you.

1. Varnish Configuration Language [↩]
2. See libnss discussions for context. [↩]
3. I made a faint attempt to circumvent this problem. It went about as well as you'd expect. [↩]