Fetlife, the meat market
Fetlife is a facebook clone on a niche. This rarely succeeds, but in this case it did - a decade ago collarme ruled the BSDM seasi, meanwhile Fetlife apparently took over.
This, I will propose, is mostly "luck", in the sense that it is based on anything and everything but "coding skills". Just like Spolsky's success (stackexchange). Such projects need a great strategist (and I do mean great, I mean better than what fiat sovereigns can affordii) and connectionsiii, that's the story of success, being the complete story of economics. Technological questions are entirely secondary considerations.iv
Its success brought out the whiny idiots en force, of course. Here's a quotev :
In my usual style, I gave a prepared talk and presented an accompanying slideshow. My talk was called “FetLife Considered Harmful: The Risks of Sex Ghettoization,” and I discussed what I see as a deeply dangerous, insular, growing monoculture within sexuality communities, epitomized by FetLife.com. This monoculture whitewashes the effects of privilege hierarchies while simultaneously reifying them in
That nonsense asidevi, Fetlife does try something or the other in the general vein of "being nice". For instance, it doesn't have a search function implementedvii because "it doesn't want to turn into a meat market". Which is silly, seeing how the first driver for people getting together in the first place, and therefore the foremost underlier of all society, is exactly trade. Not to mention that the prototype of all trade is the trade of women for that purpose. Whether done by the women themselves or whatever way it's organised, that is still the most important function of the marketplaceviii, which is the most important underpining of society. So... tough.
Let's turn fetlife into a meat market together.
for i in {1..3000000}; do curl --cookie "FL=00050ee2-a893-1536-9272-047f87682456;_fl_sessionid=9c69a3c9bb86f4b1f6ff74064e788824" https://fetlife.com/users/$i | egrep -A 64 "[1-2][0-9]F " | sed 's/<[^>]*>//g' | tr -d "\n" | tr "\t" " " | tr -s " " >> fetlife.txt && printf "======$i\n" >> fetlife.txt; done
What's this then ? Well, first of all, Fetlife never heard of salted hashing (like other experts) therefore their userbase is trivially enumerated at the uri levelix. So that's what the for is doing, iterating over their database.x
Then it calls curl, which passes along the relevant cookiesxi and queries pages.xii They're then put through a match to throw out everything but the good part of the profiles made by women under the age of 30xiii. The results are then further processed (removing spurious line feeds, tabs and whitespace) and dumped into a file. From whence you can peruse it at your leisure.
If you run this on a home connection you should go through about one page a second (per instance) so the whole shebang shouldn't really take you a whole month. If you fire up a server cluster it can probably be done in an afternoon.
So there you have it, you've now turned Fetlife into a meat market. What now ?
PS. If you're on Windows, get off, seriously. Or at least install cygwin. No, firefox plugins are not an equivalent solution for this problem, God love you.
———- It imploded recently, yielding some lulzy drama. Apparently it was run by a coufple which split up, the chick ran away with the domain, put up a lengthy if inconsistent sob story about how she had been mistreated, the guy ran off with the db/site, recreated it under a new name. [↩]
- This is possible because of that old "how much does your enemy have to pay you" dilemma.
Specifically, inasmuch as fiat sovereigns are by definition the natural enemies of great strategists (and great people generally, and thinking people even more generally), anyone that's not a fiat sovereign starts with a major head start.
This, obviously, has not been known before. But Bitcoin came around and disrupted the shit out of it. [↩]
- Ie, membership of the relevant WoT. Obviously most niches are still run on some sort of feudal or fiat remnant, so you will rarely encounter a neat and clean implementation of a WoT that'll readily be recognizable on formal grounds. Nevertheless, in some manner and to some (likely pitiful) degree, that's what it'll be. [↩]
- Think : the world existed and functioned before any arbitrarily chosen technological advance. Women rode the cock, people were happy, soups were had and bottles broken. If your idea of success is anything but "at my terminal surrounded by fetid odors" then this should be sufficient proof that technology is a second rate consideration. [↩]
- I'm not going to link to him, because he didn't publish my comment. It read "Eh, just embrace it. This derpy “feminism” thing is dead and buried, a minor niche of less interest than foreign market shut-ins." which is no excuse not to publish it.
Instead, I'm going to share a pic, so he gets to feel violated even moar. Here :
The Derpful Hipster
Nice manboobs, Meitar! [↩]
- By the way, did you know there's a "feminist antiporn" movement thingee ? O yeah, they're totally happening. [↩]
- I suspect this is because the codebase sucks and the stack underlying the site is so rotten they simply couldn't economically run a search, but hey, who am I and what do I know about such matters, right ? [↩]
- As a complete tangent : Dreptul la constiinta
De alt exemplu, in Cairo, care-i in Egipt, care nu-i neaparat perceput ca un varf de lance in ce priveste sofisticarea sau inteligenta organizarii societatii toate magazinele, inclusiv farmaciile sunt pline de ciurde de fete de maritat, 16-20 de ani, pentru ca ei asa fac pe post de discoteci, feisbook scl : trimit fetele sa stea in magazin. Rationamentul nu-i neaparat defect, dat fiind ca tipii care cumpara chestii au, logic vorbind, bani, si daca tot au bani poate doresc sa-si cumpere si-o vacuta bipeda si frumos bronzata. Nu ?
which comes to
As another example, in Cairo, which is in Egypt, which is not necessarily perceived as the bleeding edge of sophistication or the organisation of society, all the shops, pharmacies included, are packed full of gaggles of marriageable girlies, 16-20 years of age. Because that's what they do instead of Discos, facebook and so on : they send the girlies to sit in the store. The reasoning's not necessarily broken, seeing how people who buy things necessarily have money, and since they have money perhaps they also wish to buy a bipedal cowsy with a nice tan ? No ?
[↩]
- This, other than horrible coding, is also very bad for business. [↩]
- They claim "over 3mn users", which is why the upper bound. Feel free to tinker. [↩]
- You obtain these by opening a session in a browser and authenticating. Fetlife doesn't allow you to see any content unless you're logged in, which is great because it keeps the search engines out, and something they should be commended for. It also has the dubious effect of creating an illusion of security among the more herbivore of its users, but I suspect that's unavoidable. [↩]
- You might pass along a benign looking user agent, with -A. I didn't bother deliberately, and to no ill effects so far. [↩]
- Which in my experience are the more educable representatives of the gender of interest.
Much older than that and you run into a lot of "you don't teach old mares new tricks" plus all the attendant baggage (you might not be interested in your women having other people's children, for instance). Nevertheless, feel free to alter the "[1-2][0-9]F " part to suit your tastes. Notice that the final space has to be there. [↩]
Saturday, 14 February 2015
What is salted hashing?
Saturday, 14 February 2015
Suppose you actually have 3 mn users. What you are currently doing is, you're referencing them by a count, /user/1 to /user/3000000.
The correct way to do this is : take a user variable (say, their name, "Derpy McHerperson"), add a salt to it (this will be fixed and should be long and random, say "jUlXk2RvBB695XfkyG/7/DOX5mdkwJ6VD26f+iZ"). Now, hash them together :
sha512sum Derpy McHerpersonjUlXk2RvBB695XfkyG/7/DOX5mdkwJ6VD26f+iZ
3b9ff5d44b403df3bd9efb3967644facf50e638f3a03dcd8f9157c2c83b45a2fdc
9d436eb782767a186c4604ab6886bfa69664c14c81ee1e6009fbecb10849a1 -
You can now reference the user as /user/3b9ff5d44b403df3bd9efb396
7644facf50e638f3a03dcd8f9157c2c83b45a2fdc9d436eb782767a186
c4604ab6886bfa69664c14c81ee1e6009fbecb10849a1
If such a long url is inconvenient for whatever reason (say you wish for users to be able to jot down another user's profile link even if they don't have a computer available - how this works is anyone's guess, but maybe they have internet connected paper) you still have to remember that if your url is made out of numbers your space is merely ten digits, whereas if you use low and uppercase plus digits your space is 64 "digits". So 5 characters which may be [A-Za-z0-9] are roughly equivalent to 9 characters which may be [0-9] (by the following formula : 645 = 26 * 5 ~= 10 9 - because 1024 is 210 and also is ~= 1000 so then 210 is ~= 103).
So you base64 your string, and take say the fifth to tenth characters :
$ base64 3b9ff5d44b403df3bd9efb3967644facf50e
638f3a03dcd8f9157c2c83b45a2fdc9d436eb782767a186c4604ab6886bfa696
64c14c81ee1e6009fbecb10849a1
M2I5ZmY1ZDQ0YjQwM2RmM2JkOWVmYjM5Njc2NDRmYWNmNTBlNjM4ZjNhMD
NkY2Q4ZjkxNTdjMmM4M2I0NWEyZmRjOWQ0MzZlYjc4Mjc2N2ExODZjNDYwNG
FiNjg4NmJmYTY5NjY0YzE0YzgxZWUxZTYwMDlmYmVjYjEwODQ5YTE=
And so Derpy McHerperson becomes /user/mY1ZD
Why go to all this trouble ? Simply because a 3rd party (in the article above, me) is not now able to go through your entire database and enumerate all your users : they're not ordered but distributed over the space, and their relative density is low - if you have 3mn users, and the 5 character space can keep about one billion users, then their density is about 0.3% which means that if I try a random string I have about one chance in 333 to hit on an actual user. This means that I will have to make one billion requests rather than 3 million requests to enumerate your database, or in other words you've put a 333x factor on the effort I need. And if you use six instead of five characters, that's a further 64x, taking it to over 20k tries per result.
This is your first step into the wonderful world of cryptography. Make more, because survival in the future depends strictlier on familiarity with these topics than on paying your taxes.
Friday, 17 April 2015
lol
Monday, 4 May 2015
Seems like right now FetLife auto-bans user accounts for making too many requests to the server in a given period of time. Is this something they implemented after your scrape? Or did you get around it somehow? Just curious
Monday, 4 May 2015
I believe it must be novel, yes.
Thursday, 7 July 2016
Cant you just use the username ?
Thursday, 7 July 2016
I'm sure you can.