Cum procesam liste de bloguri

Pentru ca aparent se cere, hai sa vedem cum putem folosi computerele pentru a da singuri in citeva ore rezultate pe care toata hoarda de "blogari" romani nu-s capabili sa le produca in luni.

Discutia de-aici se bazeaza pe presupozitia ca folositi linux. Windows nu este un sistem de operare, este o inchisoare pentru cei cu dizabilitati intelectuale. Deci, daca rulati linux foarte bine, si daca nu vedeti acolo in articol ce pasi aveti de urmat pentru a va aduce calculatorul personal in stare de functionare.

Lista pe care o vom procesa se gaseste, cum am mai zis, in a treia nota a articolului Extractie din wordpress.com. Salvati respectivul fisier .zip undeva si extrageti-l dintru-ndesine afara la lumina.

Prima chestie peste care dam este xargs. Xargs exista pentru ca linux nu poate executa comenzi oricit de lungi, si-atunci cind avem comenzi lungi de executat le trecem prin xargs. In cazul nostru concret :

$xargs curl -I < wp.txt

Chestia asta cere sistemului sa ia la rind tot ce-avem in wp.txt si sa execute curl -I pe, care curl -I inseamna ca dorim sa vedem doar headerele. Si intr-adevar, rezultatul este

HTTP/1.1 200 OK
Server: nginx
Date: Sat, 07 Apr 2012 20:57:13 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding
Vary: Cookie
X-hacker: If you're reading this, you should visit automattic.com/jobs and apply to join the fun, mention this header.
X-Pingback: http://whoisrandylim.wordpress.com/xmlrpc.php
Link: ; rel=shortlink

HTTP/1.1 200 OK
Server: nginx
Date: Sat, 07 Apr 2012 20:57:14 GMT
Content-Type: text/html; charset=UTF-8
Connection: close
Vary: Accept-Encoding
Vary: Cookie
X-hacker: If you're reading this, you should visit automattic.com/jobs and apply to join the fun, mention this header.
X-Pingback: http://joyreturns.com/xmlrpc.php
Link: ; rel=shortlink

HTTP/1.1 200 OK
Server: nginx
Date: Sat, 07 Apr 2012 20:57:16 GMT
Content-Type: text/html; charset=UTF-8
Connection: close
Vary: Accept-Encoding
Vary: Cookie
X-hacker: If you're reading this, you should visit automattic.com/jobs and apply to join the fun, mention this header.
X-Pingback: http://avidvisions.wordpress.com/xmlrpc.php
Link: ; rel=shortlink
Last-Modified: Sat, 07 Apr 2012 20:57:16 +0000
Cache-Control: max-age=300, must-revalidate
X-nananana: Batcache

si asa mai departe.

Ce facem cu astea ? Pai, de exemplu, ar fi o idee ca-n loc de headere sa vedem titlurile ultimelor articole.

$xargs curl < wp.txt | grep 'h1 class="entry-title"'

Chestia asta cere tot idem procesarea completa a listei de url-uri din wp.txt, da' de catre curl pur si simplu (deci sa ni se dea continutul paginilor), iar rezultatul este stors prin grep, care nu publica decit acele rinduri care contin ce i-am cerut, adica h1 class="entry-title". Rezultatele sunt cam ce v-ati si astepta,

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 84505 0 84505 0 0 46892 0 --:--:-- 0:00:01 --:--:-- 54100
100 72502 0 72502 0 0 47225 0 --:--:-- 0:00:01 --:--:-- 145k
100 8275 0 8275 0 0 5555 0 --:--:-- 0:00:01 --:--:-- 34336 <h1 class="entry-title"><a href="http://avidvisions.wordpress.com/2012/03/04/destination-glen-onoko-falls/" title="Permalink to Destination: Glen Onoko Falls" rel="bookmark">Destination: Glen Onoko Falls</a></h1>
<h1 class="entry-title"><a href="http://avidvisions.wordpress.com/2012/02/25/experiences-cornell-hockey/" title="Permalink to Experiences: Cornell Hockey" rel="bookmark">Experiences: Cornell Hockey</a></h1>
<h1 class="entry-title"><a href="http://avidvisions.wordpress.com/2012/02/17/jumping-spider-macro-tips/" title="Permalink to Jumping Spider Macro Tips" rel="bookmark">Jumping Spider Macro Tips</a></h1>
100 54309 0 54309 0 0 30062 0 --:--:-- 0:00:01 --:--:-- 97327
100 35612 0 35612 0 0 29758 0 --:--:-- 0:00:01 --:--:-- 110k
102 1025 0 1025 0 0 878 0 --:--:-- 0:00:01 --:--:-- 878 <h1 class="entry-title"><a href="http://bullimiaddict.wordpress.com/2012/04/06/good-friday-lest-we-forget-why-its-called-that/" title="Permalink to Good Friday, Lest We Forget Why It’s Called That" rel="bookmark">Good Friday, Lest We Forget Why It’s Called That</a></h1>
<h1 class="entry-title"><a href="http://bullimiaddict.wordpress.com/2012/04/06/home-is-where-you-hang-your-hat/" title="Permalink to Home is Where You Hang Your Hat…" rel="bookmark">Home is Where You Hang Your Hat…</a></h1>
<h1 class="entry-title"><a href="http://bullimiaddict.wordpress.com/2012/04/06/ever-feel-this-way-maybe-not-exactly-the-same-story-but/" title="Permalink to Ever Feel This way? Maybe not exactly the same story, but???" rel="bookmark">Ever Feel This way? Maybe not exactly the same story, but???</a></h1>
100 21289 0 21289 0 0 14283 0 --:--:-- 0:00:01 --:--:-- 62543

De-aici incolo fiecare se distreaza cum pofteste. Eu mai mentionez doar ca adaugind un > output.txt la final veti obtine toate rezultatele intr-un fisier, in loc sa vi le scuipe pe ecran, si deci puteti lasa calculatorul sa munceasca in timp ce dumneavoastra faceti altceva, ca de exemplu tineti o pizda-n brate, ca munca-i pentru calculatoare si tractoare, RETINETI AMANUNTUL ca-i important.

De asemenea utile sunt manualele pentru curl, grep, awk si evident xargs, precum va pot prinde bine si niste minime notiuni de regex.

Morala intregii povesti ar fi ca de cind cu descoperirea calculatorului oamenii inteligenti au un avantaj. Iar cei care n-au un avantaj nu-s inteligenti.

Succese!