/prog/scrape
Our power was out for eight hours this Thursday, so I spent most of the day being bored and drawing diagrams. One of those diagrams turned into this, which got me started on a project I’ve been meaning to get started on for a while now.
Of course, then I lost interest and it ended up just being this bit. The rest is just GUI twattery and crafting HTTP requests in ways that won’t trigger the auto-banners anyway.
What the thing I wrote (and which I guess I’ll call /prog/scrape) does now is open or create an SQLite database, pull subject.txt, compare subject.txt to the data in the database, load the thread pages for any thread that’s out of date, subject it to horrible and primitive pattern matching to pull the individual posts (and relevant metadata), and update the database.
The code is here (and should require no modules not present in a standard Python 2.5 install), but I strongly suggest you don’t try to run it. It will pull every single thing ever posted on /prog/, which takes something like two and a half hours.
Instead, download this (13.0 MB gzipped, 54.5 MB expanded), which is the SQLite database I built today. It’s up to date as of a few minutes ago, but to update it, just untar it into the same folder as the script (preserving the name prog.db), and run the script. It’ll find and use the existing database when determining what needs to be pulled.
Obviously this isn’t very useful by itself, but people interested in random statistics can have some fun with it.
To wit:
sqlite> select count(*) from posts; 168246 sqlite> select count(*) from (select distinct body from posts); 140893 sqlite> select count(*) from posts where author = 'Xarn'; 28
Requires SQLite 3. Have fun.
(The scraper should work with any Shiichan board by just changing the variables at the top, but honestly, who really uses Shiichan?)
Edit (July 11, 2009): Yes, this thread broke it. Updated version deals with it (by ignoring broken threads altogether).
Edit (August 12, 2009): http://github.com/Cairnarvon/progscrape/tree/master