Aug 4, 2003
I wrote a spider (a.k.a. "crawler", "scraper", "robot") to browse the Friendster site, recording the following information for each person:
id (e.g. 867412)
nickname (e.g. "Ben")
address (they call it "Location")
type (valid or suspended)
friends (array of friend IDs)
My spider program starts with a given node, and walks outward through the network (breadth-first) for as long as you tell it. It writes its output to an ad-hoc text file.
On the day that i first run it, i have around 14 friends and 220,000 people in my "Personal Network". I run the spider until i have around 1000 of the people (friendsters) closest to me.
The spider is then tweaked to write output that can be used for graphing the friend network....
In case you want to try out the spider yourself:
- frspider.zip (24k) has the source, windows executable, and readme file
- frspider-DLLs.zip (483k) has some necessary DLLs, in case you don't already have them on your system.
This is my second release of the spider, on Nov 13, 2003.