second step - spider

Aug 4, 2003

I wrote a spider (a.k.a. "crawler", "scraper", "robot") to browse the Friendster site, recording the following information for each person:

  1. id (e.g. 867412)

  2. nickname (e.g. "Ben")

  3. address (they call it "Location")

  4. type (valid or suspended)

  5. friends (array of friend IDs)

My spider program starts with a given node, and walks outward through the network (breadth-first) for as long as you tell it.  It writes its output to an ad-hoc text file.

On the day that i first run it, i have around 14 friends and 220,000 people in my "Personal Network".  I run the spider until i have around 1000 of the people (friendsters) closest to me.

The spider is then tweaked to write output that can be used for graphing the friend network....

Geek info:

In case you want to try out the spider yourself:

This is my second release of the spider, on Nov 13, 2003.