Spidey. A multi threaded web crawler library that is generic enough to allow different engines to be swapped in.