cobweb. Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.