Looking for software to crawl/spider websites to find hubs, core websites on topics.

July 6, 2007 5:14 PM

Looking for software to crawl/spider a term in google to show me main websites for a topic. And if possible to generate graphs like this.

I want to find main websites, hubs for key topics, and display a graph like gnutellas maps. graph map

I dont know the correct terms to search for software that will crawl websites and make me a link map with hub/nodes.

posted by IronWolve to Computers & Internet (5 answers total)

This site does it with a Java applet called htmlgraph.jar

You might ask the author for his code, or for use of htmlgraph.jar for use in your own Web pages.
posted by paulsc at 5:59 PM on July 6, 2007

http://www.aharef.info/2006/05/websites_as_graphs.htm

looks like a possibility.
posted by rhizome at 6:35 PM on July 6, 2007

TagURLs might be something like what you're looking for - a list of the top domains by topic.
posted by lubujackson at 7:03 PM on July 6, 2007

Response by poster: Was hoping for something like
http://datamining.typepad.com/data_mining/2006/02/attenex.html
or
http://datamining.typepad.com/data_mining/2005/10/visualizing_the.html#
posted by IronWolve at 7:37 PM on July 6, 2007

Response by poster: http://www.visualcomplexity.com/vc/ shows great photos, gonna check those now.
posted by IronWolve at 7:42 PM on July 6, 2007