Comments on: Looking for software to crawl/spider websites to find hubs, core websites on topics.

Question: Looking for software to crawl/spider websites to find hubs, core websites on topics.

IronWolve — Fri, 06 Jul 2007 17:14:04 -0800

Looking for software to crawl/spider a term in google to show me main websites for a topic. And if possible to generate graphs like this.

I want to find main websites, hubs for key topics, and display a graph like gnutellas maps. graph map

I dont know the correct terms to search for software that will crawl websites and make me a link map with hub/nodes.

By: paulsc

paulsc — Fri, 06 Jul 2007 17:59:51 -0800

This site does it with a Java applet called htmlgraph.jar

You might ask the author for his code, or for use of htmlgraph.jar for use in your own Web pages.

By: rhizome

rhizome — Fri, 06 Jul 2007 18:35:40 -0800

http://www.aharef.info/2006/05/websites_as_graphs.htm

looks like a possibility.

By: lubujackson

lubujackson — Fri, 06 Jul 2007 19:03:50 -0800

TagURLs might be something like what you're looking for - a list of the top domains by topic.

By: IronWolve

IronWolve — Fri, 06 Jul 2007 19:37:18 -0800

Was hoping for something like
http://datamining.typepad.com/data_mining/2006/02/attenex.html
or
http://datamining.typepad.com/data_mining/2005/10/visualizing_the.html#

By: IronWolve

IronWolve — Fri, 06 Jul 2007 19:42:52 -0800

http://www.visualcomplexity.com/vc/ shows great photos, gonna check those now.