Can you recommend software that will generate an extensive site map of my company's intranet?

May 31, 2006 1:40 PM

I'm looking for a piece of software that will spider the company's intranet. After looking around for a few days, I've generated a list of requirements.

1. Needs to be installed on my desktop machine, as the intranet can sense whether the appropriate user is logged on to the network behind the corporate firewall, and I need access to all areas of the site.

2. I need to be able to specify what kinds of URLs it should ignore (like dynamic ones). The site is built in Coldfusion if that matters. Also, I'd like the ability to exclude certain filetypes (like images) and directories.

3. I'd like it to output in a pretty graphical format if possible. If it doesn't I will have to create a Visio graphic so management can comprehend it.

4. Here's the clincher: I want it to be able to pull the "last modified" date from the file itself and display it with the filename. One of the reasons we're doing this audit is to ferret out stale content.

5. Super mega bonus points for anyone who can figure out how to somehow combine the data contained in our WebTrends database with the sitemap, so not only can we see how the files link to each other, how old they are, but also see how heavily trafficked they are.

Thanks to everyone in advance!

posted by jennaba to computers & internet (9 comments total)

Can't do that last one, but all the others apart from the graphics are pretty easy in Perl or any similar language. I don't know of a single Perl module which will just do it, but I know where all the bits are to put one together.

Dreamweaver does nice-looking site maps now I think about it, you might check that out? I don't know about last-mod dates.
posted by AmbroseChapel at 2:13 PM on May 31, 2006

AmbroseChapel: I've tried Dreamweaver, but it has a real problem with the Coldfusion code in the pages. Every time I ran the sitemap tool it choked.

Don't blame me, I didn't write these pages, I'm just in charge of managing them now :-)
posted by jennaba at 2:17 PM on May 31, 2006

Try Xenu's Link Sleuth. It's free. We use it to check for broken links on our web sites, but it will also create a list of pages with their last updated dates, and can also do an outline-style site map. And did I say it's free? Sorry, I have no idea how to make it play nice with WebTrends. I'm still struggling with understanding the WebTrends data we have.
posted by Joleta at 2:24 PM on May 31, 2006

I have used Site Orbiter for this purpose.
-Coeval's wife
posted by coevals at 2:26 PM on May 31, 2006

I've tried Dreamweaver, but it has a real problem with the Coldfusion code in the pages.

Ah, yes, not surprised. It only maps off a local copy, as far as I can see.
posted by AmbroseChapel at 7:20 PM on May 31, 2006

Sigh... "spider" is now a verb.....
posted by longsleeves at 7:56 PM on May 31, 2006

If by "now" you mean "for the last ten years or so" yes.
posted by AmbroseChapel at 5:05 AM on June 1, 2006

Coeval's Wife: Sorry, I should have specified that I'm looking for something on the Windows platform. Thanks for the suggestion, though!
posted by jennaba at 10:50 AM on June 1, 2006

Joleta: I tried Xenu's Link Sleuth as you suggested, but for some reason it doesn't want to deal with the login system on the site. I checked the documentation and changed the INI file to allow cookies and checked the box to request passwords.

Does anyone else have ideas?
posted by jennaba at 11:17 AM on June 1, 2006