How do I find a site tree?

March 2, 2006 1:11 PM

Is there any way to see all of the HTM or HTML files under one domain, to see a site map or tree?

For example say www.example.com has two pages: main.htm and links.htm, but doesnt provide any links. How would I find out these pages exsist

Sorry this is so vague, but its the best I could do.

posted by Blandanomics to computers & internet (6 comments total)

Not really, no.
posted by odinsdream at 1:13 PM on March 2, 2006

YOU COULD use a program like HTTRACK or other "off site" browsing program to spider a site.

Just set it up to ignore any file that isn't a .htm/.html, and set it up as "mirroring" the site to keep the directory structure.
posted by Sallysings at 1:19 PM on March 2, 2006

Enter site:www.example.com into Google.

Otherwise, there's no way to get a list of what files are on a server from a server.
posted by cillit bang at 1:20 PM on March 2, 2006

Yahoo site explorer
posted by camcgee at 1:24 PM on March 2, 2006

Also, if the pages exist but are not linked by anything (which is what it sounds like you are asking), well, short of hacking into the server and visually inspecting the directory structure you won't find these pages. Although if for example hidden.html existed and you typed in that url, it would be dutifully presented by the webserver, there are no spiders that will run through the infinite possibilites for page names just to see if it missed anything.
posted by caution live frogs at 2:07 PM on March 2, 2006

No, not at all.

What you can see are:

1) URIs that someone else (like google) knows about via spidering

2) URIs that you spidered yourself

3) URIs that you guess, and turn out to actually exist.

Also, keep in mind some sites have practically infinite pages, for example, the URL "http://ask.metafilter.com/tags/monkey" is valid, even if no posts have been tagged with the word "monkey". You can make up any tag, and a page exists for it.

Yeah, you can find out what tags have been used, but other sites have similar features that'll screw ya if you tried to generalize.
posted by delmoi at 3:27 PM on March 2, 2006