Comments on: How do I find a site tree?

Question: How do I find a site tree?

Blandanomics — Thu, 02 Mar 2006 13:11:52 -0800

Is there any way to see all of the HTM or HTML files under one domain, to see a site map or tree?

For example say www.example.com has two pages: main.htm and links.htm, but doesnt provide any links. How would I find out these pages exsist

Sorry this is so vague, but its the best I could do.

By: odinsdream

odinsdream — Thu, 02 Mar 2006 13:13:06 -0800

Not really, no.

By: Sallysings

Sallysings — Thu, 02 Mar 2006 13:19:17 -0800

YOU COULD use a program like HTTRACK or other "off site" browsing program to spider a site.

Just set it up to ignore any file that isn't a .htm/.html, and set it up as "mirroring" the site to keep the directory structure.

By: cillit bang

cillit bang — Thu, 02 Mar 2006 13:20:35 -0800

Enter site:www.example.com into Google.

Otherwise, there's no way to get a list of what files are on a server from a server.

By: camcgee

camcgee — Thu, 02 Mar 2006 13:24:39 -0800

Yahoo site explorer

By: caution live frogs

caution live frogs — Thu, 02 Mar 2006 14:07:02 -0800

Also, if the pages exist but are not linked by anything (which is what it sounds like you are asking), well, short of hacking into the server and visually inspecting the directory structure you won't find these pages. Although if for example hidden.html existed and you typed in that url, it would be dutifully presented by the webserver, there are no spiders that will run through the infinite possibilites for page names just to see if it missed anything.

By: delmoi

delmoi — Thu, 02 Mar 2006 15:27:20 -0800

No, not at all.

What you can see are:

1) URIs that someone else (like google) knows about via spidering

2) URIs that you spidered yourself

3) URIs that you guess, and turn out to actually exist.

Also, keep in mind some sites have practically infinite pages, for example, the URL "http://ask.metafilter.com/tags/monkey" is valid, even if no posts have been tagged with the word "monkey". You can make up any tag, and a page exists for it.

Yeah, you can find out what tags have been used, but other sites have similar features that'll screw ya if you tried to generalize.