How do I find a site tree?
March 2, 2006 1:11 PM Subscribe
Is there any way to see all of the HTM or HTML files under one domain, to see a site map or tree?
For example say www.example.com has two pages: main.htm and links.htm, but doesnt provide any links. How would I find out these pages exsist
Sorry this is so vague, but its the best I could do.
For example say www.example.com has two pages: main.htm and links.htm, but doesnt provide any links. How would I find out these pages exsist
Sorry this is so vague, but its the best I could do.
Enter site:www.example.com into Google.
Otherwise, there's no way to get a list of what files are on a server from a server.
posted by cillit bang at 1:20 PM on March 2, 2006
Otherwise, there's no way to get a list of what files are on a server from a server.
posted by cillit bang at 1:20 PM on March 2, 2006
Also, if the pages exist but are not linked by anything (which is what it sounds like you are asking), well, short of hacking into the server and visually inspecting the directory structure you won't find these pages. Although if for example hidden.html existed and you typed in that url, it would be dutifully presented by the webserver, there are no spiders that will run through the infinite possibilites for page names just to see if it missed anything.
posted by caution live frogs at 2:07 PM on March 2, 2006
posted by caution live frogs at 2:07 PM on March 2, 2006
No, not at all.
What you can see are:
1) URIs that someone else (like google) knows about via spidering
2) URIs that you spidered yourself
3) URIs that you guess, and turn out to actually exist.
Also, keep in mind some sites have practically infinite pages, for example, the URL "http://ask.metafilter.com/tags/monkey" is valid, even if no posts have been tagged with the word "monkey". You can make up any tag, and a page exists for it.
Yeah, you can find out what tags have been used, but other sites have similar features that'll screw ya if you tried to generalize.
posted by delmoi at 3:27 PM on March 2, 2006
What you can see are:
1) URIs that someone else (like google) knows about via spidering
2) URIs that you spidered yourself
3) URIs that you guess, and turn out to actually exist.
Also, keep in mind some sites have practically infinite pages, for example, the URL "http://ask.metafilter.com/tags/monkey" is valid, even if no posts have been tagged with the word "monkey". You can make up any tag, and a page exists for it.
Yeah, you can find out what tags have been used, but other sites have similar features that'll screw ya if you tried to generalize.
posted by delmoi at 3:27 PM on March 2, 2006
This thread is closed to new comments.
Just set it up to ignore any file that isn't a .htm/.html, and set it up as "mirroring" the site to keep the directory structure.
posted by Sallysings at 1:19 PM on March 2, 2006