Comments on: Is there a programme to allow me to find redundant, non-linked files on my server?

Question: Is there a programme to allow me to find redundant, non-linked files on my server?

Hartster — Fri, 03 Mar 2006 01:57:24 -0800

Is there a programme, or way, to allow me to find 'redundant' files on my server: i.e. files that are not linked to from any other page?

Several of our larger websites have been running for 10+ years now. I'm sure there are hundreds of images and pdfs and video files and pages and entire directories that are hanging around on the server that are no longer used. Is there a programme that can spider a website, then spider the site directory on the server and then provide a list of files that are on the latter but not the former so that I can do a bit of housekeeping?

By: AmbroseChapel

AmbroseChapel — Fri, 03 Mar 2006 02:29:18 -0800

You can do it with Dreamweaver, it appears. But you need to have your website defined as a Site in Dreamweaver. You'd then use "Check Links" under the Sites menu.

Otherwise there are a ton of site management programs out there which should have this as a feature, and I think the missing piece of the puzzle is that you're not googling for the phrase "orphaned files"

By: twine42

twine42 — Fri, 03 Mar 2006 03:50:57 -0800

Can I suggest being very wary of anything that offers the ability to do this?

You need something that can not only trawl through the HTML, but which can also parse any javascript, cgi and php/asp/etc that may be on the servers as you can rarely be sure that all links are solely HTML.

Dreamweaver's abilities are severely limited in this respect.

You also need to be sure that no-one is crosslinking information from another site - internally of externally. Your tech guys may be able to give you a list of files that haven't been 'touched' (pretty sure that's the Unix term). This will give you a list of files that haven't been accessed. This doesn't tell you that it's not linked too, but if one of the files you want to delete isn't on this list, you need to find out why before you delete it.

By: twine42

twine42 — Fri, 03 Mar 2006 03:52:27 -0800

Damn...

[...] files that haven't been 'touched' in the last x months (pretty sure [...]

By: fvw

fvw — Fri, 03 Mar 2006 04:05:09 -0800

Actually, touched isn't what you're looking for. Touching sets the modification time, what you want is access time. If your filesystem stores access times (chances are it can), and it hasn't been disabled as a mount option (this is often done to enhance performance), and your web directories aren't mounted read-only, try find /path/to/web/directory -atime +61, which should list all files that haven't been read from in the last 61 days. You might want to spider your site first to make sure you don't hit files that nobody happens to have requested in the last two months, and of course you should move them out of the directory tree into a backup instead of just plain deleting& This all assumes you're using a unix, but if this is about the site mentioned in your profile, you are.

By: sfenders

sfenders — Fri, 03 Mar 2006 06:15:15 -0800

wget -m http://mirror.com/
cd html_root
find . | sort > ../root.files
cd mirror.com
find . | sort > ../mirror.files
cd ..
diff root.files mirror.files

But using atime is better, yeah.

By: yeahyeahyeahwhoo

yeahyeahyeahwhoo — Fri, 03 Mar 2006 06:17:51 -0800

nice fvw!

By: Mitheral

Mitheral — Fri, 03 Mar 2006 07:02:05 -0800

61 days is way to short, lots of activity is annual in nature. You don't want to blow away tips on surviving Burning Man or Halloween costume suggestions etc.

By: Hartster

Hartster — Fri, 03 Mar 2006 08:15:45 -0800

Perfect, thanks fvw. Works like a treat. The good thing about this method is (re: Mitheral's worries) that the searchbots ever more aggressive spidering will have accessed even the most unpopular pages, so I can be pretty sure what's produced by fvw's method is genuinely orphaned.

Thanks again AskMe!

By: yerfatma

yerfatma — Fri, 03 Mar 2006 13:43:01 -0800

Linkbot claimed to do this, but in practice it never seemed to be able to tie together spidering a site and FTPing into the root folder. That was years ago, so they may have improved it since then.

By: AmbroseChapel

AmbroseChapel — Sun, 05 Mar 2006 17:00:37 -0800

I feel like saying this exercise is not worth the trouble. If the pages which are "orphaned" aren't being accessed, they're costing you nothing in bandwidth and effectively nothing in storage.

And if somebody's got a bookmark or email link to them, or if some search engine has a record of them which might turn up, despite the fact that they're orphaned on your server, they're not really orphaned on the internet as a whole, and you risk a 404 for no good reason.