Comments on: I need a script to compare files in a directory to files referenced in web pages.

Question: I need a script to compare files in a directory to files referenced in web pages.

pissfactory — Mon, 13 Sep 2004 15:37:38 -0800

Can anyone recommend a program (in say perl or java or something) that will execute on my linux web server and a) slurp the file references in the pages (html, php) and b) compare them to the actual files in my web root tree giving me a list of all of the unreferenced files (not referenced on public web pages)? I want to clean up this junky file system, but I don't want to break any links and I inherited the mess. Thanks.

By: sfenders

sfenders — Mon, 13 Sep 2004 15:59:16 -0800

for i in `find files` ; do if ( ! grep -q $i *.html ) then echo $i ; fi ; done

...or you could use wget, then find and diff, which would be a little more reliable.

By: weston

weston — Mon, 13 Sep 2004 16:50:35 -0800

pissfactory: I wrote just such a thing years ago. Code here, and brief (almost non) manual here. No guarantees that it runs, works, does anything productive, or doesn't attract space aliens to your house.

By: weston

weston — Mon, 13 Sep 2004 17:29:39 -0800

And is sfenders some kind of shell ninja or what?

By: nicwolff

nicwolff — Mon, 13 Sep 2004 18:17:21 -0800

linklint -orphan

By: fvw

fvw — Tue, 14 Sep 2004 08:34:42 -0800

You could also just wget -r -l 0 your website. After that, all files with an access time of before you started wgetting are orphaned and can be moved or deleted. (Use find -amin -10, or ls **/*(.am-10) in the ever wonderful zsh)

sfenders' snippet will break should you have filenames with spaces in them. You can reasonably easily convert this to something^* that only breaks with filenames containing newlines, but even that can happen. All in all the only safe file-name separator is a \0, which are harder to work with in shells, sadly.

find . | while read -r i; do if ( ! grep -r $i * ) then echo $i ; fi ; done (this also works if your html files aren't all in the current directory)