What tool/program/script can I use to manipulate files based on a list of filenames?

June 25, 2009 7:39 PM

I have a text file containing a list of image filenames and I'd like to be able to point at a folder and have it select, move or label based on the list of filenames. Is there software, script or tool that makes automating this possible?

I read this question which is similar but couldn't get the solutions to work since my list is an exact filename rather than a keyword and I didn't know enough about the commands being used to adjust them.

I can also easily manipulate the list to be an exact list of paths if necessary.

As a stop-gap I am using Extensis Portfolio and tagging a custom field to move and manipulate. This limits me to one set of files rather than any file in a select folder and it also takes a very long time to catalog large numbers of files.

If there is a potential solution that has limitation please call them out. I may, from time to time, want this to work on 60k files but most days it will be only a few thousand. Your help is much appreciated.

posted by tinamonster to Computers & Internet (22 answers total)

You're going to have to be more specific about what you need. A little shell scripting will do most tasks. For example, rename a list of files:

cat list.txt | while read file;do mv $line prefix_$line;done

or use imagemagick to resize from the command line:

mkdir small; cat list.txt | while read file;do convert -resize 1024x800 $i small/$i;done

Of course, some of this depends on what operating system you have, and exactly what tasks you're trying to accomplish. Can you give more details?
posted by chrisamiller at 8:20 PM on June 25, 2009

What OS are you using?

If it ends in x, this is probably quite simple. If not, erm, dunno.
posted by pompomtom at 8:22 PM on June 25, 2009

Oops, looks like I edited out my OS details. It ends in X as in OSX.

To give an example, someone requests a CD with 3,000 files on it by sending me a list of those files as a tab delimited document. The list consists of the filename and extension.

I have several items with this filename on my drive, all in different sizes so universal searches for that filename are problematic. I want to point at the folder containing the files I want and have it corral them. If it does this by copying them to a new directory, great. If it selects them for me, also great. It can even label them orange in the finder, allowing me to sort them out from the others by label. I'd prefer it not move them but I can live with that if it's my only option.

Please let me know which additional details you will need. Thanks!
posted by tinamonster at 8:33 PM on June 25, 2009

I'd use the first example chrisamiller gives, with the small amendment:

cat list.txt | while read line;do mv "$line" ~/destination/;done

The quotes around the "$line" should allow for filenames with spaces, and this assumes there is such a place as ~/destination/. This also assumes that your list.txt holds filenames with paths which are relatively appropriate to where you're running the command from.
posted by pompomtom at 8:51 PM on June 25, 2009

on re-read, this all presumes a list.txt of one filename per line, without tabs between the file name and extension.
posted by pompomtom at 8:52 PM on June 25, 2009

So I have a sample folder, "04" on my desktop with files named TEST00001.jpg, TEST00002.jpg etc. I created a destination folder called "New" on the same desktop. I created a text file that lists ~/Desktop/04/TEST00001.jpg etc in each line and saved it on the desktop as tab delimited.

I modified the command to this: cat ~/Desktop/test.txt | while read line;do mv "$line" ~/Desktop/New;done

I ran it in terminal and nothing happened. What am I doing wrong? Thanks for your patience!
posted by tinamonster at 9:13 PM on June 25, 2009

this is what xargs is for!

xargs --arg-file=file -d '\t\r\n' mv -t dest

xargs

Takes a bunch of text and uses it as parameters to a specified program.

--arg-file

If you don't say otherwise xargs uses the input from stdin, but it can also read from a file. We could have piped the output of cat instead, like everyone else above is doing.

-d '\t\r\n'

Sets the delimiter to a tab. Each tab-separated item will be its own argument. that's the \t. the \r and \n are just in case there's newlines in there too.

mv -t dest

What to run on all those arguments. mv moves a file from one place to another. Normal use of mv is "mv src dest" but since all the xargs args go at the end, we are using the funny -t argument which tells mv which argument represents the target location (So they can be in whatever order).

posted by aubilenon at 9:13 PM on June 25, 2009

er, --arg-file=file is not as clear as it could be. Pretend I wrote
--arg-file=your_file_name_that_is_tab_delimited
instead.
posted by aubilenon at 9:15 PM on June 25, 2009

Okay actually use cp instead of mv since you don't want to make copies instead of moving them. (I just jumped on everyone else's mv bandwagon. Sorry! I hope I didn't make you ruin everything!)
posted by aubilenon at 9:24 PM on June 25, 2009

Ok, using the description of file locations from my comment above I tried this:
xargs ~/Desktop/test.txt -d '\t\r\n' cp -t ~/Desktop/New/
And I don't get an error but it also hasn't moved the files in the 10 minutes it has been running. Should this eventually move them or did I make a mistake?
posted by tinamonster at 9:37 PM on June 25, 2009

(not disputing aubilenon's method - this is what xargs is for, but I've got used to the "while read" method)

Given what you've said, you shouldn't save that text file as "tab delimited", but rather a simple txt file, one filename per line. That said, I suspect that's what it actually is anyway (cos you've only one item per line) - so ignore this bit....

So before fiddling with that try:

cat ~/Desktop/test.txt | while read line;do cp "$line" ~/Desktop/New/;done



(the only difference is the trailing slash on '~/Desktop/New/'. Without that slash, you'll be moving each file into a file on the desktop called 'New'. With the slash, it creates a file within the directory '~/Desktop/New/'. I suspect when you executed the command you rewrote the ~/Desktop/New file a lot of times...)
posted by pompomtom at 9:45 PM on June 25, 2009

aubilenon, Mac OS X is a BSD-derived UNIX, so it has the BSD-style xargs rather then the GNU version, and the options you're using aren't available. The command should be

cat ~/Desktop/test.txt | xargs -J % cp % ~/Desktop/New/

posted by nicwolff at 10:24 PM on June 25, 2009

pompomtom: It was tab delimited text saved out of Excel. After saving it as a Unicode 16 text file something happened! Unfortunately it wasn't copying. I get the following errors:

cp: ??/: No such file or directory
cp: fts_open: No such file or directory
cp: fts_open: No such file or directory

Is that a problem with my text file?
posted by tinamonster at 10:27 PM on June 25, 2009

pompomtom, as long as the "~/Desktop/New" directory already exists, cp file ~/Desktop/New does the same thing as cp file ~/Desktop/New/. Anyway, this really is what xargs is for.
posted by nicwolff at 10:28 PM on June 25, 2009

Not sure the system handles UTF-16 encoded Unicode, tina, try UTF-8 (or ASCII, if there are no non-ASCII characters in the file names).
posted by nicwolff at 10:31 PM on June 25, 2009

And please tell me Excel doesn't call UTF-16 "Unicode 16".
posted by nicwolff at 10:33 PM on June 25, 2009

Oh, and another point: if any of the filenames have spaces in them, then to use xargs you have to replace the record separators in your file with the \0 character and add the -0 flag to xargs. Assuming there's one filename on each line of your file, try:

cat ~/Desktop/test.txt | tr '\n' '\0' | xargs -0 -J % cp % ~/Desktop/New/

posted by nicwolff at 10:49 PM on June 25, 2009

Nicwolff: That's my special petname for it. Excel calls it Unicode (UTF-16).

Success! Nicwolff's suggestion moved the files.

In case anyone else ever comes across this thread and has similar problems here's what I did. I pasted the filenames from Excel into Word. Then I saved my Word file as Plain text (.txt). Based on what Nicwolff said I used Other encoding options: Western (ASCII). I ticked Insert Line Breaks and ended lines with CR/LF.

I did get errors (cp: \r: No such file or directory) but I blame my txt file for that and the files moved. This begs the question: What program do you brilliant posters use to create a simple text file?
posted by tinamonster at 10:59 PM on June 25, 2009

Wow, thanks so much to everyone who posted here. I think I learned more tonight from you all than in the many many hours I spent reading tons of "Intro to scripting" sites. I appreciate all of the explaining.
posted by tinamonster at 11:19 PM on June 25, 2009

Well, tina, in this case once you have copied the list from Excel you could skip making a file and just use the pbpaste command to feed the clipboard right to xargs:

pbpaste | tr '\n' '\0' | xargs -0 echo

But if there were tens of thousands of records I would try to save that sheet as tab-delimited ASCII or UTF-8 right from Excel.

The reason you got the warnings about \r is that Mac OS X text files use the UNIX line break, which is just an LF. CR/LF is the Windows line break, which Excel "helpfully" offered you, and the \r represents the CRs that Excel added.
posted by nicwolff at 11:20 PM on June 25, 2009

Happy to help, tina! xargs is very useful, and the UNIX tools that underlie OS X give it a lot of hidden power for manipulating text and files. Once you know their names and which one does what, you can always "man xargs" or whichever to research each command's options.
posted by nicwolff at 11:31 PM on June 25, 2009

Another neat thing you can do is use the ln (link) command instead of cp (copy); argument syntax is the same. This will not only be much faster, but it will avoid filling up your disk with redundant copies of your data.

By default, ln makes hard links. Provided the files you're marshalling and the directory you're marshalling them into are in the same filesystem (which is usually the same thing as saying they're on the same physical disk partition) then this is what you want to do. If the files could be anywhere, then add the -s option to ln and it will make symbolic links instead.

Hard links are just extra directory entries for an existing file. They work just like the file's original directory entry. By making a hard link to a file, you've effectively given the file an extra name. If a file has extra names created by hard linking, those names are all as "real" as the name the file had when first created, and you have to delete all of them to make the file actually disappear from the disk; there's no way to "break" a hard link short of filesystem corruption.

Symbolic links are links to the linked file's name, rather than its contents. That allows you to make symbolic links to files in any filesystem - but if the file you've symlinked to is renamed or moved, all those symlinks will break. Also, you might also find that archiving or CD burning software will archive or burn the links rather than the underlying files unless you tell it explicitly to follow links, so if you're using symlinks, make sure you know how your burner deals with them.
posted by flabdablet at 7:03 AM on June 26, 2009