How to go from "this_" to "this" ?

March 23, 2006 8:59 AM

Batch unzipping and file renaming on UNIX or OS X? All_ my_ files_ look_ like_ this!_.gz

I recently lost all files on one of my domains, but was able to recover everything from my server's backup. However, all the files now have an underscore appended to the end of the filename (index.html is now index.html_, for example). All the files and all the folders are like this.

And some of them are gzipped, so some files are now filename.cgi_.gz.

What's the quickest, easiest way to unzip everything and remove the _ from the end of the files, including those within nested folders? Is there a shell command I can use? (or a way to do this using Automator on OS X if I copy everything from my server to my computer?)

posted by Robot Johnny to Computers & Internet (25 answers total)

In Windows, that _ extension can mean the file is compressed. Are they the undamaged files, you just need to rename them?
posted by Malor at 9:01 AM on March 23, 2006

Response by poster: Yeah, they just need renaming... they are otherwise exact copies.
posted by Robot Johnny at 9:04 AM on March 23, 2006

"gunzip *gz" will ungzip everything.

"rename 's/_$//' *" will remove the _ from all of the files. rename may not be on OSX, it's a fairly standard part of Linux at least.
posted by zsazsa at 9:09 AM on March 23, 2006

Response by poster: Thanks zsazsa, but will that remove only the _ on the end of the filename? Or every occurance of _? (Some of my files have a _ elsewhere in the filename that I obviously want to keep)

Also, will these commands apply to files within nested folders? Or will I have to go into each folder and repeat the command?
posted by Robot Johnny at 9:13 AM on March 23, 2006

If you have menu scripts turned on there is a script
Finder Scripts > Replace Text in Item Names

The script opens a little dialog box that allows you to replace '_' with ' '.

If you don't have menu scripts on you should, there are lots of useful actions available. Spotlight 'Applescript Utility' Open it and click the check boxes 'Show Script Menu in menu bar' and 'Show Library scripts'

Like your sites, glad to pay you back for the entertainment.
posted by leftoverboy at 9:16 AM on March 23, 2006

Doh! Just say preview. To only grab the end under score find and replace '_." with '.' and you will get only the ones at the end,
posted by leftoverboy at 9:17 AM on March 23, 2006

Best answer: Oops. I didn't see the recursive argument.

"gunzip -r ." for the ungzipping...

"find -exec rename 's/_$//' {} \;" should do it.

Yes, that will only remove a _ on the end of a filename. A '$' in a regular expression means "end of a line/string", so it will only match underscores on the ends of filenames.
posted by zsazsa at 9:18 AM on March 23, 2006

(By recursive, I mean it'll work in nested folders.)
posted by zsazsa at 9:19 AM on March 23, 2006

(If rename doesn't come with macos, you can put this script in the dir you want and do:

"find . -exec perl rename.perl 's/_$//' {} \;"

Or if you'd rather not hassle with this Unix arcanery, leftoverboy's way looks fine.

(I added a '.' to the find command because some finds are different and need it, and I don't have access to a Mac right now to see if that's the case. It doesn't hurt to use it.)
posted by zsazsa at 9:31 AM on March 23, 2006

Best answer: If all else fails, here is a short batch script that should work. MAKE SURE you run this ONLY on a backup of the files. I have only tested this with Linux. It will recurse directories and seems fine, but I have not tested this extensively.


!/bin/sh
# first, set the word break to be newline only.
# This is easier for dealing with filenames.
IFS=$'\n'

# Do a search for all _ files, and then pipe that result
# through sed...stripping away the _

for i in `find . -name "*_" | sed -e s/\_$//`; do
mv $i\_ $i
done

posted by Malor at 9:35 AM on March 23, 2006

There's no "rename" in OS X. Try

  find / -name '*_' -print0 | perl -n0e '$old = $_; chop; chop; rename $old, $_;'
 
posted by nicwolff at 9:36 AM on March 23, 2006

Definitely use -print0 by the way, file names on the Mac often contain spaces.
posted by nicwolff at 9:37 AM on March 23, 2006

A bit more documentation... that recurses through your directories and generates a list of files ending with _. It strips the _ by piping the result through sed.

Then, for each line, it does:

mv filename_ filename

The perl renamer may be fine too... but I don't know if it will handle spaces properly. My shell script will... that's what the IFS command does.
posted by Malor at 9:40 AM on March 23, 2006

Response by poster: Thanks everyone.

zsazsa, the unzipping worked, but like others stated, "rename" doesn't work

nicwolff, I tried using your method, but it started to search my entire computer so I had to terminate it... how do I limit it to my current directory (and its recursive directories?)

Malor, if I use your method, how do I run a batch script? (I'm new to fiddling around in OS X's terminal)

Also, I'm not limited to OS X's Terminal... the files also exist on my Dreamhost server, so any commands that work on their Unix servers are welcome, too.
posted by Robot Johnny at 10:01 AM on March 23, 2006

Best answer: Oh, sorry, you said "everything" so I figured you wanted to do the whole filesystem. Just change the "/" to a "." to do only the current directory and its subdirectories:

  find . -name '*_' -print0 | perl -n0e '$old = $_; chop; chop; rename $old, $_;'
 
posted by nicwolff at 10:06 AM on March 23, 2006

And that'll work fine on any UNIX that has Perl installed.
posted by nicwolff at 10:07 AM on March 23, 2006

Response by poster: Thanks nicwolff!
Sorry to keep being a nitpicker, but that totally worked for the main directory, but did nothing to the files in its subdirectories.
posted by Robot Johnny at 10:13 AM on March 23, 2006

Robot johnny: use Text Editor to create a new file, fixfiles.sh, in your home directory. (that's the easy place to find it.) Copy that script in, verbatim. Save it.

Then open a terminal. cd to the directory you stored your backups in. (make sure it's not the only copy you have!). Run the script with ~/fixfiles.sh. It searches from the current directory down for _ files.

If for some weird reason you plan to run it a lot, let me know and I'll tell you how to install it semi-permanently.
posted by Malor at 10:18 AM on March 23, 2006

Crap! Run the script with:

sh ~/fixfiles.sh
posted by Malor at 10:19 AM on March 23, 2006

Something else is wrong then; that should work. Does

  find . -name '*_' -print

give you a list of the files in the subdirectories?
posted by nicwolff at 10:26 AM on March 23, 2006

Response by poster: Thanks, Malor! Worked like a charm.

A slight problem though: as the script renamed subdirectories ending in _, it spat out error messages that files within those directories couldn't be found (I assume because the name had been changed in mid-script).

So I just repeated the script until there were no more error messages.

Everyone here helped in some way, so time to hand out the "best answers". Thanks so much!

On preview: Yes, nicwolff, just checked on a backup copy of the offending files and that command does list all the files. Don't know why the renaming didn't work. Thanks for your help!
posted by Robot Johnny at 10:31 AM on March 23, 2006

Response by poster: Oh, and thanks leftoverboy, too, but your method wouldn't work because the _ came after the extension (ie hello.jpg_)
posted by Robot Johnny at 10:34 AM on March 23, 2006

Robot, I ALMOST posted back with that explicit instruction... I realized that might happen. But I didn't want to make it sound overcomplex, and I was having trouble explaining it simply. I'm glad you figured it out. :)
posted by Malor at 10:48 AM on March 23, 2006

Best answer: Um, you might want to consider quoting the filename variables in those examples - if the files have spaces in them, then you're going to get into IFS problems. (I learned this the hard way). For example, the

mv $i\_ $i

in Malor's example should be

mv "$i\_" "$i"
posted by aberrant at 11:56 AM on March 23, 2006

aberrant, it worked perfectly in my testing with filenames with spaces.

By setting IFS to newline only, and having newlines embedded in the filename, it seems to work without needing the quotes.
posted by Malor at 1:34 PM on March 23, 2006