Ask MetaFilter questions tagged with bash

Bash scripting problem

sbutler — Tue, 11 Aug 2009 16:29:56 -0800

Please hope me with a bash scripting problem. Involves: command substitution and word splitting. In my script I'm trying to wrap the curl command to give it a global retry value. So I have this.

When I call it like this, things seem to work:

retry_curl "http://example.org/file1.txt"

But I have one place where I want to call it like this:

LOCAL_CURLOPTS="--time-cond \"Tue, 11 Aug 2009 18:12:12 -0500\""

retry_curl $LOCAL_CURLOPTS "http://example.org/file1.txt

And what I get on the console is:

curl: option -0500: is unknown

curl: try 'curl --help' or 'curl --manual' for more information

So somewhere bash is breaking apart my LOCAL_CURLOPTS as words, and then using that in the command substitution. What I can't figure out is how to get it to stop.

Shell script handling CSV columns

fightoplankton — Wed, 10 Jun 2009 16:30:20 -0800

I'm using a shell script to handle some excel docs by converting to csv and reformatting it a little bit. I'm having some difficulty going on with the next part of the script. Some help would be great. :) CURRENT SCRIPT:
#!/bin/sh
ls *.xls
echo "Enter filename for excel input file: "
read filen
xls2csv -x "$filen"".xls" -b WINDOWS-1252 -c "$filen"".csv" -a UTF-8
sed '1,3d' "$filen".csv >"$filen"_temp.csv
cut -d "," -f 1,4- "$filen"_temp.csv > "$filen"_temp2.csv
awk -F, '{sub($NF, "");print}' "$filen"_temp2.csv > "$filen"_temp3.csv
sed 's/.$//' "$filen"_temp3.csv > "$filen"_fixed.csv
rm "$filen".csv
rm "$filen"_temp.csv
rm "$filen"_temp2.csv
rm "$filen"_temp3.csv
echo "$filen"".xls corrected and saved as ""$filen""_fixed.csv"

EXAMPLE OUTPUT (first three lines only):
"Device ID","1) S31 Which best describes how you answered the online reading comprehension quiz?","2) S32 Which best describes how you answered the online timed retrieval quiz?","3) B19. If you want your product to be easy to find in the supermarket then you should make its container","4) C19. So that he can shift attention between the radio and his incessantly talking girl friend when she is in the car, Joe adjusts his radio","5) B20. Early selection is most likely to occur for","6) C20. Early selection for a red target is most likely to occur when there is","7) B21. In a lexical decision task, when the target is a bird name, e.g. robin, it is usually preceded by the prime BODY but is sometimes preceded by the prime BIRD.","8) C21. In a lexical decision task, when the target is a dog name, e.g. collie, it is usually preceded by the prime CAR but is sometimes preceded by the prime DOG.","9) B23. Suppose that that you see a brief display with 12 colored letters: 4 red, 4 white, and 4 blue. At the offset of the display you hear tone. A tone instructs you to report only the letters of a particular color: high for red, medium for white, and low for blue. About how many letters do you report?","10) B22. Sperling (1960) found that partial report produced the highest estimate of the number of available letters when the tone occurred ","11) C23. According to the logic of Sperling’s (1960) partial report method, an observer who reports three letters from a row in a 4 x 4 display that was cued at the display’s offset must have seen at least","12) C22. Sperling (1960) found that the greatest difference between full and partial report in the number available of letters was when the tone occurred ____ milliseconds after the offset of the visual display"
96A39,6,4,"4 c","4 c","5 c","5 c","5 c","4 i","3 c","1 c","1 i","5 i"
1E90A4,5,3,"4 c","4 c","5 c","5 c","2 i","5 c","3 c","1 c","4 i","4 i"

First, I need to remove the numbering of the questions from the first line. So "1) S31 Which best describes how you answered the online reading comprehension quiz?" would become "S31 Which best describes how you answered the online reading comprehension quiz?"

Next I need to remove an entire column if its row header doesn't begin with either B or C or is the "Device ID" column. For example, column starting with B19 is kept, the C23 is kept, and"Device ID" column is kept. The column with the question starting S31 must be entirely removed.

Thanks.

Joining MP3 Files Together Based on Date in Filename?

WCityMike — Wed, 25 Mar 2009 16:08:55 -0800

I have a podcast that downloads as multiple segments over multiple days. Its format arrives as YYYYMMDD_showcode_segment.mp3; so, for example, one download might result in files as 20090323_goofa_01.mp3 through 20090323_goofa_21.mp3, and also within the same download 20090324_goofa_01.mp3 through 20090324_goofa_21.mp3. What I'm looking for is a Unix script (shell, perl, what have you) that would look at a directory and look at what dates are represented there, and let me join all of the segments together by day. I can use mpgtx -j to do the actual joining; I'm looking for the code that would let the script determine what dates are in the directory and then get each day's segments united into a single MP3 for that day's show.

BASH: Handling odd characters in a variable?

Ziggy Zaga — Tue, 18 Nov 2008 13:58:55 -0800

BASH Scripting. I'm trying to do a simple "mv" operation with a not-so-simple filename that contains special characters (as in brackets and hyphens). The filename itself is stored within a variable. The operation fails, and I can understand why, but I don't know how to get around it. A sample filename would be something like:

[2008] Region {XX}.pdf

...and is stored in $sFilename. Not all of the filenames are the same format although all of them start with the year in square brackets.

I've got the rest of the script set up to parse for the year in the initial set of [ ] brackets and create a folder (./2008) with that name. The second part of the script is supposed to move the aforementioned file into the created folder. I attempt to do this with the command:

mv "$sFilename" ./$sPrefix

...where $sPrefix is the parsed item from the brackets. It errs out with the message:

mv: cannot stat `[2008]': No such file or directory.

It seems to me the shell is choking on the filename. What's the best way to work around this WITHOUT renaming the files (since the program that imports them is Windows-based and requires they be intact)?

Thanks

How do I tell my Mac that I'm me?

grumblebee — Wed, 15 Oct 2008 07:17:27 -0800

Question about changing my terminal prompt (and other matters) in an OS X Bash shell. I'm not the most experienced Unix guy, but I can usually figure things out. But this one has me stumped: I just got a slightly-used Macbook Pro. The previous ower -- Michael -- gave me his admin password. So I went into System Preferences, Accounts and created a new admin, grumblebee. Then I deleted Michael.

All went well until I opened terminal, where I saw that the prompt looked like this:

michaels-mackbook-pro: ~Grumblebee

(I know that ~Grumblebee isn't part of the prompt. I just wanted to show you that I'm logged in as me and it's finding my home directory. It's just the prompt that's wrong.)

I tried to edit etc/baschrc, but it said I didn't have permission.

How can I change the prompt and is it likely I'll encounter any other problems connected to changing users?

What resources do you guys recommend on learning to BASH script, and also learn from other peoples code?

Javed_Ahamed — Sat, 11 Oct 2008 09:02:50 -0800

What resources do you guys recommend on learning to BASH script, and also learn from other peoples code? Hey guys, I am quite interested in learning to Bash script and was wondering what guides you guys thought were the best suited for this task. Also, since learning by example works better for me any resources that taught this way would also be much appreciated.

On the other hand I also realize that learning from actually programming is the much better way, so if there were any sites that like gave you a problem to solve in bash and gave a sample solution to that problem would be the best! Also a nice place to look at others Bash code to learn from it would also be much appreciated.

Quicksilver-style completion in bash shell?

dmd — Fri, 11 Jul 2008 06:44:57 -0800

How can I get Quicksilver-style completion in the bash shell? Suppose I have a bunch of files in a directory. One of them is named 'metafilter'. I'd like to say cat mflr[TAB] and have that be expanded.

I think this is equivalent to asking for an implicit * between each typed character.

Anyone have bash-completion-fu?

How to delete long path names with bash shell script?

jackofsaxons — Mon, 02 Jun 2008 22:07:17 -0800

BASHFilter! I've got a simple bash script that I've written to clean out a frequently used NAS. I run it every 30 days, it sets proper permissions, removes extraneous files and directories, but it frequently hangs on paths longer than 255 characters. I'm no bash ninja, so I've been beating my head against a wall trying to figure out how to check path length of directories, and then prompt for either renaming to shorten them, or prompt for deletion. Help me Metafilter!

Help installing Debian Etch?

geoff. — Fri, 23 May 2008 10:03:32 -0800

Okay I'm trying to set up rtorrent and wtorrent using Debian Etch and this guide. Installs and updates the kernel fine, then I get "bash command not found" when trying to run through the rest of the tutorial. I tried this from a blank slate twice. I must be missing something stupid. My Linux knowledge is limited but I've setup wikis and things just fine. It is running on VMWare ESX as "Other Linux (32-bit)" ... it updated the kernel without incident, and I have SSH running. When I terminal in as root:

rtorrent:~# apt-get install apache-common apache2-utils autoconf automake \
>
autotools-dev binutils build-essential bzip2 ca-certificates comerr-dev \

cpp cpp-4.1 dpkg-dev file g++ g++-4.1 gawk gcc gcc-4.1 libapache-mod-php5 \

libapr1 libaprutil1 libc6-dev libcurl3 libcurl3-openssl-dev libexpat1 \
eading package lists... 0%
Reading package lists... Done
libidn11 libidn11-dev libkadm55 libkrb5-dev libmagic1 libncurses5-dev \
uilding dependency tree... 0%
libneon26 libpcre3 libpq4 libsigc++-2.0-dev libsqlite0 libsqlite3-0 \

libssl-dev libssp0 libstdc++6-4.1-dev libsvn1 libtool libxml2 lighttpd \

linux-kernel-headers lynx m4 make mime-support ntp ntpdate openssl patch \

perl perl-modules php5 php5-cgi php5-common php5-sqlite pkg-config screen \

Building dependency tree... Done
apache-common is already the newest version.
apache2-utils is already the newest version.
autoconf is already the newest version.
automake is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
rtorrent:~# autotools-dev binutils build-essential bzip2 ca-certificates comerr-dev \
>
-bash: autotools-dev: command not found
rtorrent:~# cpp cpp-4.1 dpkg-dev file g++ g++-4.1 gawk gcc gcc-4.1 libapache-mod-php5 \
>
-bash: cpp: command not found
rtorrent:~# libapr1 libaprutil1 libc6-dev libcurl3 libcurl3-openssl-dev libexpat1 \
>
-bash: libapr1: command not found
rtorrent:~# libidn11 libidn11-dev libkadm55 libkrb5-dev libmagic1 libncurses5-dev \
>
-bash: libidn11: command not found
rtorrent:~# libneon26 libpcre3 libpq4 libsigc++-2.0-dev libsqlite0 libsqlite3-0 \
>
-bash: libneon26: command not found
rtorrent:~# libssl-dev libssp0 libstdc++6-4.1-dev libsvn1 libtool libxml2 lighttpd \
>
-bash: libssl-dev: command not found
rtorrent:~# linux-kernel-headers lynx m4 make mime-support ntp ntpdate openssl patch \
>
-bash: linux-kernel-headers: command not found
rtorrent:~# perl perl-modules php5 php5-cgi php5-common php5-sqlite pkg-config screen \
>
Can't open perl script "perl-modules": No such file or directory
rtorrent:~# sqlite subversion ucf zlib1g-dev

Buh. I searched around and it looks like there might be something wrong with my Bash profile? What am I doing wrong here?

rtorrent:~# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

OS X file permissions issue

caution live frogs — Tue, 20 May 2008 15:41:37 -0800

OS X Sites folder permissions and a question about flags on the files in bash. What gives? I use my MacBook Pro as a test server, running Apache. I sync my files (two-way sync, using Chronosync) between my Mac and my home Windows system. Sometimes after a sync my test site breaks - I don't have permission to access my own files. Sure, a chmod -R 755 * on my Sites folder fixes it, but why is it happening? I don't need to have files at 755 for them to work on my Linux server, so why does the Mac refuse to deliver any CSS or javascript files unless execute permissions are set?

Finally, this one is driving me nuts: What's the extra flag at the end of the file for in Terminal? When I check file permissions with ls -al some files show as -rwxr-xr-x+, others show as -rwxr-xr-x@, and some are -rwxr-xr-x with no + or @ appended. Nothing I've been able to find online explains what the @ or + flags mean, or why they may be appended to some files but not others. I've never seen similar flags on any Linux system I've used. Should I even care?

Shell scripting or something better?

Blazecock Pileon — Fri, 09 May 2008 14:19:40 -0800

Faster techniques for grabbing lines of a file and doing text manipulation and arithmetic via shell (or other) scripting? 1. Are there faster ways to grab lines of a text file?

Currently I use while read line and sed -n #p filename in bash and csh scripts to grab lines of a file I'm interested in. This seems slow. Are there better (faster) ways to get the line of a file, or to iterate through specified ranges of lines in a file?

2. Are there faster ways than awk to grab values in a line?

Let's say I have the tab-delimited line:

abc 123 345 0.52

What I'd like to do is get the second and third values, or the fourth value, as quickly as possible. Is there a better way than awk? Will a perl or other interpreted language script run faster than a shell script for scraping values from a text file?

3. Arithmetic with bash?

I've been doing $((${value1}+${value2})) for integer arithmetic and calc ${value1} / ${value2} for floating point arithmetic within bash. Will I gain a performance benefit from switching over my code from bash to another shell script language, or to another interpreted language entirely?

Thanks for any and all tips and tricks.

Simple Filename Parsing Question

yellowbkpk — Sun, 09 Mar 2008 07:33:42 -0800

Should-be-simple Linux timestamped file parsing question. I have a directory with what is now 10k jpeg files in this format (number of seconds since 1970):

1202499302.jpg
1202419201.jpg
1202439301.jpg
1202459401.jpg
1202479501.jpg
1202499602.jpg
1202419502.jpg
1202439601.jpg
1202459702.jpg
1202479801.jpg
1202499901.jpg

I'd like to use a script of some sort to shuffle these images into directories based on month, then day. Note that the "last modified" times on the file system are not necessarily the same as the timestamp in the file name, and I'd like to collate the files with the file name timestamp, not the filesystem timestamp.

Can anyone suggest a way to do this?

how split a string in bash?

jimjam — Sat, 12 Jan 2008 19:21:39 -0800

using bash, i need to split a variable in two, on whitespace, but with just the first word in one variable, and the rest in the second variable. this is a one liner with perl. how in bash? i have a properties file in the format:

variable (whitespace) value

values can include spaces, variables are always a single word. e.g.:

thisvar contents of this var

i need to extract 'thisvar' and 'contents of this var' as two strings.

in perl i'd do:

($first, $rest) = split (' ',$source)

how can i do this in a shell? (i.e. bash)

How can I insert text into an existing PDF using Ghostscript?

jackofsaxons — Fri, 21 Dec 2007 21:25:10 -0800

How can I insert text into an existing PDF using Ghostscript? I'm attempting to write a bash script that will take the output of the "tree" command, and insert it into an existing PDF template. The script works, bu I haven't been able to figure out how to take the output of it and stick it into this template. Does anyone know of a solution that does no use any additional software other than Ghostscript?

Remotely starting local X application?

Kadin2048 — Fri, 16 Nov 2007 21:16:09 -0800

How do I restart an X Windows application running on a remote machine, and have it display on that computer's locally-attached display? (I do not want to forward/tunnel it to the machine I'm sitting at.) Two computers, foo and bar. Foo is a machine connected to a big display, but lacking any local controls (keyboard or mouse). Bar is a laptop. Foo normally runs an X Windows application, but sometimes this application crashes.

How can I, by connecting to foo from bar via SSH, restart an X app running on foo, and have it come back up on foo's display? All the resources I can find on the net deal with forwarding X connections so that I could display them remotely on bar (which I can do fine), but that's not what I want. I want to issue the command via SSH, but have the application display itself on the local display. (And once I get it running, I want to close down the SSH connection.)

I'm sure it has something to do with the DISPLAY variable, but I can't figure out what to set it as. (I've tried setting it to "localhost:0" and "127.0.0.1:0" as well as ":1" on both, but both give "can't open display" errors.)

It seems like this ought to be a lot easier than tunneling an X connection, but I'm stumped.

This is all using Knoppmyth, which is Debian/X11.org-based, with Bash.

Bash, I command thee to always...

nakedcodemonkey — Mon, 22 Oct 2007 16:27:48 -0800

How do I get bash to automagically execute a line or two of code at login? I ssh into various web servers all day long. They're helpfully *cough* configured to log me off after a few minutes inactivity, so re-logging in happens a lot. Then the re-return to the correct working directory. Where do I put something like 'cd ~/foo/bar' to instruct bash to execute that step (and whatever other convenience steps I can think of) automatically at shell login? I tried just dropping that into ~/.bash_profile, but no dice. 'chmod +x ~/.bash_profile' didn't make a difference.

Help improve my scripting skills

pmbuko — Wed, 29 Aug 2007 06:55:53 -0800

I wrote a shell script for an acquaintance that renames Macs (OS 10.4) based on their existing names. It works well, but I'm convinced my code could be more efficient. Where can I make improvements? First, some background. The old naming scheme was [Building]_[CPU speed]_[Model]_[number]. The length of the names was cumbersome, so they decided to shorten them to [abbreviated building name]-[model]-[A/B][number], where A=1.42GHz and B=1.25GHz. For example, if the old names are

Berkeley_1.25Ghz_eMac_01
Washington_700MHz_iMac_01

the new names should be

berk-emac-B01
wash-imac-01 (they only have 700Mhz iMacs)

This is the bash script I came up with (with comments), that gets run as root on each computer:



# get the current computer name

NAME=`scutil --get ComputerName`



# split the name into its component parts

BLDG=`echo $NAME | awk -F_ '{print $1}'`

CPU=`echo $NAME | awk -F_ '{print $2}'`

MOD=`echo $NAME | awk -F_ '{print $3}'`

NUM=`echo $NAME | awk -F_ '{print $4}'`



# replace full building name with abbreviation

newBLDG=`echo $BLDG | sed \

    -e 's/Berkeley/berk/' \

    -e 's/Washington/wash/'`



# make model lowercase

newMOD=`echo $MOD | tr [:upper:] [:lower:]`



# replace cpu speed with letter (or no letter for 700MHz)

newCPU=`echo $CPU | sed \

    -e 's/1.42GHz/A/' \

    -e 's/1.25GHz/B/' \

    -e 's/700MHz//'`



# reassemble name from new parts

newNAME=$newBLDG-$newMOD-$newCPU$NUM



# assign the new name to the computer

scutil --set ComputerName $newNAME

scutil --set LocalHostName $newNAME

Strip me some mp3 urls.

Jimbob — Thu, 10 May 2007 05:32:14 -0800

In Linux, what's the simplest way to take a podcast RSS file as input, and output a file containing just the URLs of all the MP3s enclosed in the RSS file? Difficulty: the method must only Bash + GNU tools, Python or Perl, with no non-standard add-on libraries or dependencies required.

Help me consolidate thousands of pictures

neilkod — Thu, 25 Jan 2007 11:07:07 -0800

I have a bunch of directories of pictures across multiple disks. Many images are duplicated. Additionally, I had to use some disk recovery software to rescue other images and the filenames changed. At the moment, I can't guarantee that all of the filenames are unique. I'd like to weed out the duplicates and then ultimately consolidate everything into iPhoto. I'm talking close to 20,000 pictures. I had two backup volumes fail simultaneously. I was able to recover some pictures from drive A, some from drive B, and not really know what I have in common. I'm sure there is LOTS of overlap.

Is there a program (hello perl wizards) that will weed out duplicates by CRC? I'm sure that through the various recovery procedures, the pictures were given different names. And I'm not quite sure the naming is unique, so I dont want to just

find . -name \*.jpg -exec mv {} mynewdirectory \;

for fear that I'll overwrite files. Plus that wont weed out the dupes. Any better ideas?

iPod abuse

Kiwi — Thu, 28 Dec 2006 10:35:43 -0800

Stop me hitting my iPod! My 4th gen iPod requires frequent slapping to encourage it to unfreeze (try Googling 'slap+ipod' to see what I mean). What does this particular 'fix' actually do? How do I fix my iPod in a more permanent way? It seems that since this slapping is often successful, there might be single problem that it is fixing? Loose HD connection or something?

How to get a screen session to work the way I want to?

davar — Sun, 05 Nov 2006 09:17:07 -0800

How do I get my Linux shell to behave the same when I am using screen, as when I am not using it? I connect to my linux servers with Putty, from Windows. These are two different servers, but I have the same annoyance. When I work in the shell (bash) without using screen, and I use "less" or "man" or "vim" or something like that to read a file, and I close that file, the open file remains visible in the window. I just start typing new stuff below it. If I do not use screen, when I close the file, I get returned to the shell window I was using before, and I can see my previous commands.

Also, when I am not in a screen session, I can use the scrollbars in Putty to scroll up in history. When I am in a screen session, this does not work. I can imagine this is a limitation of screen sessions, but if there is a solution for this, I would like to know as well.

The latter annoyance is not as important, but the former has been irritating me for a long while now, and I hope there is an easy solution.

Changing gnome wallpaper

Jimbob — Tue, 26 Sep 2006 03:18:50 -0800

Linux / Ubuntu / Gnome question - how do I tell Gnome "Hey, the jpg you're using as desktop wallpaper has been updated with a new image. Go reload it and display the new version?" in a Python or Bash script? I've written a little Python script that runs once a day, and downloads the NASA Astronomy Image Of The Day, and saves it to a local file. Gnome is set up to use this file as my desktop wallpaper. The problem is, while the script downloads the new image every day, Gnome never checks the file to see if it's changed, so my wallpaper remains the same unless I restart the computer, or manually change the wallpaper on the Gnome desktop.

How can I automatically tell Gnome to "reload" the wallpaper jpg?

bash bash bash

provolot — Tue, 08 Aug 2006 13:03:24 -0800

Calling all bash/nix gurus.. please help me! I have a list of users that I'm trying to parse and convert into a html-friendly format, surrounding them around <option> tags so I can put them in an HTML select form.

I'm getting input from /etc/passwd, grepping to find users that are enabled login, cutting and getting the first segment before a colon, and sorting in alphabetical order -- then surrounding the resulting list of login IDs in option tags:

cat /etc/passwd | egrep "\[N?U" | cut -d : -f 1 | sort | sed -n 's/.*/<option value=\"&\">&<\/option>

Which yields:
<option value="loginID1"\>loginID1</option\>
,etc.

That's all fine, not a problem. But -- now I want to run each ID through 'lookup' and grep the name out, and then insert between the option tags, where the second ampersand is, like this:

<option value="loginID1">Firstname Lastname1</option>

Assuming that lookup loginID1 returns Firstname Lastname 1, how can I do this? It seems that I need to keep the original values of the /etc/password - parsed values while also running them through lookup. Is there any way I can do this in one line, without having to resort to loops/arrays in bash? Or will an in-line Perl script help?

Thanks!

No more shell games!

furtive — Wed, 31 May 2006 17:02:01 -0800

*nix shell "best practices" anyone? Over the years I've been writing more and more shell scripts (bash, ksh, perl, expect) and they've been getting more and more complexe. I haven't been able to find a comprehensive list of best practices for shell programming although I have learned a lot by looking at other people's scripts (thank you sanity checks!) I'm looking for links or your own best practices.

Here's an example:

Use ${FOO} instead of $FOO so that the variable is more likely to be interpreted when concatenated in a string and use >> "${FOO}" instead of >> ${FOO} in case ${FOO} has spaces in it.

Python - Shell Environment

kuatto — Fri, 07 Apr 2006 12:17:12 -0800

Is there a way to set an environment variable in the unix shell from a python script? Can I set a variable in the shell that executes that same script? For that matter, is there a way for any script to do this? In other words, I am trying to set up a nice environment in my unix shell (potentially bash or tcsh), and I need to set up some custom paths. I would like to do this from a script.

It seems that,

MY_PATH = os.getenv('PATH', 'Error')
os.environ["PATH"] = MY_PATH + ':' + install_path + '/bin'

sets $PATH in some subshell spawned by python, and it doesn't update the environment of the shell that executes the python script.

I had this problem in trying to write a bash script as well.