Bash scripting problem

August 11, 2009 4:29 PM

Please hope me with a bash scripting problem. Involves: command substitution and word splitting.

In my script I'm trying to wrap the curl command to give it a global retry value. So I have this.

When I call it like this, things seem to work:

retry_curl "http://example.org/file1.txt"

But I have one place where I want to call it like this:

LOCAL_CURLOPTS="--time-cond \"Tue, 11 Aug 2009 18:12:12 -0500\""
retry_curl $LOCAL_CURLOPTS "http://example.org/file1.txt

And what I get on the console is:

curl: option -0500: is unknown
curl: try 'curl --help' or 'curl --manual' for more information

So somewhere bash is breaking apart my LOCAL_CURLOPTS as words, and then using that in the command substitution. What I can't figure out is how to get it to stop.

posted by sbutler to Computers & Internet (11 answers total)

What if you used single quotes inside your CURLOPTS? (The trial and error approach)
posted by Obscure Reference at 4:38 PM on August 11, 2009

Response by poster: Ohhh... maybe I managed to fix this by changing the command substitution to:

CURLOUT=$(curl $CURLOPTS "$@")

And then splitting my call into:

if [ -n "$TIME_COND" ]; then
  retry_curl --time-cond "$TIME_COND" "http://example.org/file1.txt"
else
  retry_curl "http://example.org/file.txt"
fi

Does that look right to the bash experts out there?
posted by sbutler at 4:39 PM on August 11, 2009

braces may solve your problem.
posted by namewithoutwords at 4:49 PM on August 11, 2009

Best answer: Bash doesn't do nested quote removal and word splitting - it does only one pass of each. So the double-quotes you've inserted into $LOCAL_CURLOPTS end up with no semantic meaning to bash. A toy example will make this clearer. Try this in a terminal:

First, set things up so bash will tell us how it's parsing the commands we enter:

stephen@jellybelly:~$ PROMPT_COMMAND=
stephen@jellybelly:~$ set -x

Now set a couple of variables:

stephen@jellybelly:~$ cruel=kind
+ cruel=kind
stephen@jellybelly:~$ cry="goodbye \"$cruel world\""
+ cry='goodbye "kind world"'

Now let's see how that last variable gets expanded:

stephen@jellybelly:~$ echo $cry
+ echo goodbye '"kind' 'world"'
goodbye "kind world"

Note that echo is getting invoked with three parameters, not two - the embedded double quotes inside $cry are not examined during word splitting. Quoting $cry doesn't help:

stephen@jellybelly:~$ echo "$cry"
+ echo 'goodbye "kind world"'
goodbye "kind world"

The quotes around $cry turn word splitting off altogether, and echo gets invoked with one argument.

Now, there are assorted ways you can force bash to do multiple passes of quote removal and word splitting (hint: eval) but most of them end up turning horribly ugly really quickly and getting even more confusing. Better not to go there.

You need a different design pattern. Instead of

DEFAULTS="big list of default options"
command $DEFAULTS more options

you can use

augmented_command () { command big list of default options "$@" ; }
augmented_command more options

and then everything will Just Work.

Note in particular the use of "$@" rather than $@ inside the definition of augmented_command. This will ensure that any properly quoted single arguments you pass to augmented_command will also be expanded as properly quoted single arguments. In particular, if you've used this pattern inside retry_curl, then

local_curl () { retry_curl --time-cond "Tue, 11 Aug 2009 18:12:12 -0500" "$@" ; }
local_curl http://example.org/file1.txt

will work as you'd expect it to.
posted by flabdablet at 5:41 PM on August 11, 2009 [3 favorites]

Best answer: Alternatively, if you really truly prefer the OPTIONS= pattern, you can make OPTIONS an array variable instead of a simple string:

LOCAL_CURLOPTS=(--time-cond "Tue, 11 Aug 2009 18:12:12 -0500")
retry_curl "${LOCAL_CURLOPTS[@]}" http://example.org/file1.txt

should get the job done too.
posted by flabdablet at 5:53 PM on August 11, 2009

Response by poster: flabdablet: so if I understand your explanation correctly, in this:

LOCAL_CURLOPTS=(--time-cond "Tue, 11 Aug 2009 18:12:12 -0500")
retry_curl "${LOCAL_CURLOPTS[@]}" http://example.org/file1.txt

If I remove the double quotes from around ${LOCAL_CURLOPTS[@]}, then bash passes my function 8 parameters, instead of 3?
posted by sbutler at 6:02 PM on August 11, 2009

Best answer: Yes, and you can verify this pretty easily:

stephen@chironex:~$ PROMPT_COMMAND=
stephen@chironex:~$ set -x
stephen@chironex:~$ LOCAL_CURLOPTS=(--time-cond "Tue, 11 Aug 2009 18:12:12 -0500")
+ LOCAL_CURLOPTS=(--time-cond "Tue, 11 Aug 2009 18:12:12 -0500")

Here we see the null (:) command being invoked with two arguments, '--time-cond' and 'Tue, 11 Aug 2009 18:12:12 -0500':

stephen@chironex:~$ : "${LOCAL_CURLOPTS[@]}"
+ : --time-cond 'Tue, 11 Aug 2009 18:12:12 -0500'

Without the quotes, the whole lot gets word-split and the null command is passed seven arguments, '--time-cond', 'Tue,', '11', 'Aug', '2009', '18:12:12' and '-0500':

stephen@chironex:~$ : ${LOCAL_CURLOPTS[@]}
+ : --time-cond Tue, 11 Aug 2009 18:12:12 -0500

Have a look at the reference manual sections on $@ and the @ and * subscripts for arrays.
posted by flabdablet at 7:39 PM on August 11, 2009

Response by poster: Okay; it's making sense now.

Perl, Python, Ruby, TCL: these I'm fine with. But any length of time with Bash makes my head hurt. Thanks for clearing it up.
posted by sbutler at 7:42 PM on August 11, 2009

Perl, Python, Ruby, TCL: designed to write programs in. Bash: designed as an interactive command processor that you can write programs in.

Programming in bash is like knitting. It's fun for its own sake, it's not the fastest way to get the job done, and sometimes the things you make with it end up being favorites you keep for years and years :-)
posted by flabdablet at 7:56 PM on August 11, 2009

Response by poster: I can't remember why I chose to do this particular project in bash. I think it's because 90% of the scripts are just calling out to other programs and I thought it'd be silly to write it in perl.

And now I'm just so far along...
posted by sbutler at 8:35 PM on August 11, 2009

For what it's worth, I'd do retry_curl this way. In particular, it's important to wrap the $() call to curl in quotes - this

CURLOUT="$(curl "${CURLOPTS[@]}" "$@")"

instead of this

CURLOUT=$(curl "${CURLOPTS[@]}" "$@")

because the output from curl might well contain shell wildcards and you almost certainly don't want those expanded as globs against the current directory.
posted by flabdablet at 1:19 AM on August 12, 2009