Shell script variable inside sed instructions?

May 1, 2007 3:52 PM

Using shell script variables inside sed instructions?

This uses the literal value of the variable name, rather than the value:

cat $PRODUCTION_URL.com.conf | sed -e 's/ServerName .*/ServerName $PRODUCTION_URL/' -e 's/ErrorLog .*/ErrorLog logs\/$PRODUCTION_URL-error_log/' -e 's/CustomLog .*/CustomLog logs\/$PRODUCTION_URL-access_log common/' > $PRODUCTION_URL.com.conf


What needs to be done to the sed instructions to make it use the value of the variable?

posted by xmutex to computers & internet (12 comments total)

Double quote.
posted by synaesthetichaze at 3:54 PM on May 1, 2007

I.e.:

cat $PRODUCTION_URL.com.conf | sed -e "s/ServerName .*/ServerName $PRODUCTION_URL/" -e "s/ErrorLog .*/ErrorLog logs\/$PRODUCTION_URL-error_log/" -e "s/CustomLog .*/CustomLog logs\/$PRODUCTION_URL-access_log common/" > $PRODUCTION_URL.com.conf

However, you may want to redirect your output to a different file to avoid clobbering the one you are editing.... such as $PRODUCTION_URL.com.conf.new
posted by synaesthetichaze at 3:58 PM on May 1, 2007

Erm?

cat $PRODUCTION_URL.com.conf | sed -e 's/ServerName .*/ServerName '$PRODUCTION_URL'/' -e 's/ErrorLog .*/ErrorLog logs\/'$PRODUCTION_URL'-error_log/' -e 's/CustomLog .*/CustomLog logs\/'$PRODUCTION_URL'-access_log common/' > $PRODUCTION_URL.com.conf

gives the same result

Using " characters, same result.
posted by xmutex at 4:00 PM on May 1, 2007

Which shell/OS?
posted by synaesthetichaze at 4:00 PM on May 1, 2007

nice! worked indeed. Thanks.
posted by xmutex at 4:02 PM on May 1, 2007

(You get an Extraneous Use of "cat" Award.)

Strange. What does echo $PRODUCTION_URL "$PRODUCTION_URL" '$PRODUCTION_URL' ${PRODUCTION_URL} get you?

My suggestion:

$ sed \
 -e "s#\(ServerName\) .*#\1 ${PRODUCTION_URL}#" \
 -e "s#\(ErrorLog\).*#\1 logs/${PRODUCTION_URL}-error_log#" \
 -e "s#\(CustomLog\) .*#\1 logs/${PRODUCTION_URL}-access_log common#" \
 < ${production_url}.com.conf \br>
 > tmp$$ && mv tmp$$ ${PRODUCTION_URL}.com.conf

posted by cmiller at 4:17 PM on May 1, 2007

for next time: sed -ibak -e 'stuff1' -e 'stuff2' filename
posted by rhizome at 4:17 PM on May 1, 2007

MeFi mangled my code. Grr.
posted by cmiller at 4:18 PM on May 1, 2007

Depends on the OS rhizome... or I should say, depends on which version of sed you have. AIX's sed doesn't even have the -i flag (if you can believe it), so you have to put it down to a new file.
posted by synaesthetichaze at 4:19 PM on May 1, 2007

I see this has already been mostly answered, but I'll throw in my favorite regex-fu: instead of

's/\/some\/path/something/'

do

's,/some/path,something,'


So much easier on the eyes.
posted by oats at 7:12 PM on May 1, 2007

The way you've phrased your question says to me that your CLI background is in DOS, not in Unix, and that you've not yet fully embraced the Unix Way.

There is in fact no way to use shell variables "inside" sed scripts; sed is blissfully unaware of shell variables. What you can do, and what the double-quote solutions above do do, is get the shell to replace variable references with values before passing them to any command, including sed; this process is called parameter expansion, and you should read about it in the bash manual.

Unix shells also do several other kinds of expansion - most notably filename expansion, aka globbing - before the command they're invoking ever sees its arguments. If, for example, you were to do

rm *.doc

in a directory containing foo.txt, foo.doc, bar.txt and bar.doc, then the shell would expand this to the equivalent of

rm foo.doc bar.doc

This is different from DOS. In DOS,

del *.doc

passes the unmodified string *.doc to the del command, which is responsible for doing its own globbing.

In DOS, the command interpreter only supports one form of quoting (double-quotes) and its only effect is to avoid breaking up arguments containing embedded spaces (in Unix shell terms, quoting suppresses word splitting). Unix shells have a selection of quoting methods that give you selective control over assorted kinds of argument processing.

Double quotes turn off everything except parameter expansion and a couple of the other quoting forms, which is why you can use them to quote things like s/ServerName .*/ServerName $PRODUCTION_URL/ and not get your * all globbed up but still have $PRODUCTION_URL expanded correctly.

Single quotes turn off absolutely everything, including parameter expansion, which is why sed will end up seeing the literal value $PRODUCTION_URL if you single-quote its arguments as above. The reason you'll so often sed arguments wrapped in single quotes is that sed has its own meaning for the $ character (end of item matched by regular expression) and often needs to use embedded \ characters, both of which the shell would eat in a double quoted string.

The only thing you can't put inside single quotes is a single quote. The standard idiom for working around this is

'isn'\'t that one embedded?'

which is actually three quoted items ('isn', \', and 't that one embedded?') jammed up against each other with no spaces in between. You can get away with this in Unix shells where you wouldn't be able to in DOS, because unlike the DOS command interpreter, Unix shells remove quotes once they've done their job. With a Unix shell, you could delete a file named A file called "foo" with any of the following:

rm 'A file called "foo"'
rm "A file called \"foo\""
rm "A file called"'"foo"'

and in all cases the only thing passed to the rm command would be the literal string A file called "foo". DOS works around its command interpreter's lack of quote removal by prohibiting " from appearing in filenames altogether, and having different rules for the SET command to those for everything else. In DOS,

del "A file called 'foo'"

passes the entire string from the command line, including the surrounding " characters, to the del command.

There will be a short quiz at the end of this lecture :-)

On preview: oats raises a valuable point, which is that one of your existing substitutions uses \/ to tell sed that the / is part of a string to be substituted and doesn't mark the end of the s/// construct. If you simply use double quotes instead of single quotes as others have suggested, then the shell will interpret \/ to mean a literal / (redundant but still done) and remove it; sed will then misinterpret the / as the end of the s/// and whine about bad syntax. You could use \\/ instead, which the shell will translate to \/ and pass on, but that's horrid; use oats's tip instead.

You're also going to run into trouble with the redirect on sed's output killing your input file at the same time cat is trying to open it; far better to use the -i (in-place) sed parameter instead, as cmiller suggests. Personally I'd make the input and output separate files, and wrap the entire sed script in a single string, like this:

sed <template.com.conf >"$PRODUCTION_URL.com.conf" -e "
s,ServerName .*,ServerName $PRODUCTION_URL,
s,ErrorLog .*,ErrorLog logs/$PRODUCTION_URL-error_log,
s,CustomLog .*,CustomLog logs/$PRODUCTION_URL-access_log common,
"

posted by flabdablet at 8:13 PM on May 1, 2007 [1 favorite has favorites]

Bollocks! Metafilter ate my standard idiom. I forgot that it does't let you put two ' characters next to each other in a posting, even though they show up perfectly in preview; you have to use ' which will of course get translated to ' on preview and then eaten on post if you forget to hit the Back button first. And you thought the shell quoting rules were bizarre :-)

Try this:

'isn'\''t that embedded?'

posted by flabdablet at 8:18 PM on May 1, 2007