Multi-line shell scripts to a single line?

June 16, 2023 1:29 PM

I'm trying to get Packer for Ubuntu to install in a single copy and paste. I'm trying to avoid using a shell script for reasons beyond this question and would prefer a copy paste that outputs to a log file. My bash isn't that strong or I'm missing what some commands are doing. This has to be possible. Here's the multi-line command inside.

$ curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
$ sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
$ sudo apt-get update && sudo apt-get install packer

I'd like to make it a simple copy and paste in a readme type file. I've tried multiple things I'll just give some examples, the only thing I want to add is logging with tee (or whatever works best):

curl -fsSL https://apt.releases.hashicorp.com/gpg \
&& sudo apt-key add - \
&& sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main" \
&& sudo apt-get update && sudo apt-get install packer | tee "./logs/packer/install/install_$(date +%Y%m%d-%H%M%S).log"

And


% (curl -fsSL https://apt.releases.hashicorp.com/gpg ;\
sudo apt-key add - ;\
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main" ;\
sudo apt-get update && sudo apt-get install packer) | tee "./logs/packer/packer_install_$(date +%Y%m%d-%H%M%S).log"

The logging works fine but it'll sometimes stall out and the only log I'll get is this:

gpg: signal 2 caught ... exiting
tee: /logs/packer/install/install_20230616-143122.log: No such file or directory
Repository: 'deb [arch=amd64] https://apt.releases.hashicorp.com jammy main'
Description:
Archive for codename: jammy components: main
More info: https://apt.releases.hashicorp.com
Adding repository.
Press [ENTER] to continue or Ctrl-c to cancel.


I tried curly brackets around commands but I may not be understanding quite how bash interprets commands and at this point I'd like to get it working for my own edification. I can get it working running multi-line but would really at this point I'd like to know if this is possible or what I'm doing wrong. I omitted some other tries but I believe there's got to be a simple way to do this and I'm not catching it. Thanks!

posted by geoff. to Computers & Internet (13 answers total)

My first guess is that you need to do

sudo add-apt-repository -y "deb ..."

My guess from the output you see is that it's waiting for you to confirm and then timing out. Adding the -y option means "say yes to all questions". There may be a similar thing you need to do with the install command.

(Also note that you no longer need to use apt-get; in modern Ubuntu versions it's just apt. Not a problem with your command, but that's just the modern idiom. So just sudo apt update and sudo apt install...)

If that's not it, one other thought: what directory are you running this from? I'm wondering if you're getting permissions problems when you try to direct the output to the log file with tee because you're in a directory where you don't have permissions to create/write to the file.
posted by number9dream at 2:01 PM on June 16, 2023

/logs/packer/install/install_20230616-143122.log: No such file or directory

tee can't create this log file because the directory /logs/packer/install doesn't exist.
posted by fritley at 2:21 PM on June 16, 2023 [1 favorite]

Both the above answers are correct. (tee will still pass the output to your terminal even if it can't copy it to a file, so that doesn't stop things running, but it obviously isn't writing your log.)

You're also only saving output messages, and you probably also want to save error messages. To save both output and error messages to your log, you would do |& tee ... or |tee ... 2>&1 .

I would also recommend apt-get -y ... everywhere you use it. They will all ask questions in some circumstances; this tells them to assume the answer is always yes. Your original code was written for humans to run, and implicitly assumes that humans will respond to any of these questions without mentioning it.
posted by How much is that froggie in the window at 4:10 PM on June 16, 2023

Response by poster: Thanks all I noticed I changed directory names but it wasn't the core problem. It looks like installing Packer has some issues on Ubuntu 20.04 Jammy and I moved it into a shell script just for now to get it working. I get a bunch of errors here:

W: Target CNF (main/cnf/Commands-all) is configured multiple times in /etc/apt/sources.list.d/archive_uri-https_apt_releases_hashicorp_com-jammy.list:1 and /etc/apt/sources.list.d/hashicorp.list:1

If I remove one the errors go away but I must be doing my shell script wrong and it keeps adding it when I run it. Here's the working script:


#! /bin/bash
sudo apt install moreutils
LOG_NAME=./logs/packer/install_$(date '+%Y-%m-%d-%H-%M').log
exec > >(tee >(ts >> $LOG_NAME) 2>&1)
set -x
sudo apt install gpg
wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg
gpg --no-default-keyring --keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg --fingerprint
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update
sudo apt install packer


I'm not that familiar with adding gpg or correcting the source list so I'm guessing my script adds it twice somehow. Would also be nice to ignore the PGP key being added as it is non ASCII characters and maybe some logging that has coloring or visual indication of inputs vs outputs and errors and warnings. Right now it is a plain text file. I don't want to over complicate this but I have a bunch (~200) junior developers that won't know how to troubleshoot so my goal is to create a dashboard that will send exact logs to a central dashboard that'd be easy to read.

Thanks for everyone's help any other advice is appreciated I'll need to create these scripts for Windows and OS X and I can control the version/laptops luckily. The goal is to create a bullet proof as possible docker environment which unfortunately will need to be a mix of Windows and Linux containers in a lightweight dev box. The developers refuse to learn Docker but that's another topic I want to abstract them as much as possible from it.
posted by geoff. at 6:10 PM on June 16, 2023

You changed curl ... | sudo apt-key add -, which sends the content at the given URL to apt-key add running as root, to curl ... && sudo apt-key add - which sends the content at the given URL to the terminal (or where ever normal output is going) and then runs apt-key expecting input from the terminal (or where ever normal input is coming from), which probably is related to why this will "stall", since it's actually gpg waiting for your input.

There may be other issues as well, but the change from | (pipe) to && (conditional execution) seems likely to be an issue.
posted by the antecedent of that pronoun at 7:33 PM on June 16, 2023 [2 favorites]

I have no experience with Packer so I'll confine my remarks to dealing with the following only:

My bash isn't that strong or I'm missing what some commands are doing. This has to be possible. Here's the multi-line command inside.

$ curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
$ sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
$ sudo apt-get update && sudo apt-get install packer

I'd like to make it a simple copy and paste in a readme type file. I've tried multiple things I'll just give some examples, the only thing I want to add is logging with tee (or whatever works best):

curl -fsSL https://apt.releases.hashicorp.com/gpg \
&& sudo apt-key add - \
&& sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main" \
&& sudo apt-get update && sudo apt-get install packer | tee "./logs/packer/install/install_$(date +%Y%m%d-%H%M%S).log"

The second block of commands is not a logged form of the first, as taotp points out. If I had three sequential commands that I know are correct and I wanted to make them into a single copy/paste block that still runs interactively while also capturing all output in a logfile, then rather than altering their existing syntax I'd just wrap them in braces, like this:

{
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install packer
} 2>&1 | tee "./logs/packer/install/install_$(date +%Y%m%d-%H%M%S).log"

Explanation of rationale to follow.
posted by flabdablet at 7:48 PM on June 16, 2023

The `apt-add-repository` command also wants confirmation which will make the commands seem to stall; you can add `-y` to assume yes.

Here's a 1-liner that seems to work on ubuntu jammy (split for display on metafilter)

sudo sh -c 'curl -fsSL https://apt.releases.hashicorp.com/gpg | apt-key add - 
&& apt-add-repository -y -S "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
&& apt-get update && sudo apt-get -y install packer'  

I ran this in docker (docker run -it ubuntu:latest), though I had to

apt update && apt install -y --no-install-recommends sudo curl lsb-release software-properties-common gnupg

as a preliminary step, because none of these pieces of software are part of the minimal docker image. (and software-properties-common brings in a startling amount of STUFF!

Note that this use of apt-add-repository and apt-key add is deprecated. Here's a recipe that works in docker and avoids deprecated things (afaik) as well as installing any more software than necessary; quote it properly and prefix with sudo for "normal" usage:

apt update && apt install -y --no-install-recommends curl ca-certificates && \
curl -fsSL https://apt.releases.hashicorp.com/gpg > /etc/apt/trusted.gpg.d/hashicorp.asc &&
. /etc/lsb-release &&
echo "deb [arch=amd64] https://apt.releases.hashicorp.com $DISTRIB_CODENAME main" > /etc/apt/sources.list.d/hashicorp.list &&
apt update && apt install -y packer

posted by the antecedent of that pronoun at 7:59 PM on June 16, 2023

Best way to see what the command block I posted above does is to try it out at a terminal, pasting one line at a time. You'll find that once bash has seen the opening { it won't run commands as you enter them; rather, it will just keep on prompting for more lines until it finds the corresponding closing } and then run the whole block. So it will actually behave the same way whether you paste it line by line or all in one chunk.

Structurally, this parses as a two-component pipeline, with the braced block as the left component and the tee as the right component.

When multiple commands are invoked as a pipeline, bash runs each pipeline component in its own subshell, with standard output from each component feeding into an anonymous FIFO whose output end is supplied to the next component along. In this instance the pipeline has two components: the first is a brace-delimited block of commands to be run in sequence, and the second is tee.

The brace-delimited block has an additional 2>&1 redirection applied to it. The effect of that is to redirect output stream 2 (standard error) to the same file handle being used by output stream 1 (standard output) at the time that the redirection operator is encountered.

If we were going to redirect the entire block's output to a log file instead of having it appear in the terminal window, then the last line of the copy/paste block would have looked like this instead:

} >"./logs/packer/install/install_$(date +%Y%m%d-%H%M%S).log" 2>&1

Here we see standard output being redirected to the logfile before standard error is redirected to that same file handle. If the redirections had been applied in the other order, like this,

} 2>&1 >"./logs/packer/install/install_$(date +%Y%m%d-%H%M%S).log"

then what we'd get is standard error being redirected to standard output's existing file handle (probably the terminal) before standard output gets redirected to the logfile, and the logfile would end up failing to capture standard error.

But because we want to log both standard output and standard error and have them appear in the terminal in case of unexpected interactivity, we're not simply redirecting the braced block's output, we're piping it to tee. When bash sets up a pipeline, the redirection of standard output is implicit and happens before any explicit redirections get processed. Which is why, in the command block as I originally presented it, the 2>&1 redirection works as intended even though it appears before the pipe operator in the command line.

In fact if we'd tried to put it after the pipe operator, like this

} | 2>&1 tee "./logs/packer/install/install_$(date +%Y%m%d-%H%M%S).log"

then what bash would do is apply the redirection to the tee command rather than to the left-hand pipeline component, which isn't what we want.

The need to redirect both standard error and standard output along a pipeline is common enough that modern versions of bash have the special-purpose |& operator to do just that. The likelihood of encountering older versions that don't have it is still high enough, though, to make the 2>&1 | construction more reliable.

On input: because the braced block is the first component in a pipeline, its own standard input doesn't get redirected. Rather, the subshell it runs inside inherits the standard input from its invoking shell, which in this case will be attached to the terminal. So if any of the commands inside the braced block do end up needing to be interacted with, that remains possible. That said, using -y options as appropriate to turn off as much interaction as you can is probably a good idea.

You can use a similar braced-block-piped-to-tee wrapper around pretty much anything to capture a log of its output. Specifically, it should apply just fine to the updated command sequence recommended above by the antecedent of that pronoun.

As a final note, taotp's line that reads

apt update && apt install -y --no-install-recommends curl ca-certificates && \

would work equally well without the trailing \. If the last thing on a line that you're pasting into bash is an operator like && or | or a reserved word like { then bash will just expect the command to be continued on the next line without needing \ to make line continuation explicit.

Final note: as a quirk of history, Bourne-derived shells including bash treat braces as reserved words rather than operators. One effect of that is that when you're trying to write a one-liner that includes brace delimited blocks, the last command before the closing brace still needs the same ; or & or newline terminator that it would need if the brace were not there. Also, if you're putting an opening brace on the same line as the first word inside the braced block, you need whitespace to separate it from that word. { for works, {for doesn't.

By way of contrast, parentheses are treated as operators and will get automatically separated from stuff you jam them right up next to, so it's pretty common to see parens used instead of braces to delimit blocks purely because the syntax is less ugly. But they don't do quite the same thing. Any block of commands inside parens gets run inside its own subshell regardless of whether it's part of a pipeline or not; so if those commands do variable assignments, or change the current directory, or do global file handle redirections with exec, all those changes are sandboxed inside the subshell and have no effect outside it.
posted by flabdablet at 8:48 PM on June 16, 2023 [1 favorite]

As another aside: in sudo apt-key add - the trailing - has no special meaning to bash; it's being handed to apt-key as an ordinary command line argument in the position where a pathname would normally go, and apt-key is taking that to mean that it should read the key it's supposed to add from its already-open standard input stream rather than from somewhere in the filesystem.

That input stream, in turn, is whatever apt-key inherits when launched by sudo. So when that happens via

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -

it's the output end of an anonymous FIFO, the input end of which is soaking up whatever curl spits out.

sudo is an odd beast, in that it sometimes needs to prompt the user for a password but is also regularly used as above, where it just needs to pass a clean input stream down to the command whose privileges it's being invoked to elevate. So none of the password prompting or reading stuff that sudo needs to do ever involves its standard input, output or error streams, which also means you can't automate away its password prompts by redirecting those streams. If you invoke sudo in an environment where it can't find a tty to use for that stuff, and ask it to elevate something for which no sudoers entry exists to permit passwordless elevation, it just fails (optionally complaining about that to the root user via email).
posted by flabdablet at 9:22 PM on June 16, 2023

This line in your working script strikes me as dubious:

exec > >(tee >(ts >> $LOG_NAME) 2>&1)

If the intent is to capture a line-by-line timestamped copy of everything that also appears in the terminal window, this won't do it.

The process substitution construct >(ts >> $LOG_NAME) launches a ts process, all of whose output will be appended to $LOG_NAME (incidentally, if I'd written that script I'd have used "$LOG_NAME" rather than $LOG_NAME because it would save me needing to worry about whether the pathname held in LOG_NAME could ever contain whitespace).

The standard input of the ts process gets connected to the output end of a FIFO that also gets given a name prefixed with /dev/fd/. That name is then substituted into the command line in place of the entire >( ... ) construct, so the command line now looks something like

exec > >(tee /dev/fd/63 2>&1)

Now the outer process substitution >(tee /dev/fd/63 2>&1) is evaluated, and much the same thing happens: tee gets launched, with its standard input fed from a pipe, which gets its own name under /dev/fd/ for later substitution into the outer command line.

The 2>&1 redirection is applied inside that process substitution, which makes anything that tee writes to its standard error go to the same place as its standard output (n.b.: not to any of the files named on its command line). And since in this case tee's standard output and standard error are already the same file - the terminal's pseudo-tty - the redirection actually achieves nothing.

The process substitution happens, making the outer command line look like

exec > /dev/fd/62

which makes the shell connects its own standard output to /dev/fd/62 for the rest of the script.

What it doesn't do is connect its standard error there. So you won't see any of the command tracing turned on by set -x appear in your log file.

To make that happen, you need to make the 2>&1 redirection apply to exec, not to tee, like this:

exec > >(tee >(ts >> $LOG_NAME)) 2>&1

posted by flabdablet at 5:16 AM on June 17, 2023

If you'd rather avoid rolling your own session log, you might want to see if script will work for you.
posted by flabdablet at 7:37 AM on June 17, 2023

The three commands you did are separate commands as they need to wait for one to finish before you can start the other.

My question for you is... are you *sure* you need to single-line this, or automate this? Since this seems to be a one-and-off situation? You're hardly going to do this more than once per user. Is that even a good idea?
posted by kschang at 8:17 AM on June 19, 2023

The canonical way to do a single-line copy-paste install from a Readme file involves working up a solid, robust, well tested installer script, sticking it somewhere on a web server you control, then having your users paste something that looks like this into a terminal window on Linux or OS X:

bash -c "$(curl -fsSL 'https://server.that.geoff.controls/linux-installer.sh')"

or something like this into a PowerShell window on Windows:

iex (iwr 'https://server.that.geoff.controls/windows-installer.ps1').Content

posted by flabdablet at 9:33 AM on June 19, 2023