I has a pain in my linux logging daemon

December 13, 2008 3:58 PM

Calling all *nix sysadmins in the hive mind ... a bunch of Unix and Linux logging/syslog/syslog-ng questions inside.

Here's the setup. Syslog-ng 1.6+ on a bunch of SuSE 10.3 and 11.0 / BSD / RHEL 4.x machines set up to stream via tcp to a central machine. Central machine is, at the moment, set up to accept anything coming from machine X into /var/log/central/(machine name)/(facility)-(tag).log unless it matches another rule (for instance, auth and authpriv facilities get combined into one log with a different name.)

There's several questions blobbed in here with a lot of technical details. All of this is being done to meet requirements handed to us by auditors.

The first of those audit requirements is to log when a user accesses a resource by authenticating correctly, and to log when an attempted access is made at both the system, application engine (i.e. apache, mysql, tomcat) and application (i.e. the php, java code) levels. This is completely under control for system level accesses; apache is another matter since we use mod_auth_mysql heavily, but I'm working on that. (And as far as I can tell, it's not even possible with MySQL.) Any suggestions?

The second issue is with logs like php application logs (which are just appended to a file) mysql-error.log, mysql-slow.log, apache access logs, etc -- none of that even goes through syslog in the first place (with the notable exception of apache error logs, which can go to local7:info), so to get it into syslog as a transport layer I have a bunch of lazy man's daemons ... really, shell scripts running a "tail -f | /bin/logger.sh -t (sourcename) &" ... obviously a messy proposition that could cause system load to spiral out of control. Syslog-ng can "follow" a named pipe (which is pretty much what logger.sh does -- route any input to syslog-ng through the pre-existing named pipe), but cannot follow a file that's appended to. The pipe through logger.sh is required because I need to tag the log entry with the source so that we know where it came from.

On one of our systems, which runs a couple of applications plus nagios, I have about ten of these damned things running. With this setup, I dread getting an app hammered on that system because I can see the load average shooting through the stratosphere as all those nice slow shell scripts joust for I/O.

Now, why syslog? Why not just rotate the logs? Well, part of it's for security. I work for a big U. One of the obvious plusses of logging to a remote write-forward-only machine is that you give a cracker TWO systems that they have to remove traces of their nefarious work on. That means that updates to system logs need to be streamed 'live' over the network, not cached on the local machine and then FTP'd over at night when logrotate runs. I swear that these auditors were hired by Catbert.

I can't be the only one with the requirement to reinvent the wheel here. There's gotta be a better solution. Right? Oh, god, please tell me I'm right!

Seriously, any suggestions on how to meet this would be appreciated. Other than just "it's gotta be built out of somewhat stock distributed tools" (In other words, no, I don't want to implement an entirely custom logging solution, I can buy one that will be just as tough to maintain) I don't have any constraints or requirements -- disk space, CPU usage, network bandwidth, etc. are all pretty much moot. I just really don't want to have to reinvent the wheel and it feels like I am... which to me usually a huge sign that "ur doin' it wrong."

posted by SpecialK to Computers & Internet (4 answers total)

Instead of having Apache write to access log files and then reading them with shell scripts, you can have apache log directly to a log-handler process with CustomLog "| log-handler.pl". This has a couple of benefits - one log-handler process handles all the Apache processes, and Apache will restart it if it abends.

vlogger is a log-handler made to break logs out by virtual host - normally to log files, but it's just Perl so it should be easy to make it write either to local syslog (with Unix::Syslog) or directly to your syslog-ng boxes (with Sys::Syslog).

And can't PHP log to syslog?

error_log string
Name of the file where script errors should be logged. The file should be writable by the web server's user. If the special value syslog is used, the errors are sent to the system logger instead.
posted by nicwolff at 5:28 PM on December 13, 2008

Having you looked at Splunk? It can pull it data via syslog, local FIFOs, network sockets, files, and more. The free version can handle up to 500MB of data/day.
posted by larsks at 5:48 PM on December 13, 2008

Your logging requirements are easily handled with syslog and small amounts of configuration for each source.

You can standardize on syslog for all apps that support it, and standard IO piped to the logger command for anything else. The logger command comes with syslog and sends its standard input to syslog using whatever facility and tags you specify.
There is no performance penalty for piping things to logger, whether or not it is wrapped in a shell script; the shell script won't be doing IO.

Your service supervision system (init.d, or if you don't have one yet, I recommend runit) just has to start a pipe comprising the command and the logger. If you use runit, this is better achieved by putting:

#!/bin/sh
# $COMMAND does not fork, and writes to stdout
$COMMAND

into /etc/sv/$SERVICE/run,

#!/bin/sh
/usr/bin/logger -t $SERVICE

into /etc/sv/$SERVICE/log/run, and making both files executable.
posted by Tobu at 7:27 AM on December 14, 2008

Wow that markup was ugly, and I forgot to use exec.

Put
#!/bin/sh
# $COMMAND does not fork, and writes to stdout
exec $COMMAND
into /etc/sv/$SERVICE/run,

#!/bin/sh
exec /usr/bin/logger -t $SERVICE
into /etc/sv/$SERVICE/log/run, and make both files executable.
posted by Tobu at 7:33 AM on December 14, 2008