Logrotate cronjob fails but running at command-line works.

March 21, 2009 11:32 PM

Why does my logrotate.conf file work correctly when logged in as root, but fail when executed as part of cron.daily?

I have a few custom application logs that I want to rotate automatically using logrotate. I picked as a base template, the logrotate.conf file that gets installed for Apache:

/var/log/httpd/*log {
    missingok
    notifempty
    sharedscripts
    postrotate
        /sbin/service httpd reload > /dev/null 2>/dev/null || true
    endscript
}

Here are two logrotate.conf entries that I created:

/var/log/hellavcr.log {
        ifempty
        missingok
        nomail
        weekly
        rotate 1
}

/var/log/hellanzbinit.log {
        ifempty
        missingok
        nomail
        daily
        rotate 2
}

If I sudo as root and execute "/usr/sbin/logrotate hellanzb" the command executes without any errors. However, every time the cronjob "/etc/cron.daily/logrotate" runs I get a bunch of error messages:

sh: line 2: missingok: command not found
sh: line 3: nomail: command not found
sh: line 4: weekly: command not found
sh: line 5: rotate: command not found
sh: line 2: ifempty: command not found
sh: line 3: missingok: command not found
sh: line 4: nomail: command not found
sh: line 5: daily: command not found
sh: line 6: rotate: command not found

I've checked the permissions for both the logrotate.conf files and the actual log files and they seems to be identical as far as my applications and Apache goes - so why the errors?

OS Info - CentOS 5, 64-bit running on a VPS.

Please help!

posted by your mildly obsessive average geek to computers & internet (17 answers total)

This may be obvious, but it seems like the conf file is being treated as if it were the executable. What is the command line that cron is using?
posted by idiopath at 11:42 PM on March 21, 2009

I agree with idiopath; that's exactly the output you'd get if the shell is trying to exec the logrotate.conf file.
posted by caphector at 11:51 PM on March 21, 2009

idiopath, caphector: This is the script that is being executed by cron:

#!/bin/sh

/usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0

logrotate.conf has the following permissions:

-rw-r--r-- 1 root root 520 Dec  1  2007 /etc/logrotate.conf

The additional configuration files under /etc/logrotate.d/ have the following permissions:

-rw-r--r-- 1 root root   75 Feb 12 06:10 hellanzb
-rw-r--r-- 1 root root   72 Feb 12 06:10 hellavcr
-rw-r--r-- 1 root root  167 Nov 12 10:43 httpd

posted by your mildly obsessive average geek at 12:03 AM on March 22, 2009

What is the contents of /etc/logratotate.conf? A typo there could be doing this.
posted by idiopath at 12:13 AM on March 22, 2009

idiopath: Here's what logrotate.conf looks like:

# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp -- we'll rotate them here
/var/log/wtmp {
    monthly
    minsize 1M
    create 0664 root utmp
    rotate 1
}

# system-specific logs may be also be configured here.

posted by your mildly obsessive average geek at 12:38 AM on March 22, 2009

Are hellanzb, hellavcr, and httpd the only files in /etc/logrotate.d/?

If not, check the files before hellanzb (alphabetically) and make sure that there's an "endscript" line for each "prerotate", "postrotate", "firstaction" and "lastaction" line. These errors could be caused by a missing "endscript" in an earlier include file, making logrotate try to execute everything that comes after as part of the unterminated script.

If those are the only three files in logrotate.d/, then, yeah, that's a stumper.
posted by hades at 4:04 AM on March 22, 2009

To expand on that, logrotate can execute arbitrary commands during various stages of the process. prerotate, postrotate, firstaction and lastaction are all lines which indicate the beginning of a set of commands to run during their respective stage. "endscript" is what tells logrotate that the commands to execute are done, and it should start treating the following lines as configuration options again. If you're missing an "endscript", any configuration options will be treated as if they're part of the most recent set of commands, even if that list started in another include file.

You could probably solve this problem by adding the line "endscript" to the very top of your hellanzb file, but it would be better to track down where it's coming from in the first place, and fix it there.
posted by hades at 4:10 AM on March 22, 2009

If it works while for you from a command line, but not from cron, I'd check for pathing problems. For cron jobs, you need explicit paths for all commands, but when you're logged in, you probably have some paths defined that allow commands to work without explicit paths in your scripts and conf files.
posted by ursus_comiter at 6:29 AM on March 22, 2009 [1 favorite]

Following up on what ursus_comiter said, I would try running /usr/bin/printenv from both the command line, and from within cron, and comparing the two, as it could easily be pathing or other environment variable issues.
posted by fings at 7:28 AM on March 22, 2009

Thirding path or environment.
posted by intermod at 7:48 AM on March 22, 2009

what happens when you run:

/usr/sbin/logrotate /etc/logrotate.conf

directly as root?

I'm wondering if there are errors somewhere else (aside from your conf files).
posted by alikins at 9:40 AM on March 22, 2009

Whoa, lots of answers but so far not too much luck :( . I'll try to reply one by one:

hades: This is the output from ls-l /etc/logrotate.d

hellanzb
hellavcr
httpd
named
proftpd
rkhunter
rpm
rtorrent
syslog
yum

As you can see, hellanzb is the first listing in the folder so it's unlikely there's a conflict.

ursus_comiter, fings, intermod: I don't know enough about CentOS to be certain but I *believe* this particular cronjob runs under the account "nobody". It certainly doesn't run under root, as it isn't in the crontab -l listing for root and there is no user called "cron" in /etc/passwds.

Now, the "nobody" account shell is set /sbin/nologin which means I can't su in to check the environment. I could probably modify the account to a regular shell, but then I wouldn't be replicating the problem exactly.

I could be completely wrong about the account details, but this is my (vague) understanding.

alikins: If I run that command as root, no errors at all - process exits cleanly.

I'm pretty sure I'm failing to state something obvious which could resolve this problem in a jiffy :(
posted by your mildly obsessive average geek at 10:32 PM on March 22, 2009

Huh. Well, I don't have an answer, but I do have an observation:

sh: line 2: missingok: command not found

"missingok" isn't line two of the hellanzb file, it's line three. The errors for that file appear to be one line off, as if the "ifempty" line weren't there at all. I don't know if that's relevant or not, but it's odd.

The cron job is almost certainly running as root, out of /etc/crontab. So the fact that running it as root from the command line works, but it running as root from cron doesn't--that's interesting.
posted by hades at 10:57 PM on March 22, 2009

hades: I just checked the man page and discovered that ifempty *is* the default, so I'm going to remove that entry to see if helps things. Just for reference, here's the entire error log from the cron job:

/etc/cron.daily/logrotate:

sh: line 2: missingok: command not found
sh: line 3: nomail: command not found
sh: line 4: weekly: command not found
sh: line 5: rotate: command not found
sh: line 6: postrotate: command not found
sh: line 2: missingok: command not found
sh: line 3: nomail: command not found
sh: line 4: weekly: command not found
sh: line 5: rotate: command not found
sh: line 6: postrotate: command not found
sh: line 2: ifempty: command not found
sh: line 3: missingok: command not found
sh: line 4: nomail: command not found
sh: line 5: daily: command not found
sh: line 6: rotate: command not found
sh: line 7: postrotate: command not found
sh: line 2: ifempty: command not found
sh: line 3: missingok: command not found
sh: line 4: nomail: command not found
sh: line 5: weekly: command not found
sh: line 6: rotate: command not found
sh: line 7: postrotate: command not found
posted by your mildly obsessive average geek at 11:12 PM on March 22, 2009

Ok, so you're getting errors from at least four config files, not just (or even necessarily) from the hella* files. I still think it could an unterminated prerotate/postrotate line. If you haven't yet, check every file in logrotate.d to make sure that there are an equal number of pre/postrotate lines and endscript lines.
posted by hades at 12:23 AM on March 23, 2009

hades: Ah! Light finally dawns. I was missing a endscript entry for the files where I had a prerotate entry. In other words I was doing:

/var/log/rtorrentInit.log {
	prerotate
		/sbin/service rtorrent stop > /dev/null 2>/dev/null || true

missing an endscript here!

	ifempty
	missingok
	nomail
	daily
	rotate 2
	postrotate
		/sbin/service rtorrent start > /dev/null 2>/dev/null || true
	endscript
}

I knew that posting the full error message would be useful for something!
posted by your mildly obsessive average geek at 12:46 AM on March 23, 2009

Sweet! I'm glad to hear it.
posted by hades at 12:54 AM on March 23, 2009