Comments on: Please recommend an Apache log analysis?

Question: Please recommend an Apache log analysis?

Nelson — Tue, 20 Mar 2007 09:02:07 -0800

What open source package do you like for Apache logfile analysis?

I need to set up some significant usage reporting for a web application. I'm looking for a reporting application for Apache style logs that's robust and customizable. Not looking just for some little report of top URLs for the day, but something that goes into detail on user agents, page load times, IP addresses, etc. Something with a pluggable report system would be ideal because I need to do some app-specific custom reports.

Much of my traffic is API calls, not web browsers, so I think Javascript-based logging like Mint and server-hosted things like Google Analytics won't work. I need something that works primarily off of logs. Storing data in MySQL is ok, but I'd prefer just to work straight from log files.

What's good, flexible, and not ugly? I know about Webalizer, Analog, and AWStats, but in the past when I've used them they've all seemed a bit awkward and limited. It's been awhile, maybe they've improved? Or maybe there's an alternative?

By: SpecialK

SpecialK — Tue, 20 Mar 2007 09:08:39 -0800

I like AWstats. It's not the easiest to set up in a shared hosting environment, but it's a cakewalk if you're the server admin. Compared to Google Analytics, it's very thorough.

By: SpecialK

SpecialK — Tue, 20 Mar 2007 09:09:06 -0800

(Not sure about custom reports, but if you write Perl you can mod it to do anything.)

By: Remy

Remy — Tue, 20 Mar 2007 09:54:52 -0800

AWStats meets your needs. It certainly does custom reports (although you can't take that to every last degree).

One thing to note is that "page load times" is generally incalculable just based on server logs.

By: qvtqht

qvtqht — Tue, 20 Mar 2007 12:12:17 -0800

I came here to suggest Webalizer and Analog... Analog has really come a long way in the last 2-3 years.

By: Nelson

Nelson — Wed, 21 Mar 2007 19:36:58 -0800

Thanks for the replies. Looks like I haven't overlooked any options. AWstats seems the best of the lot, but its graphing is awfully hard to read.

By: lhl

lhl — Thu, 22 Mar 2007 01:19:31 -0800

Nelson, quite frankly, the open source log analysis packages out there all pretty much suck.

If can use commercial software, Sawmill is the best (affordable) log processing package I've found - it's reasonably priced, and allows for very extensive customization.

It's a little bit of a bear to get going, but is flexible enough to do just about everything you want, including customized scripting, and with the EE, MySQL queries (and on clusters, which was important at my last job where we were crunching very large log sets).

Once you have your data parsed, you have all sorts of visualization options at your disposal (jpgraph and php/swf charts are good for simple graphs, or lots of flash or processing options available for fancier infoviz).

As Remy mentioned, you're not going to be able to get page load times unless w/o beaconing.

I've always felt that it's too bad that there's never been a focused open source project on making better log analysis tool, but I guess it's one of those things that's both difficult enough and valuable enough that anyone who spends enough time to write something decent is bound to commercialize it (or it stays in-house)?