What's better than awstats?

September 19, 2007 8:03 AM

I've got a website that does a lot of traffic. I'm using awStats. It's slow as hell because we're adding ten million hits a day to it's database. We don't want to use an external logging solution because we've found that they underreport dramatically (by 40% in the case of Google Analytics). Are there any free stats processors that will be able to process lots of records AND have better drill-down capability than awstats? I have plenty of hardware to throw a tthe problem.
posted by SpecialK to Computers & Internet (8 answers total)

webalizer
posted by Baud at 8:58 AM on September 19, 2007

analog is about as fast as they get, but requires a bit of fiddling to be made pretty.
posted by jquinby at 8:59 AM on September 19, 2007

This: analog, these days it goes further than just web logs, but that is still the main focus. And it's really customizable, if you run a site that big, that's a big bonus in my book.

(Incidentally, who _hasn't_ seen an analog stats page, they used to be all the rage ever since web started gaining momentum... or am I just old?).
posted by phax at 9:05 AM on September 19, 2007

Bear in mind that much of the underreporting by external providers is really them filtering out spammers and robots, as well as excluding users without JavaScript.

Ten million hits a day is always going to require a fair bit of processing, especially if you want even more detail (which you won't get from faster options such as Webalizer and Analog). Have you already tried excluding images etc. to reduce the amount of data and perhaps speed things up a bit? You could customise Apache's logging to skip certain file types, or move static files to a subdomain to be logged separately.
posted by malevolent at 9:43 AM on September 19, 2007

Are you positive that Google Analytics was underreporting by that much? As malevolent notes, parsing logs tends to pull in a large amount of garbage, so it could be that your current numbers are artificially inflated.
posted by sad_otter at 11:25 AM on September 19, 2007

i just read about this startup yesterday over at techcrunch40 that claims to be a server based google analytics... check them out here.
posted by hummercash at 11:52 AM on September 19, 2007

The techcrunch link says "Users are able to place a snippet of javascript" so that solution would only monitor users with javascript as with google analytics.

I think that Google Analytics is probably reporting quite accurate numbers - a recentish article at Dreamhost's blog discussed the vast number of page views from bots. W3schools shows 94% of users having javascript enabled, so of those the only ones not being monitored would be those who block Google Analytics specifically with something like adblock.
posted by JonB at 1:26 PM on September 19, 2007

Response by poster: Let's just say it would be politically unfeasable to put my stock in a solution that would cause our pageviews/visitors to go down, even if they aren't real eyeballs.
posted by SpecialK at 2:27 PM on September 19, 2007