Looking for log file analyzer

November 30, 2005 8:40 AM

I'm looking for a website log file analyzer. This is not to display summaries or graphs, as in this thread, but to examine the activity of individual IPs and get a handle on what kind of hacking attempts, leeching, and other abuse might be going on. I have direct access to logs (something that can DL them would be nice, too). The $100+ software I found isn't an option.
posted by rolypolyman to Computers & Internet (9 answers total)

I think what you actually want is an Intrusion Detection system. If you are analyzing the log files after the event you could be too late. It doesn't have to cost a lot. Try this PDF for an intro to the subject: http://www.techsupportalert.com/pdf/t1523.pdf
posted by spock at 9:19 AM on November 30, 2005

What web server software are you using?

For IIS, I would recommend this TechNet article. For Apache, AWStats is generally considered one of the best out there.

Both solutions are free.
posted by purephase at 9:26 AM on November 30, 2005

Sorry, I should have gone through the earlier thread. I didn't realize that it mentioned AWStats.

Most intrusion detection systems are expensive (orders of magnitude higher than $100) so the log file analyzers mentioned in the other thread and my own above are probably you're best bet.

Could you explain why AWStats is not going to work for you? It is pretty extensive in it's reporting capabilities.
posted by purephase at 9:30 AM on November 30, 2005

Response by poster: Well, looking at the AWStats demo this is overkill and has only limited capability to track the actions of individual users. It does report bandwidth of various IPs, which is nice, but it doesn't even sort by bandwidth, which is kind of useless.

For what it's worth, I'm on Dreamhost and run Apache.
posted by rolypolyman at 9:54 AM on November 30, 2005

Response by poster: I've given some thought to what Spock said, and I think maybe I am barking up the wrong tree. Maybe I can diverge this question a bit.

Basically what happened is my homepage was defaced due to an exploit of xmlrpc.php, which is a third-party file packaged with WordPress. I would love a reference to a blog or resource dedicated to webmasters on hosted servers that warns of exploits like these they need to be aware of. A lot of the stuff I've seen is for corporate webmasters and techies who are running servers. Isn't there something for the rest of us?
posted by rolypolyman at 10:26 AM on November 30, 2005

An IDS will not help you with a hosted solution as it is a physical device that sits between the server and the internet.

Dreamhost has some pretty decent statistics available in the control panel. I haven't used them in a long time so I am not sure what is available and how in-depth the statistics are. As for other options, you could go through this Slashdot thread. A lot of different products are mentioned that would hopefully do what you're looking for.

Finally, upgrade WordPress (if you can). The latest version does not suffer the same vulnerability.
posted by purephase at 10:44 AM on November 30, 2005

Dreamhost uses Analog for its stats control panel. Analyzing a specific IP or other subjects of your log data isn't possible through their panel. Possibly Google Analytics (Urchin) would do what you need, though right now they're temporarily closed to new reigstrations.

Personally, I just grep. In seconds you can extract all the log entries for the suspect IP, then do whatever analysis you want on the subset.

my homepage was defaced due to an exploit of xmlrpc.php, which is a third-party file packaged with WordPress. I would love a reference to a blog or resource dedicated to webmasters on hosted servers that warns of exploits like these

An xlmrpc exploit warning has been running on the WP Dashboard. If you have a newsreader, you can pick up that feed direct instead of seeing it only at your blog's admin panel.
posted by nakedcodemonkey at 11:58 AM on November 30, 2005

Basically what happened is my homepage was defaced due to an exploit of xmlrpc.php, which is a third-party file packaged with WordPress. I would love a reference to a blog or resource dedicated to webmasters on hosted servers that warns of exploits like these they need to be aware of. A lot of the stuff I've seen is for corporate webmasters and techies who are running servers. Isn't there something for the rest of us?

Yes, in this case the Wordpress blog would have alerted you when the XMLRPC problems first started happening.

Every project like wordpress has some kind of announcement/updates mailing list or blog that will announce when security flaws are found or when bugfix releases are made. As a user of any kind of server-side software it is a good idea to subscribe to these announcement lists of every large package that you use.

There are also resources that aggregate all these security announcements, such as BugTraq or Full-Disclosure. However, they are much higher traffic because they deal with all sorts of topics. They are also geared towards more advanced users.

Really the best possible way for you to be secure is to simply make a list of all the scripts/applications you use and then find the annoucements/development mailing list for each.
posted by Rhomboid at 3:24 AM on December 1, 2005

I typically use Log Parser[1] from MS. There are articles [2] floating around on how to use it do conduct forensics.

Oh, and it's free.


[1] http://tinyurl.com/9epod
[2] http://tinyurl.com/2eokb
posted by GernBlandston at 4:54 AM on December 2, 2005