Analyze server logs for site visitors?
April 18, 2017 8:50 AM Subscribe
I have a self-hosted WordPress site. I have access to the server logs and can see the gzipped files going back to when the site was launched. What tool or procedure can I use to analyze these server logs? I am most interested in site visitor information. Ideally, the information would be easy to send to another party as well.
Are you sure this wouldn't be better solved with something like Google Analytics? That would allow you to create reports and collaborate with other people.
posted by primethyme at 10:24 AM on April 18, 2017
posted by primethyme at 10:24 AM on April 18, 2017
primethyme is right going forward you should use Google Analytics, but you can't import old logs into Google Analytics...
posted by gregr at 11:40 AM on April 18, 2017
posted by gregr at 11:40 AM on April 18, 2017
Response by poster: Yes, I need to process and analyze these logs, not future ones.
There are web stat tools on the server, but they were not enabled when the site was created.
Does anyone have a tool they can recommend that can import "old" logs and analyze them? Preferably for Mac. Happy to go a direct Unix route too, if that works.
posted by jammy at 11:57 AM on April 18, 2017
There are web stat tools on the server, but they were not enabled when the site was created.
Does anyone have a tool they can recommend that can import "old" logs and analyze them? Preferably for Mac. Happy to go a direct Unix route too, if that works.
posted by jammy at 11:57 AM on April 18, 2017
What information are you attempting to obtain? My understanding is that Wordpress doesn't log very much by default, so unless you have a logging plugin enabled, there's not going to be a lot of granular information. Your logs may not contain things like the referrer unless that was modified from the default.
The afore mentioned AWStats or Webalizer can ingest your logs and give you a variety of information about them, like what IP addresses accessed parts of the site most often and guesses on their geolocation.
It's probably overkill for what you're thinking of but if you need to run sophisticated queries on a lot of data, you can use the ELK stack. It can be a pain to get going though.
posted by Candleman at 12:20 PM on April 18, 2017
The afore mentioned AWStats or Webalizer can ingest your logs and give you a variety of information about them, like what IP addresses accessed parts of the site most often and guesses on their geolocation.
It's probably overkill for what you're thinking of but if you need to run sophisticated queries on a lot of data, you can use the ELK stack. It can be a pain to get going though.
posted by Candleman at 12:20 PM on April 18, 2017
The analog script will spit out quite a bit of useful information in the form of html pages. If you’re self hosting Wordpress you could probably just install it & run it over the logs. It’s packaged in Debian et al.
posted by pharm at 1:17 PM on April 18, 2017
posted by pharm at 1:17 PM on April 18, 2017
Use AWStats.
posted by humboldt32 at 1:46 PM on April 18, 2017
posted by humboldt32 at 1:46 PM on April 18, 2017
Response by poster: What information are you attempting to obtain?
As noted in the original question: "I am most interested in site visitor information."
Use AWStats.
There does not seem to be an option with AWStats to import old logs. If there is, please advise as to the methodology. As noted in my comment just above: "I need to process and analyze these logs, not future ones."
posted by jammy at 3:12 PM on April 18, 2017
As noted in the original question: "I am most interested in site visitor information."
Use AWStats.
There does not seem to be an option with AWStats to import old logs. If there is, please advise as to the methodology. As noted in my comment just above: "I need to process and analyze these logs, not future ones."
posted by jammy at 3:12 PM on April 18, 2017
I believe I understand your question. You have logs, old logs, and want to get useful historical data about your site from them.
How big are the logs, what format are they in, and what information do you want from them? Do you need this to integrate with your current site stats? Do you know exactly what types of queries you want to run, and it is not too many? Could you get away with command line text processing utilities + Excel?
posted by batter_my_heart at 4:13 PM on April 18, 2017 [1 favorite]
How big are the logs, what format are they in, and what information do you want from them? Do you need this to integrate with your current site stats? Do you know exactly what types of queries you want to run, and it is not too many? Could you get away with command line text processing utilities + Excel?
posted by batter_my_heart at 4:13 PM on April 18, 2017 [1 favorite]
As noted in the original question: "I am most interested in site visitor information."
That is a vague enough statement as to be close enough to meaningless. Seeing as you appear to be a web designer, imagine a client tells you that they want a web site but no further information and when asked for clarification just angrily repeats that they want a web site.
posted by Candleman at 7:24 PM on April 18, 2017 [1 favorite]
That is a vague enough statement as to be close enough to meaningless. Seeing as you appear to be a web designer, imagine a client tells you that they want a web site but no further information and when asked for clarification just angrily repeats that they want a web site.
posted by Candleman at 7:24 PM on April 18, 2017 [1 favorite]
Response by poster: I believe I understand your question. You have logs, old logs, and want to get useful historical data about your site from them. How big are the logs, what format are they in, and what information do you want from them? Do you need this to integrate with your current site stats? Do you know exactly what types of queries you want to run, and it is not too many? Could you get away with command line text processing utilities + Excel?
Yes, batter_my_heart, that is correct. The logs are in gzip format, organized by date, about a dozen months worth. When unzipped, most logs appear to be less than 20MB.
I am looking for as much site visitor information as I can get - if I had to prioritize, it would probably be for overall number of unique site visitors, IP location, and links they followed on the site. After that, browser info would be great. I don't really need information concerning bot activity at the moment.
I do not need it integrated with current site stats. I have basic experience using UNIX. I do not have Excel.
--
AWstats can absolutely import old logs.
Thank you hanov3r for that link - I will check that out.
posted by jammy at 12:51 PM on April 19, 2017
Yes, batter_my_heart, that is correct. The logs are in gzip format, organized by date, about a dozen months worth. When unzipped, most logs appear to be less than 20MB.
I am looking for as much site visitor information as I can get - if I had to prioritize, it would probably be for overall number of unique site visitors, IP location, and links they followed on the site. After that, browser info would be great. I don't really need information concerning bot activity at the moment.
I do not need it integrated with current site stats. I have basic experience using UNIX. I do not have Excel.
--
AWstats can absolutely import old logs.
Thank you hanov3r for that link - I will check that out.
posted by jammy at 12:51 PM on April 19, 2017
« Older How do leftist anarchists engage with current... | Experiences to Have In Cologne, Germany Newer »
This thread is closed to new comments.
posted by COD at 9:22 AM on April 18, 2017 [1 favorite]