Question: I need a freeware that will scan an NCSA logfile

ZenMasterThis — Mon, 20 Dec 2004 08:36:47 -0800

I want a utility (freeware?) which will scan an NCSA extended logfile and create a simple report on which referrers generated how many sales transactions.

Any recommendations?

By: RustyBrooks

RustyBrooks — Mon, 20 Dec 2004 09:25:17 -0800

You'll probably have to write something for yourself. It might not be easy. Like, do the log files have something in them that will tell you when a sale was made?

By: weston

weston — Mon, 20 Dec 2004 09:59:29 -0800

I don't think there is anything standalone out there that does this, though I know some commerce packages have this built in, as I've written this feature for at least one of them.

High precision might be somewhat difficult, given that there's really nothing that goes into a standard extended log file that can completely reliably identify an individual. It shouldn't be too hard to do roughly, though. What you really want is to catch two ends of a click trail -- one that starts with a referer from a foreign domain, one that ends with the call of a specific resource (a page or a script) that signifies a completed sale. You can use the remote host/ip field somewhat reliably to identify an individuals requests.

A quick and dirty algorithm in something like perl would be to generate a hash of all requests from a given host/ip, with the key based on that host/ip. Make special note of host/ip's from which a request to the "sale complete" resource is made. Then, for each of those hosts, find the initial request and print the referer if any.

while($line = <LOGFILE>)
{
$line =~ m/$someRegExpThatParsesLogFiles/;
$host_or_ip = $1;
$request = $4; // or something like that
push $requestsByHost{$host_or_ip}, $line;
if($request =~ m/$saleResourceName/)
{ push @hostsThatBought, $host_or_ip; }
}

foreach $hostThatBought (@hostsThatBought)
{
@requestsFromHost = $requestsByHost{$hostThatBought};
foreach $requestFromHost (@requestsFromHost)
{
if($requestFromHost =~ m/$regExpThatGetsReferer/)
{ print $requestFromHost; }
}
}

This is ineffecient, imprecise, and has at least one reference problem. But it might give you a start.

By: grouse

grouse — Mon, 20 Dec 2004 10:07:59 -0800

Be sure you remember that you can't rely on referer to be reported accurately.

Comments on: I need a freeware that will scan an NCSA logfile

Question: I need a freeware that will scan an NCSA logfile

By: RustyBrooks

By: weston

By: grouse