What is my bandwidth going to?

March 21, 2006 6:27 AM

Host Network Monitoring: I have a colo box. I can watch traffic levels on it using mrtg. But how do I find out what is causing the traffic I see on the pretty graphs?

I have my own machine on colocation. I currently run mrtg graphs so I can watch my bandwidth and can see apache requests and throughput, amongst other things.

Sometimes, I see big spikes in my bandwidth usage. It isn't that big I guess, but it is large compared to normal traffic levels and stays consistently large for long periods of time (a couple hours usually). I'm pretty sure it isn't anything to worry about and I'm not near my bandwidth limits or anything. I'm just curious really what it is!

I use awstats to monitor my web serving and I can never find anything to correlate in awstats to that increase in traffic. I also use awstats to monitor mail traffic. The machine also has ssh/sftp, pop3s, and imaps that it is serving, but I don't really know how to monitor those more closely.

Is there any tool that I can use to answer the question "What caused this big prolonged uptick in my bandwidth from 11pm to 1am last night?" (preferrably without putting my NIC into promiscuous mode, as that really screws with my mrtg bandwidth graph)

posted by cmm to computers & internet (7 comments total)

Are you using Apache webserver... and what OS are we talking about?
posted by Witty at 6:36 AM on March 21, 2006

The apache server log.
posted by orthogonality at 6:36 AM on March 21, 2006

Ooops. :) I read it twice too. I was going to suggest turning on server-status in the httpd.conf file. Turn on the ExtendedStatus and limit access by IP address.

SetHandler server-status
Order deny,allow
Deny from all
Allow from ip.add.re.ss

Then you can access it through a browser http://ip.add.re.ss/server-status... which will give a snap shot of what's going on at that very moment.
posted by Witty at 6:43 AM on March 21, 2006

You can use either tcpdump or tcpflow to examine the traffic as it occurs, or to log traffic to file for later examination. Both support the command line option "-p", which tells either tool to avoid putting the network interface into promiscuous mode. You can examing the output directly, or, if you wish, Ethereal's GUI can parse and display dumps generated by tcpdump.

I usually find it more convenient to invoke the tools interactively while the traffic I wish to examine is taking place, rather than trying to create a multi-hour long exhaustive dump for later examination.
posted by RichardP at 6:43 AM on March 21, 2006

It's a linux box running apache, openssh, courier-imaps/pop3s, and postfix. Those are the only externally available services.

I use apache server-status to get requests per hour and k per hour to graph with mrtg. I'd prefer not to have to read the raw apache logs. awstats does an ok job of summarizing, but I do wish it could do more time-based reporting, but rolling up by IP. So like "in the past hour" bandwidth breakdowns per IP. But I'm fairly sure the extra traffic isn't web traffic since I don't see any bumps in my mrtg requests per hour and k per hour graphs, but there is a bump in my eth0 graph.

I've used tethereal and tcpdump a bit, but for the most part I don't catch this when it is happening. Usually it is the day after.
posted by cmm at 6:53 AM on March 21, 2006

I've used tethereal and tcpdump a bit, but for the most part I don't catch this when it is happening. Usually it is the day after.

You can do a 24 hour dump with tcpdump, and then slice and dice it later using tcpdump's filtering features or your preferred analysis tool (ethereal if you wish). Be prepared for the file to be BIG, even if you're only capturing the headers.

If the event is very infrequent (perhaps once a week) and you usually only notice a day later, set up a script that repeatedly invokes tcpdump passing it the -c argument to tell it to exit after a reaching a fixed number of packets. Each time through the loop increment the name of the dump file and search the dump directory for files that are older than 24 hours and delete them. This way you'll always have a series of dump files that go back 24 hours that you can examine.
posted by RichardP at 7:26 AM on March 21, 2006

I'd also recommend ntop. I've found it an great tool for determing the traffic flow on a remote server. It runs as a service and captures all traffic, you can log into with a web browser. Lots of nifty graphs and charts.
posted by lfaren at 7:37 AM on March 21, 2006