Sniffing HTTP traffic

March 30, 2005 12:09 PM

What is the easiest way, in Windows, to sniff the HTTP (and HTTPS) requests and responses back from the server for an application that won't let you tell it to use a proxy?

I have an application that makes HTTP requests and I'd like to find out what they are and what the responses are.

Normally, I'd run HTTPSniffer, a Perl application which acts as a proxy. I'd change the target applications proxy settings to 127.0.0.1:8080 and then all requests and responses would get passed back and forth through this application.

However this target application doesn't allow you to set the proxy and attempts to connect directly to the internet.

This has caused problems. I can't create a proxy locally because when I set Windows (dial-up) proxy settings, it catches the request from the application, passes it to my proxy which promptly attempts to make a connection to the outside world (to pass it on) only to have its own connection caught by Windows and passed back to itself. A loop which means that it ends up crashing.

I've tried setting the Windows proxy to 10.6.26.1 (my network IP) instead of 127.0.0.1 and making the Windows proxy settings ignore connections to the other, but that doesn't work. I've tried it the other way around too, but to no avail.

Is there some simple (and free) application I can install that can do this all easily for me? Or do you know of some way I can find out what this application sends and receives?

posted by ralawrence to Computers & Internet (9 answers total)

Instead of a network-level sniffer, would LiveHTTPHeaders work? It'll show you all the headers from all the requests in real-time.
posted by revgeorge at 12:39 PM on March 30, 2005

You can install Ethereal, which will capture all your packets passing through your network card, HTTP and otherwise.
posted by Jairus at 12:41 PM on March 30, 2005

Ethereal rocks. One of my favorite programs.
posted by sonofsamiam at 12:52 PM on March 30, 2005

Would it make sense to use the IIS (or, Apache) logfiles? At least they are handled at the filesystem level and can be processed (if what you are doing could be resource intensive) on another machine. The only problem that I can see is that it wouldn't exactly be realtime. Otherwise, try the TrafMeter full version. It does a whole lot of stuff like Ethereal but, has an easier to use HTTP header monitoring tool.
posted by Dean_Paxton at 2:16 PM on March 30, 2005

Ethereal will work... or, if you just need something basic, you can just use windump. It's a port of tcpdump, which you might know from the unix world.

Along the same lines as revgeorge's suggestion, microsoft has its own http debugger now: the fiddler. For folks using IE, it ain't bad, even if it's not exactly what you're looking for.
posted by ph00dz at 2:17 PM on March 30, 2005

Ooh, nice one there with the Fiddler. I'm going to have to try that. I've used ieHTTPHeaders, which is like LiveHTTPHeaders, but for IE.
posted by RikiTikiTavi at 2:45 PM on March 30, 2005

I'm guessing that as you said "application", rather than "browser", none of the browser-based solutions are going to cut it.

Getting the HTTP data is easy, ethereal or any other sniffer will do. Getting HTTPS directly is extremely difficult. Your app is either doing the encryption itself or passing it off to the IE DLLs to do it. Either way, unless you want to use an interactive debugger like Softice, I don't see how you're getting it before it leaves the host.

The last time I had to do this (get HTTP out of HTTPS), I identified the IP of the remote host that the server was connecting to and created a test host on my network with that IP (you'll need to set up the routing correctly). After testing that I could ping the IP, I ran a test SSL server (using stunnel I believe) that would decrypt the SSL and show me the HTTP traffic. This was then sent it on to the real server, after re-SSL'ing. There was a little perl glue to take the output from the server stunnel coming from the app and feed it back into a client stunnel connecting to the real server.

In order to trick the application into allowing the server-side certificate of my dummy SSL server, I created a CA cert using OpenSSL and used that to sign my dummy SSL server cert. I then added the CA cert to the Windows box's Trusted CA certs list, and it all worked nicely.

The whole process took a few hours by someone very comfortable with low-level networking, perl and certificates.

So, to summarise, if they're using HTTPS, the solution is non-trivial.

If you're not actually sure what your application is doing, start with the good folk at Sysinternals, grab Process Explorer and TCPView and see what the app is trying to do.
posted by quiet at 3:31 PM on March 30, 2005

I am not sure how it works, but HTTPLook worked for me when I was looking to capture HTTP traffic. It would capture both IE and Mozilla request/response packets without having to change the settings on either one, so I imagine it should work for any standard Windows application.
posted by garth at 7:31 PM on March 30, 2005

Justin Frankel's assniffer, which relies on packet capture. From the page: assniffer can monitor (using winpcap or pcap) a network, and for every HTTP transfer it sees, save a copy of the transferred data.

Ethereal is nice, too.
posted by gentle at 11:41 AM on March 31, 2005