L2TP, You're Killing Me

July 25, 2007 8:31 AM

Why can't I connect to an L2TP VPN on my Mac?

I'm trying to set myself up to work from home, and am failing miserably. My office runs an L2TP VPN on Microsoft IIS. I would like to connect to it with my Mac, which runs 10.4.10. When I attempt to connect, however, after about a minute of nothing I get the message: "The server did not respond. Please verify the server address and try again."

The Internet Connect Connection Log says merely:
L2TP connecting to server 'servername' (X.X.X.X)...
L2TP cannot connect to the server

My understanding is that L2TP is handled by racoon under the hood, but I haven't been able to find any trace of /var/log/racoon.log (which presumably could get me more detailed information). The entries in system.log are identical to the Internet Connect Connection Log.

Configuration:

Given the paltry of options provided by Apple in Internet Connect, I think I've got everything configured per the instructions from my office. User authentication is via password. Machine authentication is via certificate.

Network:

I have failed to connect from behind a NAT, behind a NAT with ports 500, 4500 and 1701 forwarded, behind a NAT but DMZ'd, and just plain out on the internet. My system firewall is turned off, and although I run Little Snitch I've got it turned off for testing purposes. The VPN server isn't behind any other devices.

For the especially technical among you, I have included the following tcpdump, which includes all of the packets going between the two machines during a connection attempt:

IP tsmo.500 > vpnserver.500: isakmp: phase 1 I ident
IP vpnserver.500 > tsmo.500: isakmp: phase 1 R ident
IP tsmo.500 > vpnserver.500: isakmp: phase 1 I ident
IP vpnserver.500 > tsmo.500: isakmp: phase 1 R ident
IP tsmo.4500 > vpnserver.4500: NONESP-encap: isakmp: phase 1 I ident[E]
IP tsmo > vpnserver: udp
IP vpnserver.4500 > tsmo.4500: NONESP-encap: isakmp: phase 1 R ident[E]
IP vpnserver > tsmo: udp
IP tsmo.4500 > vpnserver.4500: NONESP-encap: isakmp: phase 2/others I inf[E]
IP vpnserver.4500 > tsmo.4500: NONESP-encap: isakmp: phase 1 R ident[E]
IP vpnserver > tsmo: udp
IP vpnserver.4500 > tsmo.4500: NONESP-encap: isakmp: phase 1 R ident[E]
IP vpnserver > tsmo: udp
IP tsmo.4500 > vpnserver.4500: NONESP-encap: isakmp: phase 1 I ident[E]
IP tsmo > vpnserver: udp
IP vpnserver.4500 > tsmo.4500: NONESP-encap: isakmp: phase 1 R ident[E]
IP vpnserver > tsmo: udp
IP tsmo.4500 > vpnserver.4500: NONESP-encap: isakmp: phase 1 I ident[E]
IP tsmo > vpnserver: udp
IP tsmo.4500 > vpnserver.4500: NONESP-encap: isakmp: phase 1 I ident[E]
IP tsmo > vpnserver: udp
IP vpnserver.4500 > tsmo.4500: NONESP-encap: isakmp: phase 2/others ? inf[E]
IP tsmo.4500 > vpnserver.4500: NONESP-encap: isakmp: phase 1 I ident[E]
IP tsmo > vpnserver: udp
IP vpnserver.4500 > tsmo.4500: NONESP-encap: isakmp: phase 1 R ident[E]
IP vpnserver > tsmo: udp
IP tsmo.4500 > vpnserver.4500: NONESP-encap: isakmp: phase 1 I ident[E]
IP tsmo > vpnserver: udp
IP tsmo.4500 > vpnserver.4500: NONESP-encap: isakmp: phase 1 I ident[E]
IP tsmo > vpnserver: udp
IP tsmo.4500 > vpnserver.4500: NONESP-encap: isakmp: phase 1 I ident[E]
IP tsmo > vpnserver: udp
IP tsmo.4500 > vpnserver.4500: NONESP-encap: isakmp: phase 1 I ident[E]
IP tsmo > vpnserver: udp
IP tsmo.4500 > vpnserver.4500: NONESP-encap: isakmp: phase 1 I ident[E]
IP tsmo > vpnserver: udp
IP tsmo.4500 > vpnserver.4500: NONESP-encap: isakmp: phase 1 I ident[E]
IP tsmo > vpnserver: udp
IP vpnserver.4500 > tsmo.4500: NONESP-encap: isakmp: phase 1 R ident[E]
IP vpnserver > tsmo: udp
IP vpnserver.4500 > tsmo.4500: NONESP-encap: isakmp: phase 2/others R inf[E]

Lil help?

posted by tsmo to computers & internet (6 comments total)

Unfortunately, I don't have my Mac machine here with me to give you details (I can check back tonight if no one has posted the solution) but if you're poking around on Google trying to solve this, it's Microsoft ISA software that handles VPN services, not IIS (that their webserver).

Also, try using PPTP not L2TP.
posted by purephase at 11:09 AM on July 25, 2007

Thanks for the correction on ISA vs IIS, purephase.

And regarding PPTP: I was able to convince the office sysadmin to turn on PPTP momentarily so I could test it, and it works. But he doesn't want to leave it on all the time because it's supposedly not secure enough.
posted by tsmo at 11:11 AM on July 25, 2007

It should be noted that the above tcpdump is from behind NAT with ports forwarded. Outside of NAT, all the activity is on port 500, and it never sets up the NONESP-encap. Definitely confuses me that Internet Connect is failing with what looks like a generic timeout error when there are a bunch of packets coming through...
posted by tsmo at 11:45 AM on July 25, 2007

PPTP security is based on the strength of the password. If you have strong passwords where you work, you should be fine leaving it enabled.

However, with L2TP over IPSec you'll need to import a certificate into your keychain to support the initial handshake that will setup the secure tunnel for credential passing. Have you set all that up?

If you're authenticating to AD, then your user authentication should be Kerberos.

We use PPTP but I was able to connect to ISA 2004 with L2TP over IPSec once I did all the above. This is on OS 10.4.10.
posted by purephase at 3:16 PM on July 25, 2007

Thanks for taking the time to dig into this, purephase.

Yes, I've got the machine certificate and the certificate authority installed into the system keychain and X509Anchors respectively, which I think is how things are supposed to be. I'm not using Kerberos though; my admin said I should log in with my usual system login. (Update: just tried Kerberos. No dice.)

I think there's also a chance my admin has locked things down too tight for MacOS to get in: authentication is restricted to MS CHAP V2, encryption is restricted to ESP 3DES and compression is MPPC. Presumably if MacOS didn't support one of those protocols I'd be up the proverbial creek.

One thing I'd like to emphasize from my original post is how distressing it is to not be able to find racoon.log, or its contents. I'm getting awfully jealous of linux users pulling all kinds of helpful error messages from racoon.log, when all I've got is a generic timeout that contradicts what I'm seeing in the tcpdump.

Perhaps I can convince my admin to enable PPTP with strong passwords...
posted by tsmo at 3:58 PM on July 25, 2007

Well, the major issue he is probably trying to protect against is that, with simple passwords, the hash generated during authentication can be easily cracked (there is software that will do it quite well) if the passwords are shorter than 8 (or 9 -- can't remember) characters. If you have a password policy that enforces a minimum length of 8 characters with strong passwords enabled (in AD this means: 1 special characters, 1 upper case, 1 numeric, and 1 lower case -- 3 of the 4 mentioned) the likelihood of anyone cracking a PPTP password hash is incredibly unlikely.

PPTP and L2TP can use MS CHAP V2 (I believe it's the default for PPTP and ISA 2004 and 2006) so Mac OSX will work that type of authentication.

One other thing to mention is that L2TP sometimes has difficulty connecting through a NAT'ed environment. If you're using a wireless router at home and NAT'ing your workstations you might want to try taking the router out of the equation or see if there are any configuration options on the router about passing L2TP properly.

This should have been addressed already (not sure) but it's possible that OSX is still experiencing the setback.
posted by purephase at 7:00 PM on July 25, 2007