Advanced Network Routing

April 26, 2007 12:37 PM

Is it possible to configure a Mac OS X Server to route incoming and outgoing traffic via two different interfaces?

Here’s the problem: we have an Xserve providing various services to a small, company-wide network. Among other services, it handles DNS, DHCP, and NAT, and acts as the gateway through an asymmetric DSL connection to the Internet. We’ve recently begun noticing extremely high latency (over 1000ms,) with no appreciable degradation in download throughput (generally better than 3 Mib/s, as good as 6 Mib/s.) We seem to have finally diagnosed the source of the latency: our pitiful upstream is being saturated by larger uploads at just over 384 kib/s.

Is it possible, using ipfw/dummynet or some other tools, to filter packets destined to travel out on this interface, and reroute them through a secondary interface dedicated to uploading? Can return traffic still be delivered via the primary interface?

Any other possible solutions are also welcome.

posted by ijoshua to computers & internet (9 comments total)

you're talking about asymmetric routing, where you have one connection for inbound traffic and another for outbound. typically this is done further upstream from you, but there may be some BSD techniques/utils for this. I know there's a BGP daemon for BSD, but that's a bit beyond the scope of this issue.
posted by rhizome at 12:54 PM on April 26, 2007

If you're looking to change the interface that you send traffic out, then yes, absolutely. It's the easiest possible question you could ask in this realm, I think.

You want to set a new route for some network (probably "the entire network" -- 0.0.0.0/0 -- called the "default route" often) to go to the upstream router's address out your target interface.

$ man route
$ sudo netstat -nr
$ sudo route change -net default ${other_interface_upstream}

Note that your upstream may not expect traffic from network X to be coming from what it thinks is network Y. If they're unusually diligent and paranoid, they'll probably drop it. Most sane net admins wouldn't though.
posted by cmiller at 12:56 PM on April 26, 2007

(Note, I'm a Linux guy, so these BSD semantics on "route" may need prodding to be right. I didn't test that, above.)
posted by cmiller at 12:57 PM on April 26, 2007

Asymmetric routing is always painful. You want a network expert involved anytime you get into this, and you can run into just a metric assload of different possible problems. Many firewalls depend on seeing all the packets go by, and if some are going via one interface and some via another, they're easily confused. And that's just the tiniest possibility in a whole universe of potential pain and suffering. Dante would have been a big fan of asymmetric routing.

You don't have to do that at all. Your easiest solution: bring in another DSL line. Use that for your office internet and web browsing. Use something little, like a Linksys WRT54GL, as your gateway/firewall.

The way you'd configure this:

External1 -> Linksys -> Internal net, address 192.168.0.1
External2 -> XServe -> Internal net, address 192.168.0.2, but with NAT and forwarding disabled.

You set your default route to be 0.1, and all your internal web browsing goes out through the new link. Your uploads from the XServe go out the dedicated line it has for itself. You folks can still talk to your XServe on 0.2, and you can still talk to the Internet, and IT can still talk to the Internet, but you don't share upstream bandwidth.

This will be enormously easier than trying to route everything through the XServe. Trust me. It can probably be done, but you REALLY don't want to.
posted by Malor at 1:43 PM on April 26, 2007

Oh, note: if your problem is that someone internally is uploading big files and killing your bandwidth, this won't really fix that problem. To fix that, you'd need some kind of QoS solution -- packet prioritization. Asymmetric routing won't help with that at all.

The problem that the solution above solves: your XServe is sending big files down the same interface that you're trying to use for everything else. If that's not actually the problem, then adding the second line won't help much.
posted by Malor at 1:48 PM on April 26, 2007

Sheesh, I should try to get all my thoughts in one post. If your upload source is just one person or a couple of people, you could add the Linksys and just have those couple people using that line... set their default gateway to be the Linksys, and have everyone else pointed at the XServe.

If it's the whole office, you either need more upstream bandwidth or QoS.
posted by Malor at 1:50 PM on April 26, 2007

Malor, thanks for the suggestion, but the Xserve itself isn’t consuming much bandwidth on either side. The main consumers of upstream bandwidth are in one department of the company, although from time to time, any user could potentially saturate the upstream. Allocating a separate line to that department might be a possibility.
posted by ijoshua at 2:05 PM on April 26, 2007

There are also complicated traffic shaping things you can do to keep one range of IP addresses, or one TCP stream, or whatever, from taking too much more than its share of bandwidth. I don't know if OSX ships with a traffic shaper, but I know there are some available for Linux.
posted by hattifattener at 6:34 PM on April 26, 2007

I'd go with either getting the XServe its own pipe, or getting a real router and doing some hard QoS on the streams. Any router worth its salt (and some that aren't) can do it, you could even repurpose an old PC and throw zebra or something else on it.
posted by Skorgu at 1:48 PM on April 27, 2007