Networking Protocols

April 8, 2004 11:30 PM

Networking question: What could cause one protocol to be drastically slower than another between the same two computers? [more inside]

I have a Mac OS X desktop and a Windows XP desktop sitting next to each other both connected to a gigabit Ethernet switch. If I copy files from the Windows XP machine to the OS X machine using Samba, it's quite speedy, as I would expect. But if I try to transfer from the Windows XP machine to the OS X machine using SCP (or SFTP), I only get about 15KB/second. In a related problem, if I SSH into the OS X machine from the Windows machine, the SSH connection is exceptionally laggy and occasionally drops entirely.

If I plug my Powerbook into the same switch as well, SCP and SSH work fine (and fast) between the Powerbook and the OS X desktop, or between the XP desktop and the Powerbook.

Have any networking gurus out there experienced a problem like this before? What could be causing problems with one protocol, but not the other, and only between one pair of machines?

posted by m-bandy to work & money (11 comments total)

From my use of SSH / SFTP / whatever at work I've picked up some basic layman stuff, so take this with a small pinch of salt maybe.

I think that SSH/SFTP work by effectlively opening a file on the target computer, and then copying and pasting the contents into the target file in small chunks, then closing the file and saving them ( I visualise this happening in VI regardless of it's accuracy ).

Samba kind of links straight into the file structure of the target machine and uses it like it was it's own drive. Or something.

When we use Samba at work to connect XP machines to a Linux box they are apparently instantanious while the same SSH connection takes ages.
posted by twine42 at 3:50 AM on April 9, 2004

I don't really know the ins and outs of SCP or Samba, but Twine42's description is consistent with what would speed up or slow down communications in general.

All other things being equal, it comes down to a few things:
1. Payload size: how much real data is being communicated per chunk.
2. Header size: how much extra stuff gets tacked onto the chunk to route it, help detect errors, etc.
3. Error recovery: How the protocol deals with lost packets.
4. Encoding: how many bits (or fractions of a bit) you get per modulation--this is more of an issue with wireless and long-haul communications than with local communications, and at any rate, is operating at a lower level in the protocol stack.

If you have small payloads, you're going to divide your message into many small chunks, and spend a lot of time just sending around "did you get it?" "yes, I got it." type messages. If you have large headers, you're using up more of your total bandwidth just on the overhead of delivering packets. And if you've got a dumb error-recovery system, you're going to waste more time re-sending lost packets (and even correctly received packets, in some cases).

There are other factors, like negotiating the connection (part of the reason HTTP is relatively slow), but in general terms, I think those are the main factors.
posted by adamrice at 6:25 AM on April 9, 2004

Well, one thing that does have an effect on SCP/SSH is the speed of the computer particularly with regard to the quality of the implementation of SSH.

How fast is the XP machine? SSH/SCP are both encrypting the file on the fly, so moving the file from one machine to the other takes a LOT more CPU than moving it over with samba. If the XP machine is not particularly fast, or is already overloaded with software running, that can have an effect on it.

It's also possible that there is a compatibility issue between the SSH software you're using on the Macs and the XP box.
posted by jaded at 6:59 AM on April 9, 2004

so there are three machines: XP (windows desktop), PB (powerbook) and OX (OSX desktop).

for ssh/scp/sftp the following are fine:
PB - OX
PB - XP
but this one is slow:
XP - OX

so it's nothing to do with any single machine, because all three machines are capable of a fast link. nor is it anything to do with the protocol in general, because you get fast links with ssh/scp/sftp in some cases.

it's possible that the implementations used for the XP-OX link have some bug that makes them incompatible. maybe they negotiate an unusual cipher set (the actual encryption used depends on the two machines involved and is negotiated before data is transferred) that is buggy. i doubt very much that one cipher set is so much more expensive (computationally) than another that it can explain the problem.

but, to be honest, i'd be more suspicious of your hardware. it's pretty easy to have a bad connection (bad cable or bad socket on the switch) that get's knocked around when you plug things in and out. you start imagining that the behaviour is associated with completely unrelated things, like protocols and ciphers, when in fact is purely random... no criticism intended (it's happened to me, more than once).
posted by andrew cooke at 8:10 AM on April 9, 2004

get's -> gets
fact is -> fact it is
etc
posted by andrew cooke at 8:11 AM on April 9, 2004

I have found that AFP over TCP (using netatalk) is much slower than just plain old FTP.
posted by oaf at 8:16 AM on April 9, 2004

twine42: Not exactly accurate. SFTP is pretty much just regular ftp with ssh wrapped around it. Scp doesn't use any editor either, what it does is login to the remote machine, start a process on that machine that sends some metadata followe by the file (if available this is scp on the remote end too) and then it just reads and writes to the (local) destination file just like everything else does. Same with the roles reverse for copying to a remote destination.

As far as I can see there are two possibilities: Either one of the machines is pegging out on CPU (ssh transfers take more cpu because they encrypt everything), or you're having Long Fat Network problems. TCP has a sliding window of how much data can be in transit, to limit packetloss due to link saturation. However if you have a very high bandwidth link with (moderately) high latency, it can happen that the "amount of data allowed to be in transit" counter gets to its maximum, so even though the link isn't full the bandwidth used gets capped. Note the latency isn't necessarily caused by the link but also by the OS overhead, the data has to get all the way to the application afterall. This probably isn't the greatest explanation, search the RFC's for "LFN" for the full story, plus extensions to fix it; Also, this could only be the problem if samba either uses different tcp options or hacks like multiple connections to solve the problem. Also, it couldn't account for going anywhere near as low as 15KB/s.

All in all, the best advice I can give you is to find out if either side is constantly busy, and if not try tracing the programs and see what they're up to. (Does OSX have ktrace/truss/strace?)
posted by fvw at 10:54 AM on April 9, 2004

The CPU usage on both the PC and the Mac desktop are <5% during the transfer, so I don't think either one is having trouble encrypting/decrypting the data fast enough.

It had occurred to me that maybe there was some incompatibility between XP and OS X, but the Powerbook is running the same release of OS X as the desktop, and it's fine. Maybe it is a hardware problem; I'll try swapping out all the cables and plugging into different ports on the switch (or swapping it temporarily for my old 100BaseT switch.)

On preview, OS X does seem to have ktrace, so perhaps I'll try that as well.

Thanks to everyone for your help.
posted by m-bandy at 11:13 AM on April 9, 2004

Whose SCP on you using on the XP box? I seem to recall having trouble with early versions of Putty's PSCP doing slow transfers, but recent versions are better as are Cygwin's SCP.
posted by Kip at 1:31 PM on April 9, 2004

I'm using WinSCP for SCP and Putty for SSH. I've got the Cygwin scp and ssh installed as well, but I haven't tried either of them to see if they exhibit the problem too.
posted by m-bandy at 1:38 PM on April 9, 2004

Update: the Cygwin SCP works much, much faster between the XP box and the Mac desktop than WinSCP. I'm still a bit curious what WinSCP is doing wrong, but it looks like using Cygwin's SCP instead will fix my problem.

Thanks for the suggestion, Kip -- I was assuming all SCP clients were created equal, but that doesn't seem to be the case.
posted by m-bandy at 12:48 AM on April 10, 2004