Very frustrating, oddball SSH/network problem

February 17, 2010 10:54 AM

Very frustrating, oddball SSH/network problem. Let's take it inside:

There's one server in particular that I access via SSH. I also occasionally access that server via https.

Last week, I went out of town, and after that I could no longer access that server. I figured it was because I was out of town or whatever, but when I returned home, the problem was still there.

Here are some facts about the situation:

1. Macbook Pro, 10.5.8
2. When I SSH, I get "Connection refused."
3. I can access the server via https with no problem.
4. When I ping the server from terminal, I get "no route to host."
5. I can SSH into other servers from this computer with no problem.
6. I can SSH into the server from other computers on my same network. (Same router, modem, etc.)

#5 and #6 tells me that it's a problem with this computer specifically -- not the server, or my home network. But what?

I have, of course, rebooted, but I'm at a loss as to what would be causing the problem with my connection to this server, from this computer, specifically.

Ideas?

posted by nitsuj to Computers & Internet (34 answers total)

It's a problem on the server. Sounds like there is a restrictive firewall in place. A telling sign: you can access HTTPS but your pings are being blocked.

Contact the administrator.
posted by sbutler at 11:01 AM on February 17, 2010

Response by poster: It's a problem on the server.

But like I said in #6: I can SSH into the server from another computer on my same home network, from the same IP.

Oh, and I am the administrator. :)
posted by nitsuj at 11:03 AM on February 17, 2010

Are you trying to ssh from the Mac using a hostname or an IP? If it's by hostname, try using "hostname.local"

Also, could you be a little more specific about how you're trying to access it from out of town? Are you getting through the firewall via VPN, port forwarding?

Re-reading your question, I'm also not clear on whether this server is part of your home network or if it's in a remote location.
posted by contraption at 11:05 AM on February 17, 2010

Are you sure your https requests are getting through and you're not maybe just seeing a cached page? Do you have multiple static IPs at home, or a dynamic IP with your router doing NAT?
posted by Zed at 11:07 AM on February 17, 2010

"Connection refused" means there was no reply from TCP port 22. (HTTPS is on a different port. Pings are RCMP. Therefore none of those failures necessarily indicate a problem.)

Try "ssh -v" for more on what's actually failing.

So, your problem is that your Mac B0 cannot ssh to server S0, but you can connect from another machine on the same network B1 to S0, and you can connect from B0 to another server S1? It looks like a firewall on S0 is not letting in packets from B0.
posted by phliar at 11:07 AM on February 17, 2010

Response by poster: I've tried SSH'ing to the server using both the hostname and the IP. Both produce the same results. From other computers on my home network, I can SSH to the server by both IP and hostname.

The server is not a part of my home network. It's a virtual server leased from a hosting company.
posted by nitsuj at 11:07 AM on February 17, 2010

Turn on debugging for client and server and try a connection. Is the server seeing the connection request? Does the client ever talk to the server? Does the server have an /etc/hosts.deny and/or /etc/hosts.allow file?
posted by nomisxid at 11:08 AM on February 17, 2010

Response by poster: Here are the results of -v.

So, your problem is that your Mac B0 cannot ssh to server S0, but you can connect from another machine on the same network B1 to S0, and you can connect from B0 to another server S1? It looks like a firewall on S0 is not letting in packets from B0.

Yes, that is correct. As I haven't actually modified anything on S0 to prevent access, I'd guess my best course of action is to contact my web host with this information?
posted by nitsuj at 11:11 AM on February 17, 2010

Response by poster: Here's what my webhost support has to say:

It's not possible that you're experiencing a firewall problem because all the computers in your home network should have the same IP (you can double check this but I would be extremely surprised to find differently - go to whatsmyip.net from each machine).

Both machines do indeed have the same IP.
posted by nitsuj at 11:33 AM on February 17, 2010

Take a look at the access logs of your SSH server when you initiate a connection from B1 and again when you try from box B0 (failure). It could be a wealth of things. You could be using a very old version of SSH. You could have a bad ssh key stored for it. You could have some crazy fail2ban by MAC address.

Or you could get nothing at all. But you won't know if you can't check.
posted by pwnguin at 11:43 AM on February 17, 2010

You can try ssh -vv or -vvv if you want more detailed debugging info, but since it just fails with the connection refused error, that probably won't tell you more information.

Try running "telnet S0 22" to see if that can connect to the SSH port at all. If that gives a connection refused error, but you can do say "telnet S0 443" to the https server, that would greatly confuse me.

Is there anyway you can try using your laptop to connect to the server from a different network? That way you can rule that out and place blame.
posted by skynxnex at 11:43 AM on February 17, 2010

Response by poster: Is there anyway you can try using your laptop to connect to the server from a different network? That way you can rule that out and place blame.

Yes. I tried connecting to my work VPN first, then ssh'ing into S0, and it still didn't work... which tells me again that it's something on my computer specifically.
posted by nitsuj at 11:46 AM on February 17, 2010

clear the arp cache on both sides and restart sshd on the server.
posted by rhizome at 11:46 AM on February 17, 2010 [1 favorite]

Response by poster: I tried "telnet S0 443" and that "worked" (in that it said "Connected to S0... escape character is ^]")
posted by nitsuj at 11:48 AM on February 17, 2010

You can get maximum debug info with ssh -vvv but that's probably not going to help. I think "connection refused" means the server is blocking or dropping the connection before you even try to authenticate. When I saw this it was because I'd been blacklisted for too many failed auth attempts, which was actually due to some weird incompatibility between OS X's SSH client and the Windows SSH server. The following entry in ~/.ssh/config fixed it for me, after the admin removed the blacklist entry:

Host example.com
User serathen
PreferredAuthentications publickey

That might be your problem if you're using key-based authentication, but I really don't understand how it could be blacklisting you but not other machines sharing your IP. Try verboser ssh, but also nth-ing the suggestion of checking server-side logs; that's your best bet.
posted by serathen at 11:58 AM on February 17, 2010

Another place to look would be whether there is a _local_ firewall rule preventing this access.

If you haven't installed any third party programs that might do that, check that the firewall is turned off in the firewall tab in the Security prefpane.

If that is turned off, you can also open the Terminal and, as root, run "ipfw list". The response should be "65535 allow ip from any to any".
posted by rcs at 12:08 PM on February 17, 2010

I recently had a very similar situation - could not ssh into the home server (ubuntu 9.10) form the laptop (macbook 10.5). In the end only uninstalling ufw (apt-cache search ufw: ufw - program for managing a Netfilter firewall, so not even a firewall but a front end) fixed it.
posted by mce at 12:15 PM on February 17, 2010

Response by poster: Firewall is turned off in Security. ipfw list says "65535 allow ip from any to any"
posted by nitsuj at 12:24 PM on February 17, 2010

Are you browsing to www.S0 and sshing to S0? Maybe you have an old entry for S0 in your /etc/hosts file and the IP has changed. The www.S0 and ssh on machines without the old IP in a hosts file would resolve via DNS and ssh from the machine with the old IP in the hosts file would have nothing to connect to. Don't laugh, I've done this a few times.
posted by zengargoyle at 12:30 PM on February 17, 2010

nthing the suggestion that you need to check the server-side logs to determine why it is deciding to refuse the connection.

I would also verify that S0 resolves to the same thing on your non-working laptop and a known-working host on the same network, so that you can rule out DNS.

it would be useful to know what OS S0 is running - Linux, one of the BSDs, OSX Server...
posted by namewithoutwords at 12:47 PM on February 17, 2010

Are you browsing to www.S0 and sshing to S0?

Or are you browsing to [S0's domain] and sshing to [S0's actual machine name, plus the domain)? In other words, are the names you're using to access via both methods resolving to the same address?
posted by RikiTikiTavi at 1:00 PM on February 17, 2010

Using the ip address, please run the following:

telnet x.x.x.x 443
telnet x.x.x.x 22

and show us the output.
posted by devnull at 1:03 PM on February 17, 2010

Response by poster: Telnetting to 22: connection refused
Telnetting to 443: works
posted by nitsuj at 1:05 PM on February 17, 2010

Response by poster: the host is a subdomain, so no www when ssh'ing or https'ing
posted by nitsuj at 1:05 PM on February 17, 2010

Both machines do indeed have the same IP.

This is a problem, fix it.
posted by themel at 1:09 PM on February 17, 2010 [1 favorite]

Response by poster: This is a problem, fix it.

Not the same internal IP, but the same external IP. Sorry, my ISP only gives me one IP, and I distribute the access through a router.
posted by nitsuj at 1:18 PM on February 17, 2010

If you have only one external IP, and can't ssh to it, but ssh to the machine from other internal machines, that suggests that the problem is that the router isn't port-forwarding properly. Perhaps it lost the port 22 forwarding rule, or it's forwarding to the incorrect internal IP.
posted by fings at 1:26 PM on February 17, 2010

Response by poster: If you have only one external IP, and can't ssh to it, but ssh to the machine from other internal machines, that suggests that the problem is that the router isn't port-forwarding properly. Perhaps it lost the port 22 forwarding rule, or it's forwarding to the incorrect internal IP.

Hmm. I just put the internal IP of the nonworking machine on the DMZ host, and that didn't change anything, which tells me it's probably not a router/port-forwarding issue.
posted by nitsuj at 1:34 PM on February 17, 2010

When I ping the server from terminal, I get "no route to host."

That's a networking issue. Perhaps https works because its going through some invisible web proxy and skipping over your local network issue. You need to resolve that networking issue first before you can start troubleshooting application level issues.
posted by damn dirty ape at 1:39 PM on February 17, 2010

Check stuff like your default gateway, DNS, subnet mask, etc.
posted by damn dirty ape at 2:04 PM on February 17, 2010

Try starting another sshd on a different port, say 9919, and ssh to that one...does that get thru?
posted by nomisxid at 2:23 PM on February 17, 2010

Well if you definitely want to rule out NAT problems as the issue (and frankly that's the only explanation I can think of at this point) then just temporarily bypass the router and plug your MacBook Pro directly into the modem.
posted by serathen at 3:25 PM on February 17, 2010

Shots in the dark:
Are you using password authentication or using keys?
If using keys do you use a utility like 'keychain'?
Have you authenticated your private-key/keychain properly?
Do you have proper permissions(at least 0700) on your ~/.ssh directory on the host you're connecting from & the proper permissions (at least 0600) on any config or id_rsa files?

(Basically you don't want your authentication mechanisms available to the public and sshd keeps you from connecting with sensitive files that have shoddy perms.)

Troubleshooting:
Try connecting in 'more verbose' mode(ssh -vv hostname) and look for problem areas(rejected modes of authentication, improper perms) in the gobs of output it spits back at you.

Good luck!
posted by mnology at 4:04 PM on February 17, 2010

wait, so you're ssh'ing to the server inside your firewall via the network's external NAT/IP address from an internal machine? what is your topology here? you should be ssh'ing to the server's internal IP when you're both inside the firewall.
posted by rhizome at 9:37 AM on February 18, 2010