Bizarre Linux Networking Problem

October 1, 2007 4:13 PM

I have a server running Centos 5. It's currently refusing outside connections to any port except port 22. ip-tables and SELinux are both disabled. Whiskey Tango Foxtrot?

By outside connections, I mean any connection that's not localhost. So, for example, I can telnet to localhost port 25, but from a machine on the same subnet (255.255.255.0) it refuses the connection. I've also checked /etc/hosts.allow and /etc/hosts.deny, both are empty and in xinetd.conf no_access and only_from remain unset.

Seriously guys, I'm sure I'm missing something really stupid, but I'm baffled at the moment.

posted by signalnine to computers & internet (20 comments total)

How are the machines networked? Do you have a firewall running on your router?
posted by Zed_Lopez at 4:21 PM on October 1, 2007

What iptables rules are you using? Have you tried flushing them?
posted by odinsdream at 4:25 PM on October 1, 2007

I dimly recall that this was the default behaviour on lots of RedHat-derived distros. Nothing's open until you open it.
posted by genghis at 4:27 PM on October 1, 2007

What's the output (as root) of:
iptables -L
netstat -ap
posted by Geckwoistmeinauto at 4:27 PM on October 1, 2007

Are you positive SELinux is completely disabled? Something seems familiar about this question, but I can't quite place it. I ran across something when I was planning an SELinux install, and I seem to remember something about needing an extra step to completely disable it.
posted by bh at 4:31 PM on October 1, 2007

It's possible that each individual service is only configured to listen for connections on localhost. Then, even without a firewall, the SMTP server won't respond to outside connections because it isn't even listening for them.

The solution would be to alter each service's configuration so that it listens on all interfaces, or to specify (usually by IP) the interface(s) to listen to.
posted by CrayDrygu at 4:36 PM on October 1, 2007

I think CentOS uses APF (advanced protection firewall or something). Check the manpage.
posted by Good Brain at 4:43 PM on October 1, 2007

http://wiki.centos.org/HowTos/Network/IPTables

The HowTos/ folder has other setup issues.
posted by a robot made out of meat at 5:16 PM on October 1, 2007

This sounds like default RedHat behavior.

Check your /etc/mail/sendmail.cf for the DaemonPortOptions string.

It'll most likely be something like this -

O DaemonPortOptions=Port=smtp,Addr=127.0.0.1, Name=MTA

That makes sendmail only listen on the loopback address.

Change that to something more like this -

O DaemonPortOptions=Name=MTA


What else isn't listening correctly, or is it just sendmail/port 25?
posted by fnord at 5:16 PM on October 1, 2007

This is almost certainly apf being "clever." apf -l will (verbosely) list the rules it's using, apf -f will flush all rules to their default (open) state.


Also useful is apf -a <machine> which will allow a single ip or FQDN to bypass the firewall rules.
posted by Skorgu at 5:38 PM on October 1, 2007

Are you sure about iptables? I'm a CentOS newbie, 'apf' is "command not found' on my box. '/etc/init.d/iptables status' is the check:

# /etc/init.d/iptables status
Firewall is stopped.

if not stopped, do '/et/init.d/iptables stop and check again. I used the 'system-config-securitylevel' interface to turn off SELinux and the Firewall. Except for SSH, and HTTP(S), I would guess that others are correct that the default is to answer to localhost only.

Sorry, another CentOS newbie here. I'm used to Gentoo, CentOS is making me pull out my hair.
posted by zengargoyle at 6:19 PM on October 1, 2007

Check the output of netstat -l and see what is actually listening where.

I'd wager as others said that Sendmail is only listening on localhost.

You will see something like:

tcp 0 0 6.53.123.156:53 0.0.0.0:* LISTEN
This is listening to port 53 only on the indicated IP.
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN
This is listening ONLY on localhost
tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN
This is listening for connections on any available interface, signified by 0.0.0.0
posted by TravellingDen at 9:47 PM on October 1, 2007

If you get a connection, which is then immediately dropped, you might be encountering 'tcpwrapper', which isn't used much anymore, but which is still fully functional.

/etc/hosts.allow and /etc/hosts.deny list services and hosts; this is generally used by inetd, but many applications are tcpwrapper-aware. On first connection, they consult those files, and if the connection isn't allowed, it's immediately dropped before any conversation takes place. Again, this is rarely used, but if you have the connect-then-drop symptom, that's where to look.

If it's not connecting AT ALL, then you either do have your firewall on, or you haven't configured your daemons to listen to external ports. Many daemons, nowadays, listen only on the loopback interface by default, meaning they can't be reached from the outside world. You have to fix this on a per-daemon basis, generally by telling it to listen to the external IP address instead of 127.0.0.1.
posted by Malor at 10:27 PM on October 1, 2007 [1 favorite has favorites]

This script will absolutely purge any rules you have in iptables:

IPTABLES="/sbin/iptables"
IP="/sbin/ip"

EXT_IFACE="eth0"

echo "0" > /proc/sys/net/ipv4/ip_forward

$IPTABLES -P OUTPUT ACCEPT
$IPTABLES -P INPUT ACCEPT
$IPTABLES -P FORWARD ACCEPT

cat /proc/net/ip_tables_names | while read table; do
     $IPTABLES -t $table -L -n | while read c chain rest; do
          if test "X$c" = "XChain" ; then
               $IPTABLES -t $table -F $chain
          fi
     done
     $IPTABLES -t $table -X
done

This assumes that your external interface is eth0, and it disables TCP forwarding, meaning that it will stop routing packets to machines behind it.

I tried to get the script to indent properly, but Metafilter strips out spaces on preview, so I have to post blindly, getting it right the first try. If I failed, my apologies.
posted by Malor at 10:33 PM on October 1, 2007 [1 favorite has favorites]

Oh good, that came out right. I'm not sure who the original author was of that script, but it's very effective at completely wiping out your iptables rules, no matter how complex they are.
posted by Malor at 10:35 PM on October 1, 2007

Oh, duh, you already checked for tcpwrappers.

If you try to connect and it just hangs, it's probably your firewall stopping it. If it instantly comes back with a 'REFUSED' message, it's probably that the daemon simply isn't listening on the external interface.
posted by Malor at 10:42 PM on October 1, 2007

I remember having to reboot to fully disable SELinux on RHEL/CentOS 4 (although this smells more like a problem with the daemon not listening on all interfaces, as was said above).
posted by yomimono at 4:16 AM on October 2, 2007

[root@staging-dbserve ~]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination


[root@staging-dbserve ~]# netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:57795 *:* LISTEN
tcp 0 0 *:964 *:* LISTEN
tcp 0 0 *:sunrpc *:* LISTEN
tcp 0 0 staging-dbserve:ipp *:* LISTEN
tcp 0 0 staging-dbserve:postgres *:* LISTEN
tcp 0 0 staging-dbserve:smtp *:* LISTEN
tcp 0 0 *:ssh *:* LISTEN
udp 0 0 *:filenet-tms *:*
udp 0 0 *:filenet-nch *:*
udp 0 0 *:958 *:*
udp 0 0 *:961 *:*
udp 0 0 *:mdns *:*
udp 0 0 *:sunrpc *:*
udp 0 0 *:ipp *:*
udp 0 0 *:filenet-rmi *:*
udp 0 0 *:mdns *:*
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 8070 /tmp/.s.PGSQL.5432
unix 2 [ ACC ] STREAM LISTENING 13230 /tmp/ssh-jEEpLE5641/agent.5641
unix 2 [ ACC ] STREAM LISTENING 8472 /var/run/avahi-daemon/socket
unix 2 [ ACC ] STREAM LISTENING 8514 @/var/run/hald/dbus-lO8fS3c7Di
unix 2 [ ACC ] STREAM LISTENING 8513 @/var/run/hald/dbus-D9D0iLYOVH
unix 2 [ ACC ] STREAM LISTENING 8326 /dev/gpmctl
unix 2 [ ACC ] STREAM LISTENING 7158 /var/run/audit_events
unix 2 [ ACC ] STREAM LISTENING 7442 /var/run/dbus/system_bus_socket
unix 2 [ ACC ] STREAM LISTENING 7519 /var/run/sdp
unix 2 [ ACC ] STREAM LISTENING 7664 /var/run/pcscd.comm
unix 2 [ ACC ] STREAM LISTENING 7841 /var/run/acpid.socket
unix 2 [ ACC ] STREAM LISTENING 7884 /var/run/cups/cups.sock
posted by signalnine at 8:31 AM on October 2, 2007

Oh, and there it is. None of the daemons are listening on anything but the loopback by default. Weird. I guess I expected to see at least *one* daemon listening to eth0. Thanks, guys.
posted by signalnine at 8:38 AM on October 2, 2007

Actually, the *:57795 syntax means "any interface, port 57795". Almost all of your daemons appear to be listening on eth0. Postgres, CUPS, and your mail program might not be, though.

If you do 'netstat -ln', it will print the actual numeric values instead of 'staging-dbserve'. If it's 127.0.0.1, that's your problem.
posted by Malor at 9:23 AM on October 2, 2007