Comments on: Bizarre Linux Networking Problem

Question: Bizarre Linux Networking Problem

signalnine — Mon, 01 Oct 2007 16:13:23 -0800

I have a server running Centos 5. It's currently refusing outside connections to any port except port 22. ip-tables and SELinux are both disabled. Whiskey Tango Foxtrot?

By outside connections, I mean any connection that's not localhost. So, for example, I can telnet to localhost port 25, but from a machine on the same subnet (255.255.255.0) it refuses the connection. I've also checked /etc/hosts.allow and /etc/hosts.deny, both are empty and in xinetd.conf no_access and only_from remain unset.

Seriously guys, I'm sure I'm missing something really stupid, but I'm baffled at the moment.

By: Zed_Lopez

Zed_Lopez — Mon, 01 Oct 2007 16:21:44 -0800

How are the machines networked? Do you have a firewall running on your router?

By: odinsdream

odinsdream — Mon, 01 Oct 2007 16:25:32 -0800

What iptables rules are you using? Have you tried flushing them?

By: genghis

genghis — Mon, 01 Oct 2007 16:27:46 -0800

I dimly recall that this was the default behaviour on lots of RedHat-derived distros. Nothing's open until you open it.

By: Geckwoistmeinauto

Geckwoistmeinauto — Mon, 01 Oct 2007 16:27:55 -0800

What's the output (as root) of:
iptables -L
netstat -ap

By: bh

bh — Mon, 01 Oct 2007 16:31:51 -0800

Are you positive SELinux is completely disabled? Something seems familiar about this question, but I can't quite place it. I ran across something when I was planning an SELinux install, and I seem to remember something about needing an extra step to completely disable it.

By: CrayDrygu

CrayDrygu — Mon, 01 Oct 2007 16:36:11 -0800

It's possible that each individual service is only configured to listen for connections on localhost. Then, even without a firewall, the SMTP server won't respond to outside connections because it isn't even listening for them.

The solution would be to alter each service's configuration so that it listens on all interfaces, or to specify (usually by IP) the interface(s) to listen to.

By: Good Brain

Good Brain — Mon, 01 Oct 2007 16:43:57 -0800

I think CentOS uses APF (advanced protection firewall or something). Check the manpage.

By: a robot made out of meat

a robot made out of meat — Mon, 01 Oct 2007 17:16:25 -0800

http://wiki.centos.org/HowTos/Network/IPTables

The HowTos/ folder has other setup issues.

By: fnord

fnord — Mon, 01 Oct 2007 17:16:36 -0800

This sounds like default RedHat behavior.

Check your /etc/mail/sendmail.cf for the DaemonPortOptions string.

It'll most likely be something like this -

O DaemonPortOptions=Port=smtp,Addr=127.0.0.1, Name=MTA

That makes sendmail only listen on the loopback address.

Change that to something more like this -

O DaemonPortOptions=Name=MTA

What else isn't listening correctly, or is it just sendmail/port 25?

By: Skorgu

Skorgu — Mon, 01 Oct 2007 17:38:12 -0800

This is almost certainly apf being "clever." apf -l will (verbosely) list the rules it's using, apf -f will flush all rules to their default (open) state.

Also useful is apf -a <machine> which will allow a single ip or FQDN to bypass the firewall rules.

By: zengargoyle

zengargoyle — Mon, 01 Oct 2007 18:19:54 -0800

Are you sure about iptables? I'm a CentOS newbie, 'apf' is "command not found' on my box. '/etc/init.d/iptables status' is the check:

# /etc/init.d/iptables status
Firewall is stopped.

if not stopped, do '/et/init.d/iptables stop and check again. I used the 'system-config-securitylevel' interface to turn off SELinux and the Firewall. Except for SSH, and HTTP(S), I would guess that others are correct that the default is to answer to localhost only.

Sorry, another CentOS newbie here. I'm used to Gentoo, CentOS is making me pull out my hair.

By: TravellingDen

TravellingDen — Mon, 01 Oct 2007 21:47:23 -0800

Check the output of netstat -l and see what is actually listening where.

I'd wager as others said that Sendmail is only listening on localhost.

You will see something like:

tcp 0 0 6.53.123.156:53 0.0.0.0:* LISTEN
This is listening to port 53 only on the indicated IP.
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN
This is listening ONLY on localhost
tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN
This is listening for connections on any available interface, signified by 0.0.0.0

By: Malor

Malor — Mon, 01 Oct 2007 22:27:04 -0800

If you get a connection, which is then immediately dropped, you might be encountering 'tcpwrapper', which isn't used much anymore, but which is still fully functional.

/etc/hosts.allow and /etc/hosts.deny list services and hosts; this is generally used by inetd, but many applications are tcpwrapper-aware. On first connection, they consult those files, and if the connection isn't allowed, it's immediately dropped before any conversation takes place. Again, this is rarely used, but if you have the connect-then-drop symptom, that's where to look.

If it's not connecting AT ALL, then you either do have your firewall on, or you haven't configured your daemons to listen to external ports. Many daemons, nowadays, listen only on the loopback interface by default, meaning they can't be reached from the outside world. You have to fix this on a per-daemon basis, generally by telling it to listen to the external IP address instead of 127.0.0.1.

By: Malor

Malor — Mon, 01 Oct 2007 22:33:08 -0800

This script will absolutely purge any rules you have in iptables:

IPTABLES="/sbin/iptables"

IP="/sbin/ip"



EXT_IFACE="eth0"



echo "0" > /proc/sys/net/ipv4/ip_forward



$IPTABLES -P OUTPUT ACCEPT

$IPTABLES -P INPUT ACCEPT

$IPTABLES -P FORWARD ACCEPT



cat /proc/net/ip_tables_names | while read table; do

     $IPTABLES -t $table -L -n | while read c chain rest; do

          if test "X$c" = "XChain" ; then

               $IPTABLES -t $table -F $chain

          fi

     done

     $IPTABLES -t $table -X

done

This assumes that your external interface is eth0, and it disables TCP forwarding, meaning that it will stop routing packets to machines behind it.

I tried to get the script to indent properly, but Metafilter strips out spaces on preview, so I have to post blindly, getting it right the first try. If I failed, my apologies.

By: Malor

Malor — Mon, 01 Oct 2007 22:35:05 -0800

Oh good, that came out right. I'm not sure who the original author was of that script, but it's very effective at completely wiping out your iptables rules, no matter how complex they are.

By: Malor

Malor — Mon, 01 Oct 2007 22:42:49 -0800

Oh, duh, you already checked for tcpwrappers.

If you try to connect and it just hangs, it's probably your firewall stopping it. If it instantly comes back with a 'REFUSED' message, it's probably that the daemon simply isn't listening on the external interface.

By: yomimono

yomimono — Tue, 02 Oct 2007 04:16:37 -0800

I remember having to reboot to fully disable SELinux on RHEL/CentOS 4 (although this smells more like a problem with the daemon not listening on all interfaces, as was said above).

By: signalnine

signalnine — Tue, 02 Oct 2007 08:31:53 -0800

[root@staging-dbserve ~]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

[root@staging-dbserve ~]# netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:57795 *:* LISTEN
tcp 0 0 *:964 *:* LISTEN
tcp 0 0 *:sunrpc *:* LISTEN
tcp 0 0 staging-dbserve:ipp *:* LISTEN
tcp 0 0 staging-dbserve:postgres *:* LISTEN
tcp 0 0 staging-dbserve:smtp *:* LISTEN
tcp 0 0 *:ssh *:* LISTEN
udp 0 0 *:filenet-tms *:*
udp 0 0 *:filenet-nch *:*
udp 0 0 *:958 *:*
udp 0 0 *:961 *:*
udp 0 0 *:mdns *:*
udp 0 0 *:sunrpc *:*
udp 0 0 *:ipp *:*
udp 0 0 *:filenet-rmi *:*
udp 0 0 *:mdns *:*
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 8070 /tmp/.s.PGSQL.5432
unix 2 [ ACC ] STREAM LISTENING 13230 /tmp/ssh-jEEpLE5641/agent.5641
unix 2 [ ACC ] STREAM LISTENING 8472 /var/run/avahi-daemon/socket
unix 2 [ ACC ] STREAM LISTENING 8514 @/var/run/hald/dbus-lO8fS3c7Di
unix 2 [ ACC ] STREAM LISTENING 8513 @/var/run/hald/dbus-D9D0iLYOVH
unix 2 [ ACC ] STREAM LISTENING 8326 /dev/gpmctl
unix 2 [ ACC ] STREAM LISTENING 7158 /var/run/audit_events
unix 2 [ ACC ] STREAM LISTENING 7442 /var/run/dbus/system_bus_socket
unix 2 [ ACC ] STREAM LISTENING 7519 /var/run/sdp
unix 2 [ ACC ] STREAM LISTENING 7664 /var/run/pcscd.comm
unix 2 [ ACC ] STREAM LISTENING 7841 /var/run/acpid.socket
unix 2 [ ACC ] STREAM LISTENING 7884 /var/run/cups/cups.sock

By: signalnine

signalnine — Tue, 02 Oct 2007 08:38:12 -0800

Oh, and there it is. None of the daemons are listening on anything but the loopback by default. Weird. I guess I expected to see at least *one* daemon listening to eth0. Thanks, guys.

By: Malor

Malor — Tue, 02 Oct 2007 09:23:48 -0800

Actually, the *:57795 syntax means "any interface, port 57795". Almost all of your daemons appear to be listening on eth0. Postgres, CUPS, and your mail program might not be, though.

If you do 'netstat -ln', it will print the actual numeric values instead of 'staging-dbserve'. If it's 127.0.0.1, that's your problem.