vanishing sockets

May 9, 2010 9:02 AM

What the hell is clobbering my ssh-agent and screen sockets in /tmp and how can I make it stop?

Programs that rely on long-lived sockets in /tmp keep falling down because the sockets get removed after a few hours to a day. This means that ssh-agent stops authenticating, I can't reconnect to screen sessions, etc. The processes in question still have file descriptors pointing to the vanished sockets:

$ sudo lsof -p `pidof ssh-agent`
Password:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
[snipped]
ssh-agent 1675 enn 3u unix 0xf6a5f800 0t0 5577 /tmp/ssh-VWCeSB1674/agent.1674

How can I figure out what is happening to these sockets? I don't see any likely suspects in /etc/cron.daily or /etc/cron.hourly.

I am running Arch Linux, but I've run into this problem in the past on Debian, Ubuntu, and Red Hat (though I've had other systems using those distributions where sockets lasted forever with no problems). I realize I can configure ssh-agent and screen to store their sockets elsewhere than in /tmp and have done so in the past as a workaround but I'd really rather not have to do this for every program I might use that wants to keep a socket open for a long time.

posted by enn to Computers & Internet (7 answers total)

Best answer: It sounds like some brain dead shell script that tries to clean up after its mess by indiscriminately unlinking things in /tmp. I know that some distros nuke /tmp at bootup, but never after that, and I've never personally experienced anything like that so I posit that nothing in the distribution is doing it. Check for third party scripts that you might be running, such as install scripts for unpackaged software -- some of those are notoriously shitty. As a workaround you might be able to set the immutable flag on the socket, see chattr(1). You also might want to set something up that alerts you as soon as the socket disappears, using either fam or polling; that way you'll know the guilty party as soon as it happens which should help to narrow things down.

Incidently,

The processes in question still have file descriptors pointing to the vanished sockets:

That means that the socket still exists on disk (i.e. has an inode) but it does not have a directory entry so it cannot be opened by name. The filesystem only actually considers the file deleted when all outstanding handles are closed. So another workaround might be to make a hard link of the socket somewhere outside of /tmp, and then when you notice that it's disappeared, make another hard link of that back in /tmp. This is essentially like creating a directory entry for a file that already has an existing inode.
posted by Rhomboid at 10:13 AM on May 9, 2010 [1 favorite]

In general, /tmp is cleaned on reboot. If it is cleaned more frequently that is normally performed by a crontab entry. I would look for a cron job other than cron.daily or cron.hourly that is being executed in /etc/crontab.
posted by Lame_username at 10:19 AM on May 9, 2010

Response by poster: Lame_username, there are no system cron jobs and the only user crontab is root's, which does nothing but run the cron.hourly, cron.daily, etc. jobs.

Rhomboid, thanks for the chattr tip and the idea of making extra hard links. It makes sense that another program might be over-aggressive in cleaning up after itself; I've set up an inotifywait on a screen socket and will see what happens.
posted by enn at 11:07 AM on May 9, 2010

Are the permissions on /tmp correct? It should be drwxrwxrwt aka 4777. The t is important.
posted by fritley at 2:23 PM on May 9, 2010

crap, I meant aka 1777
posted by fritley at 2:24 PM on May 9, 2010

If it's a shell script doing it, you should be able to find it by replacing your rm with a script that calls the real rm but also logs that action in a file somewhere. If it's something else, you could always hack something like that into the kernel or into unlink() in glibc, depending how desperate you get to figure this out.
posted by Xezlec at 2:26 PM on May 9, 2010

Response by poster: fritley: yes, the sticky bit is set.
posted by enn at 4:22 PM on May 9, 2010