Computer says no one more time, it's going out the Window^W Linux

February 18, 2011 8:11 AM

Why can't I delete these files from my Linux server?

I've got a Centros 5 server that's crapping out on an update because of some files it can't use. So it's been decided that we should delete the files. Except we can't...

Logged in as root.
/usr is set as drwxr-xr-x root root
/usr/sbin is set as drwxr-xr-x root root

The files are -rw-r--r-- root root
and lsattr gives -uS---dAc----

All well and good (as I understand it).

But, when I type rm filename I get the error rm: cannot remove `(filename)': Operation not permitted

Now, the filenames have semicolons in them, but I'm using "partfilename*" to refer to them.

What's happening and how do I fix it? I thought root let me do anything, regardless of how stupid or dangerous it was...?

NB: I've tried searching google, but all I'm getting is people claiming you need to be root and you need to unset 'i', which isn't set here.

Hope me?

posted by twine42 to Computers & Internet (27 answers total)

Try escaping the semicolon, if that's the character you think is making it difficult.

rm foo\;bar

It would be a lot easier if there was an example file name.
posted by o0o0o at 8:28 AM on February 18, 2011

Don't know. But you can escape the semicolons with backslash to address the file directly, instead of using a wildcard:

rm my\;weirdly\;named\;file
posted by qxntpqbbbqxl at 8:29 AM on February 18, 2011

The "partfilename*" is being expanded at the shell level before rm is called. The semicolons may not be escaped correctly. Try entering the filename by hand and escaping the semicolons yourself.
posted by ChrisHartley at 8:29 AM on February 18, 2011

rm -- file* should do it.
posted by Cat Pie Hurts at 8:32 AM on February 18, 2011 [1 favorite]

To explain, from the bash manpage:

-- A -- signals the end of options and disables further option processing. Any arguments after the -- are treated as filenames and arguments. An argument of - is equivalent to --.
posted by Cat Pie Hurts at 8:34 AM on February 18, 2011

Response by poster: Sorry guys, I automatically removed the filename without considering its relevance...

rm convert4\;4d5e5d72
rm: cannot lstat `convert4;4d5e5d72': No such file or directory

When I try the * version I get...

rm convert4*
rm: remove regular empty file `convert4r4;4d5e5d72'? y
rm: cannot remove `convert4r4;4d5e5d72': Operation not permitted
posted by twine42 at 8:35 AM on February 18, 2011

The speculation about the file name and the shell interpretation seem wrong, since OP is getting an error about can't delete file, meaning the name is parsed correctly.

Have you tried readmail real fast ? ("rm -rf filename") (or just -f, especially if you're using wildcards, even more especially if you're whacking things in /sbin ..)
posted by k5.user at 8:40 AM on February 18, 2011

Response by poster: k5.user - tried and failed I'm afraid...

We tried rebooting the machine in case the files were in use by something but no luck with that either...
posted by twine42 at 8:44 AM on February 18, 2011

Is the file open?

$ lsof | grep 'convert4'

If it's there, you need to stop or kill the process that has it open.
posted by advicepig at 8:48 AM on February 18, 2011

Might be SELinux. I'm not a Redhat guy, and I've never really worked with SELinux, but there should be a method to turn it off... on Debian, it would likely be something like /etc/init.d/selinux stop. Try that, and see if you can delete the files.

Also check to see if you're running Tomoyo or AppArmor... any of the security modules could be preventing that file access.

Remember that, if these files are that strongly protected, there may be a reason. Tread carefully.
posted by Malor at 8:50 AM on February 18, 2011

Response by poster: advicepig - Ooooo cool command.

Sadly, nothing.
posted by twine42 at 8:52 AM on February 18, 2011

Response by poster: Malor - SELinux isn't on the system.

And I take the point about the files being protected for a reason but they're 0bytes long so...
posted by twine42 at 8:56 AM on February 18, 2011

I came in to suggest a possible SELinux issue as well. I've admined several RHEL boxes, and always end up disabling SELinux cause it's a pain in my ass (and the boxes were always on private networks so I had less security risk).

Check here for some info on SELinux and Centos.
posted by blind.wombat at 8:57 AM on February 18, 2011

Go to init run level 1 (single user mode) and try again - that should disable any funny business. Or boot from a livecd, mount the fs and delete it that way. But yeah, probably better to figure out what is really going on before pulling out the big bowie knife.
posted by ChrisHartley at 8:57 AM on February 18, 2011

Since you are doing low-level maintenance, it behooves me to ask, have you checked to make sure /usr is mounted read-write at your current run-level?
posted by nomisxid at 8:57 AM on February 18, 2011

I just Googled it, and you can see if selinux is being enforced with this command:

cat /selinux/enforce

If it's non-zero, remember the value, and:

echo 0 >/selinux/enforce

Then try deleting your files. Whether it works or not, if the enforce value was non-zero, remember to restore it.
posted by Malor at 8:58 AM on February 18, 2011

find . -type f -iname 'convert4*' -print0 | xargs -0 echo rm --

Assuming that seems to find the files, remove the echo.

However, it shouldn't be that hard to hit a file with the semi-colon. I'd use chattr to remove all of the attributes; especially undelete, since you're getting operation not permitted.
posted by nobeagle at 9:03 AM on February 18, 2011

Response by poster: Just suggested the "run level 1" bit to the guy who knows enough to say "Oh yeah...".

For anyone worrying about what's happening, we're trying to update exim after someone used an exploit to send a nice load of spam. Every time we do, yum complains about not being able to access one of these files. Here's the origins of the problem...

rpm -i --force exim-4.63-5.el5_5.2.i386.rpm
error: unpacking of archive failed on file /usr/sbin/convert4r4;4d5ea683: cpio: open failed - Operation not permitted

Mind you, that file has the timestamp of now, so maybe my question is all arse about face...
posted by twine42 at 9:06 AM on February 18, 2011

Did you try my suggestion? rm -- filename*

If that doesn't work, I would drop to singleuser as ChrisHartley suggests. You may also want to run a fsck to make sure everything is ok.
posted by Cat Pie Hurts at 9:07 AM on February 18, 2011

Best answer: Exploit? I would run a rootkit checker (http://www.chkrootkit.org/) ASAP.
posted by Cat Pie Hurts at 9:09 AM on February 18, 2011

Best answer: If you did experience an exploit you might want to take the time to rule out a rootkit or other tampering. The real explanation is almost certainly more mundane but a poorly implemented rootkit could screw up your shell or rm/ls/etc commands in a way that produces curious errors.
posted by ChrisHartley at 9:11 AM on February 18, 2011

Response by poster: Sorry Cat Pie Hurts, yes I did - I just forgot to say so.

Mentioned rootkit checkers and got told that we were clean... *shrug*
posted by twine42 at 9:14 AM on February 18, 2011

Fsck it! =)
posted by blind.wombat at 9:16 AM on February 18, 2011

Response by poster: On a whim, just used vi to create a file in sbin. vi announced that the swap file couldn't be used. Deleting the file now creates the same error message.

Is that of any help to anyone? Or is that to be expected?

I'm really appriciating this help guys.
posted by twine42 at 9:18 AM on February 18, 2011

that "cpio: open failed - Operation not permitted" is often indicative of a rootkitted server. Did the person who said the server was clean actually run chkrootkit?
posted by Cat Pie Hurts at 9:20 AM on February 18, 2011

Response by poster: Well... it's still searching sniffer logs, but I'm getting infected messages for ls, ps and top, so I think the word of the day is "cock".

Thanks cat...
posted by twine42 at 9:38 AM on February 18, 2011

Response by poster: Right... we have a few suspects for possible rootkits from cat's suggestion and from rkhunter. I'm shutting the machine down and leaving it til Monday.

Thanks for the help guys.
posted by twine42 at 10:06 AM on February 18, 2011